Cloud SQL - Private Service Access - Error: Permission denied on resource project ...

[lakatostomi]

When I am trying to create a Cloud SQL instance with private ip I get the following error message when configuring the Private Service Access:

Permission denied on resource project 822070555046. Help Token: AX4KC-gUWKiK5KFuPLH6UY4uFhqTf93FkyuzUgiagGyDq7nVWZfIDDAaaYC_PkkLIt1EFU3rfXsen1Ui7tZ_eJMTrksH06t5-12cK1WDFZXavamQ

I have the right permissions I checked the documentation! The referenced project (822070555046) in the message does not exist under my projects (checked with gcloud projects describe 822070555046). 

Any idea why I am getting this error message? A few weeks ago I was able to create PSA without any error, with the same permissions in the same project!!!

Thanks for any help!!!

Tamas

[end]

I am having exactly the same problem. There seems to be some maintenance actions going on at GCP, maybe these have some effect?

[lakatostomi]

Yes maybe, this error has existed for about 2 days. Others have the same problem:

https://www.reddit.com/r/googlecloud/comments/1b18zd7/permission_denied_on_different_project_number/...

https://www.googlecloudcommunity.com/gc/Infrastructure-Compute-Storage/Error-waiting-for-Create-Serv...

[ms4446]

This error in Google Cloud SQL typically indicates that your user account or service account lacks the necessary permissions to perform the actions needed to create a Cloud SQL instance with Private Service Access (PSA).

Troubleshooting Steps:

1. Project Verification:

   • Project ID vs. Project Number: Ensure you are using the correct project ID. The error message may provide a project number, but you likely need the project ID for most actions. Find this in the Google Cloud Console dashboard.
   • Project Selection: Double-check that you have the correct project selected in the Google Cloud Console or when using gcloud commands. Use gcloud config list project to verify your gcloud configuration.

2. IAM Permissions:

   • User Account Roles: Verify you possess:
     • roles/compute.networkAdmin or at least roles/compute.networkUser for network management.
     • roles/cloudsql.admin for Cloud SQL administration.
   • Service Account Roles: If you are using a service account, ensure it has the above roles within the project where the Cloud SQL instance will be created.
   • Service Networking Permissions: The roles/servicenetworking.connectionUser role is essential for establishing PSA connections.

3. API Verification:

   • Enabled APIs: Make sure the following APIs are enabled within your project (APIs & Services in the console):
     • Compute Engine API
     • Cloud SQL Admin API
     • Service Networking API

4. Additional Considerations:

   • Recent Changes: Were there recent changes to IAM roles, firewall rules, or service accounts in your project? These might have inadvertently revoked permissions.
   • Logging: If possible, access detailed logs that may shed more light on the specific cause of the permission denial.

If the Issue Persists (Requesting Further Assistance):

To get more targeted help, please provide the following:

• Exact Commands: Share the commands used for Cloud SQL instance creation and PSA setup.
• IAM Setup: Describe your IAM roles (or those of the service account).
• Screenshots: Provide screenshots (redact sensitive data) if they help illustrate your configuration.

If all else fails, seek assistance from Google Cloud Support.

[lakatostomi]

Thank you for the detailed explanation and troubleshooting steps, but the problem is not solved 😞

1. The project is verified, and the right project is selected!
2. IAM Permissions
   
   • User roles: roles/owner, roles/servicenetworking.networksAdmin, roles/compute.networkAdmin, roles/cloudsql.admin
   • Service account roles: roles/owner, roles/servicenetworking.networksAdmin, roles/compute.networkAdmin, roles/cloudsql.admin
3. All APIs are enabled!
4. There were no recent changes to IAM roles, I previously deployed the infrastructure with the same roles a few times.

When I use Cloud console to create PSA I get this message:

 This project in the message does not exist, here are my projects with gcloud projects list:

 With terraform I get the same error message, here is the code snippet: 

I use Cloud Foundation Fabric modules in these snippets.

The error message when using Terraform:

As I mentioned, I have already deployed this project with Terraform with the same roles and everything went well. Have no idea why I get this error. 

Thanks for further help!

Tamas

[ms4446 · Accepted solution]

Based on the details and your previous troubleshooting, here's the plan you might to follow:

1. Resource Manager API:

   • Verify Enablement: Go to APIs & Services in your project and search for "Cloud Resource Manager API." Ensure it has a green checkmark indicating it's enabled.

2. Project Linkage and Quotas:

   • Billing: In the Cloud Console, go to "Billing." Make sure the project is linked to an active billing account.
   • Organization: If under an organization, navigate to "IAM & Admin" -> "Settings" and examine restrictions that might hinder your setup.
   • Quotas: In "IAM & Admin" -> "Quotas," search for "Service Networking" quotas specific to your region. Check that usage isn't reaching limits. Do the same for other potentially relevant quotas, like global IP addresses.

3. Cross-Project Considerations:

   • Scrutinize Shared VPCs: If your setup involves shared VPCs or any resources spanning multiple projects, meticulously check permissions for your service account/user in all involved projects.

4. Terraform State & Service Account:

   • terraform refresh: Run this to ensure your Terraform state is in sync with Google Cloud's reality.
   • Service Account Scopes: If you use a service account, double-check the scopes it's been granted are sufficient for all the APIs involved in setting up PSA.

5. Explicit Network User Role:

   • Try Granting It: Even with broader roles present, sometimes explicitly granting roles/compute.networkUser to the service account on the relevant network can resolve subtle permission hiccups.

Support & Community Engagement:

• Google Cloud Support: If previous steps haven't yielded a solution, this is your next step. Prepare detailed documentation of your setup, previous successes, the error, and steps you've already taken.
• Community: Leverage forums like Google Cloud's own or Stack Overflow. Even if you don't find a solution immediately, describing your situation meticulously can trigger insights from others.

Additional Thoughts:

• Versioning: I like the suggestion to ensure compatibility between Cloud Foundation Fabric modules, Terraform Provider, and the Google Cloud APIs themselves. While less likely than permissions, mismatches can cause headaches.
• Verbose Logging: A great mention! Terraform's TF_LOG environment variable and detailed Google Cloud logs often reveal clues.

[lakatostomi]

Thanks for the troubleshooting steps again. I followed your instructions and checked everything you mentioned. The issue is still not solved, BUT I have created a new project where I was able to set up a PSA without any problem! So more or less the issue is solved, but still have no idea what caused the problem. This is a side-project without organization and using the free trial usage.  Maybe it has some limitations, although the limits and quotas are not reached!

Thanks again for your help!  Your troubleshooting steps were very useful and instructive!

Tamas

[BartPudlo]

@lakatostomi was it solved? If so, do we know exactly what helped to solve it? 

Or as per reddit post - was it solved by not changing anything and it was solved from Google end? 

[lakatostomi]

I could solve the issue only by creating a new project and a new deployment as I posted to Reddit!

I do not know the exact reason why PSA works in a new project and fails in the old one.

[end]

We got answer from GCP support:

we are seeing this issue not due to a permission aspect, but due to the fact that the resources associated with the tenant project 1234567890 have been removed previously. When there are no resources such as a servicenetworking PSA connector, the tenant project is not populated and automatically marked for deletion. This is the reason for which we are seeing this following message:

Permission denied on resource project 1234567890. Help Token: AX4KC-iDTcdGoCHgq--qFVhQ5FvMFgL2PmuYV

Therefore, in regards to unlocking this we have two options - wait for the resources to be automatically cleared or issue a request to restore the currently marked for deletion 104096410161 project.”

[lakatostomi]

Great information! Thanks for sharing!

[rkvvva]

This error appeared in my project that has been stable and working fine for many years. Waiting for 24 hours has not fixed the issue. 

Thanks for posting Google Support reply @end. Unfortunately their reply does not seem actionable.

How do we fix our broken projects? 

[BartPudlo]

What error message are you getting? 

[rkvvva]

Hi @BartPudlo , the error occurs when I try to create a Private Service Connection to my VPC.

This is the error message:
"Error waiting for Create Service Networking Connection: Error code 7, message: Permission denied on resource project 01234567890123."

Things to note:
 - project 01234567890123 is not owned by me. I cannot see it or change permissions in it.
 - this same configuration previously worked fine for years. 
 - several other people had this issue start happening in the last week. there are several GCP support forum posts and reddit posts with same error message about missing permissions in a project they don't control

Other posters worked around the issue by abandoning their GCP project and creating a new GCP project. This workaround is not an option for me.

[rkvvva]

Update: resolved the issue by changing the name of the VPC I was trying to create and deploy.

In terraform resource "google_compute_network" I changed the 'name' element by one character.
And now resource "google_service_networking_connection" applies onto that VPC with no problem.

It seems something with the VPC name conflicted in the backend.  Maybe I deleted a VPC with that same name a few years ago?

[BartPudlo]

Got it! Happy to hear that now it is solved. 

I will try to investigate it deeper to see what might be causing this issue. 

Thanks! 

[rubensayshi]

We're also hitting this issue.

In our case we've also destroyed the VPC in the past and created a new one and are hitting this issue... We've done this many times in the past without issues, this is a new issue. 

It's not really a permanent solution, but to confirm I also tried changing the name of the VPC that I'm using and it indeed resolves the issue.
Which to me sounds like a bug, specially considering this used to work fine!