Error during `gcloud app deploy` for GAE app: "Failed to create cloud build: invalid bucket"

[myconode]

After creating a new Google Cloud project and following all steps carefully from this deployment documentation I see the following error (edited for brevity) after running `gcloud app deploy`:

Error Response: [13] Failed to create cloud build: default Cloud Build service account or user-specified service 
account does not have access to the bucket

I have been careful to follow all requirements (i.e. enabling APIs/services, enabling billing for the project, etc).

In years prior App Engine would abstract the artifact storage component of the platform. Now, it seems that "legacy" cloud build service accounts are being provisioned without the right permissions. I have tried many mitigation steps from posts like this as well as StackOverflow posts like this but to no avail.

To fix the specific error above, I have granted (and confirmed) that both the AppEngine default service account and Cloud Build service account have admin permissions for all related services (cloud build, artifact registry, cloud storage), and have also used `gsutil` to confirm that the service account in question has access to the bucket, but the deployment command still fails and emits the same error.

Any help is greatly appreciated.  For anyone on the GCP/GAE team - if this is a user error I think it would be beneficial for the community if the troubleshooting documentation is updated as well.

[DamianS]

Hello @myconode  ,Welcome on Google Cloud Community.

What does the log says ( Logs from Logs Explorer) ? Are you sure, that service account which you are using have Storage Object Viewer/Admin role ?
You can check you Service Account y entering CLoud Build -> Settings -> Service Account

--
cheers,
DamianS
LinkedIn medium.com Cloudskillsboost

[myconode]

Hi DamianS, thank you for your reply.

I've ensured the service account setting above is enabled, but no luck. 

This morning I started from scratch again: new GCP project, Enabled Cloud Build as instructed here, confirmed billing is enabled, created a basic node.js app that meets the requirements, and attempted deployment. Same failure. I checked the logs and here is a summary of the actions:

• Action: AssignResourceToBillingAccount

  • Status: Success
  • Performed By: <REDACTED USER NAME>
  • Details: Assigned resource to billing account for project <REDACTED PROJECT NAME>.

• Action: EnableService (cloudbuild.googleapis.com)

  • Status: Success
  • Performed By: <REDACTED USER NAME>
  • Details: Enabled Cloud Build service for project <REDACTED PROJECT NAME>.

• Action: SetIamPolicy

  • Status: Failure
  • Performed By: <REDACTED CLOUD BUILD SERVICE ACCOUNT>
  • Reason: Concurrent policy changes; required retry with exponential backoff.
  • Details: Attempted to set IAM policy for project <REDACTED PROJECT NAME>.

• Action: EnableService (cloudbuild.googleapis.com)

  • Status: Success
  • Performed By: <REDACTED USER NAME>
  • Details: Enabled Cloud Build service for project <REDACTED PROJECT NAME>.

• Action: SetIamPolicy

  • Status: Failure
  • Performed By: <REDACTED CLOUD BUILD SERVICE ACCOUNT>
  • Reason: Concurrent policy changes; required retry with exponential backoff.
  • Details: Attempted to set IAM policy for project <REDACTED PROJECT NAME>.

• Action: CreateServiceAccount

  • Status: Success
  • Performed By: <REDACTED USER NAME>
  • Details: Created a service account for project <REDACTED PROJECT NAME>.

• Action: CreateApplication (App Engine)

  • Status: Success
  • Performed By: <REDACTED USER NAME>
  • Details: Created an App Engine application for project <REDACTED PROJECT NAME>.

• Action: SetIamPolicy

  • Status: Failure
  • Performed By: <REDACTED CLOUD BUILD SERVICE ACCOUNT>
  • Reason: Concurrent policy changes; required retry with exponential backoff.
  • Details: Attempted to set IAM policy for project <REDACTED PROJECT NAME>.

It appears as though the provisioning of the cloud build service account and related IAM policies are failing, and despite manual attempts to grant the appropriate permissions, there are still failures.

I have been banging my head against this problem for days. Just a few months ago I had no problem creating a new App Engine project (took minutes to create/deploy). I think there is a bug and I hope someone from the GCP team is reading. Also happy to try any other remediation steps, but so far granting and confirming (and double confirming) that the service accounts in question have Storage Object Viewer/Admin role and other related roles (Artifact Registry Writer, etc) has resulted in the same errors.

In short, the standard documentation to create a new GAE project is either out of date (steps missing) or there is a bug somewhere in the provisioning process.

[DamianS]

Hmm, I've used documentation from your first post "all steps carefully from this deployment documentation" and my GAE app works like a charm. New project, I've enabled API and follow steps from tutorial. 

[myconode]

Thanks for your reply. Other members of my team have tried to repro in their own GCP organizations and they have not had success. The current organization I created is pretty new, perhaps there is some global policy missing? At a loss as to what the problem could be. I've created several other new projects from scratch following the documentation closely and confirming each step along the way and each project results in this issue.

[DamianS]

Okay. So you did following steps, right?
1. You've created new project, from scratch?
2. You've enabled CLoud Build API.
3. You've created your folder with needed files for deploy AppEngine
4. When you've hit gcloud app deploy, additionally being at your app folder, you've received errors with cloud build permissions.

--
cheers,
DamianS
LinkedIn medium.com Cloudskillsboost

[myconode]

Yes, exactly. After step 1 I ensured a billing account was connected to the project. At step 3 I've ensured that the app runs locally (copying the example express app from the instructions), has a valid app.yaml, etc. At step 4 I've ensured that I'm logged in with gcloud for the right account, set/initialized the default project and then attempted `gcloud app deploy`. Same issue every time.

[myconode · Accepted solution]

After pulling my hair out for a few days, I finally figured it out with a little help from this kind person who responded to my SO. I had a feeling there was some global policy at fault here:

• As of May 23rd 2024, Google changed default global permissions for new organizations, in particular disabling iam.automaticIamGrantsForDefaultServiceAccounts for improved security defaults.
• This broke IAM provisioning for new service accounts associated with App Engine (and probably other offerings)
  • This of course was not mentioned in any of the App Engine starter documentation

Fix:

The default app engine service account (or whatever service account is created/selected) needed to have three permissions:

• Artifact Registry Create-on-Push Writer
• Storage Admin
• Logs Writer

@DamianS it makes sense why it worked for you if your GCP organization was created prior to 5/23/2024 and iam.automaticIamGrantsForDefaultServiceAccounts is enabled.

This was a very painful incident and extremely difficult to debug. I suspect that anyone trying to create a GAE app with a new organization is having a bad time. I hope someone from the GCP product team updates the documentation and/or creates a bug ticket to reduce developer friction.

[myconode]

Epilogue: perhaps one of the most frustrating things about this was that "Artfact Registry Administrator" does not encompass the "Artifact Registry Create-on-Push Writer" permissions, which is why I was losing my mind trying to figure out why the service account couldn't push artifacts during the build process (you'd think that administrator could create-on-push....)

[DamianS]

Ahh,
I didn't been aware of such change. Thanks for posting @myconode 🙂

[daskalou]

Quick question / clarification: When you say "... needed to have three permissions", I assume you mean "... three roles", is that right?

[myconode]

Right, three roles

[daskalou]

Thanks for posting this and the solution.

I've just run into the same issue. Hopefully your solution fixes my problem too.

[myconode]

Did it?

[daskalou]

I think so. It's working now, but to be honest I can't remember if your solution fixed it or other tinkering did (I'm assuming it was your solution that worked).

[davidtr]

---

@myconode wrote:

I suspect that anyone trying to create a GAE app with a new organization is having a bad time.

---

Can confirm, had quite a bad time, thanks for the solution. If anyone wonders how to add the roles I used the gcloud command line (haven't found how to in the Google Console):

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" \ --role="roles/artifactregistry.createOnPushWriter"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" \ --role="roles/storage.admin"

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" \ --role="roles/logging.logWriter"

[mjodmk]

Hi @myconode, 

Just a quick thank you for saving me here!

Also I came across your issue quicker than anything in the docs that mentions this problem, so they appear to be yet to address this.