App Engine gcloud app deploy permission Regions.get and Zones.get are already migrated

[JamesFHeath]

EDIT: After poking around, I was able to deploy by adding "Regional Endpoint Admin" to my service account. Leaving for anyone else who hits this.

I've been deploying to app engine with the same service account from google cloud build with no issues, but today I got this error:

ERROR: (gcloud.app.deploy) PERMISSION_DENIED: Request to https://compute.googleapis.com/compute/v1/projects/projects/my-project/regions/us-central1?key failed, details: Regions.Get and Zones.Get are already migrated to regional or zonal esf-proxy/Mixer. See go/gce-location-apis-r13n. For more information please contact GCE API Frontend oncall https://oncall.corp.google.com/cloud-cluster-frontend-dev. This command is authenticated as app-deploy-prod@my-project.iam.gserviceaccount.com which is the active account specified by the [core/account] property.

I am able to deploy from my laptop with my owner account with all permissions, but I don't understand what the error message wants me to do. I tried to access the oncall website but I didn't have access.

Command in my cloud build:

gcloud app deploy app_prod.yaml

app_prod.yaml

# [START django_app]
runtime: custom
env: flex
entrypoint: gunicorn -b :$PORT app.wsgi --timeout=120
service: default

env_variables:
ENV: "PROD"
SETTINGS_NAME: "app-env-prod"

handlers:
- url: /static
static_dir: public/

beta_settings:
cloud_sql_instances: my-project:us-east1:app-prod

manual_scaling:
instances: 8

resources:
cpu: 4
memory_gb: 12
disk_size_gb: 10

readiness_check:
path: "/readiness-check"
check_interval_sec: 15
timeout_sec: 5
failure_threshold: 2
success_threshold: 2
app_start_timeout_sec: 120

[koconnor-ampion]

I'm afraid you are not the only one, James.  This just started for me too. I was even able to deploy earlier today, but no longer.

I also cannot follow the onCall link.

I will try your solution as well and let you know what happens.

Kevin

[koconnor-ampion]

A deploy subsequent to adding that permission to the service account failed as well.  However before that addition a deployment of another service that had failed the first time, did succeed...  Perhaps a network glitch of some type?

[JamesFHeath]

Hi Kevin,

I was able to deploy last night and this morning without any issues. The full permissions for my deployment service account are:

• App Engine Deployer
• App Engine Service Admin
• Cloud Build Editor
• Cloud Build Logging Service Agent
• Logs Writer
• Regional Endpoint Admin
• Secret Manager Secret Accessor
• Service Account User
• Service Usage Viewer
• Storage Object Admin

Hope this helps

James

[koconnor-ampion]

Thank you, James, just to keep you up-to-date if you are interested  

It has been a weird afternoon here....

Here was the sequence I tried relating to this (we have 3 services that are part of this app).
One of our devs started deployment

• scheduler succeeded
• web failed
• worker failed

The error was not helpful, same as what the you shared;
I tried a rerun from failed

• web succeeded
• worker failed

I added the suggested permission from your post

• worker failed

I removed the permission

• worker succeeded
• db migration failed (it turns out because there was no active worker)

I think there is something wonky at GCP

I sent a message to our customer engineer too.. Waiting for a response.

BTW, this uses a flex deployment of a custom image.  

Kevin

[JamesFHeath]

That sounds incredibly frustrating! When things don't fail consistently it makes you crazy. We're also using flex deployment with a docker image, but we get app engine to build the image for us.

I have not experienced the deployment issue again today, but we had massive service degradation at ~3pm EST. The 5 second health checks were timing out and causing our instances to restart.

[koconnor-ampion]

Oy, you said it. That last one, while the worker deployed, it was never started, so the next step when the deployment tries to communicate with it failed.  

I started it by hand and continued the deployment, then a Cloud Function failed in a way I'd never seen before... 

The request was aborted because there was no available instance. Additional troubleshooting documentation can be found at: https://cloud.google.com/functions/docs/troubleshooting#scalability

And refreshing this GoogleCloudCommunity page is failing with a bad response from the server when I first try to see one of your responses.... what a day!  

I'm glad thing sound like they are working more consistently for you!

Kevin

[koconnor-ampion]

Ah well, James, things are 'back to normal' today.  No explanation from Google in general or our customer engineer in particular.  And Google status was all green yesterday.  Spirits in the Wires, I guess.

Thanks for your responses yesterday.

Kevin