Google cloud app engine flex environment deploymen...

Blair_Holmes · 08-14-2022 08:43 AM

I've done a lot of researching on this error and found some posts here and on github but none of the resolutions they suggest are working for me. I'm receiving a pretty ambiguous error when trying to deploy my application to a flex environment on google cloud service app engine.

It works fine when I deploy to a standard app service in the app engine, but when I try to deploy to a flex service I receive this error:

ERROR: (gcloud.app.deploy) Error Response: [13] An internal error occurred during deployment.

I know it is nothing wrong with my application code nor my app.yaml because when I do the same deployment on another google cloud account, the service deploys fully, publishes, and I'm able to use the API. It is definitely something failing with my google cloud setup for this specific account. I'm pretty confident that I have all of the apis enabled and correct service account roles applied to this account that is not working, but obviously something is different.

One suggestion I read was to apply the role "App Engine flexible environment Service Agent" to my app engine default service account. That role did not exist on my other account that is working, but I tried anyway - no joy.

Here are my roles assigned to my service accounts on both my working google cloud account, and the non-working one.

Here is a comparison between my enabled APIs on both accounts:

I set both of these cloud accounts up and I do not recall doing anything differently for the one that is working than I did for the one that is not.

I've tried deploying through my cloud build trigger and manually running gcloud app deploy.

I also tried the latter with --verbosity=debug which gives me only a little more information about the failure:

"done": true,
"error": {
    "code": 13,
    "message": "An internal error occurred during deployment."
},
"metadata": {
    "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
    "endTime": "2022-08-14T00:25:57.780Z",
    "insertTime": "2022-08-14T00:25:54.623Z",
    "method": "google.appengine.v1.Versions.CreateVersion",
    "target": "apps/my-app/services/your-app/versions/20220814t002258",
    "user": "me@you.com"
},

This is a resource I've found similar to my issue: https://github.com/GoogleCloudPlatform/nodejs-getting-started/issues/153

I've tried every suggestion there and still having the problem.

I'm 100% certain it's not the code itself nor the app.yaml because it works fine on my other account. Any other suggestions of places to look would be greatly appreciated.

TIA

comaro

Hi, Blair,

You mentioned the same app is working fine with App Engine Standard, but it is not with App Engine Flex.

Also, it is working fine with, let’s say, account A but not with account B.

I would like to ask you:
1) Does it work correctly when trying to deploy to App Engine Standard with account A?
2) Does it work correctly when trying to deploy to App Engine Flex with account A?
3) Does it work correctly when trying to deploy to App Engine Standard with account B?
4) Does it work correctly when trying to deploy to App Engine Flex with account B?

Blair_Holmes

if we're calling account A the working account:

1) Yes
2) Yes
3) Yes
4) no

everything on my working account works fine for the application in question. A few things I've done since making my post:
1) verified billing is ok on non-working account
2) tried deploying in incognito
3) tried deploying using the google cloud sdk console
4) made both account APIs identical
5) made service accounts and permissions identical on both accounts
6) checked location matches on both accounts (central-us)

For now I've given up and am just running the app in the standard environment.

comaro

The issue with deployment you are facing is because of a known rollout issue from App Engine Flex’s end. The Product team is actively working on fixing this issue, and we should see an update on the fix during Monday, Pacific Business hours. Please let us know if it works.

Blair_Holmes

great, I shall try it again on Monday afternoon and report back my results. Thank you.

Blair_Holmes

unfortunately the issue still persists when trying to deploy through the cloud sdk console.

Blair_Holmes

I just tried a deployment through the release pipeline - it failed as well. Same error as the original.

comaro

Hi, Blair,

Could you please run `gcloud config set app/trigger_build_server_side false` and re-try?

It should temporarily fix the problem while the App Engine Flex dev team digs into this.

Blair_Holmes

comaro, I just replied to your DM - running this config setting change prior to the deployment did NOT fix the issue, same error as before.

comaro

Hi, Blair,

I discussed your concerns with a colleague. Since the mentioned workaround did not work, we are afraid this is a different issue than the App Engine team has recently fixed.

Your project would need to be inspected by support.

This can be done over a [support case](https://cloud.google.com/support-hub) under a valid business justification.

Google cloud app engine flex environment deployment failure (NestJS) - Error code 13 (Internal Error