gcloud app deploy fails with "Error Response: [9] Cloud build ... status: FAILURE An unexpected"

[jraf]

I am unable to deploy any services to App Engine Standard. Strangely, this is only a problem in one project and I am able to deploy the exact same app successfully in different projects.

The only error message provided is "Error Response: [9] Cloud build ... status: FAILURE An unexpected". The cloud build pipeline run by app engine goes through the three normal steps successfully, but then fails after the last step succeeds with no further logging. Messages "PUSH" and "DONE" are logged, and then nothing, but the build fails. Previous successful builds have ended with these same logs. There are no error messages such as the typical "npm install" failure. Methods like using "--no-cache" or different versions for the deployment result in the same error.

I've found plenty of discussions on app engine build failures, but haven't found any where deployment just fails with no further information like this. Does anyone have any information that might be helpful in troubleshooting this?

[jraf]

Another interesting aspect of this: all cloud build logs since this started are displayed differently than in the past. Recent builds have a log explorer similar to the Cloud Logging interface, whereas older builds have a simpler "text-only" display.

No configuration has been changed on the project, and I haven't found any configuration that would control this.

[jraf]

Deploying with debug verbosity provided an error result:

Result: {      
    "done": true,
    "error": {
        "code": 9,
        "message": "Cloud build id-redacted status: FAILURE\nAn unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;redacted\nFull build logs: https://console.cloud.google.com/cloud-build/builds;redacted"
    },
    "metadata": {
        "@type": "type.googleapis.com/google.appengine.v1.OperationMetadataV1",
        "endTime": "2024-05-16T20:33:49.260Z",
        "insertTime": "2024-05-16T20:33:17.037Z",
        "method": "google.appengine.v1.Versions.CreateVersion",
        "target": "apps/app-redacted/services/test-service/versions/20240516t153312",
        "user": "email-redacted"
    },
    "name": "apps/app-redacted/operations/id-redacted"
}

This is from deploying the Node "hello world" example at https://cloud.google.com/appengine/docs/standard/nodejs/building-app/writing-web-service

[NoCommandLine]

1. Is the problematic project tied to an active billing account? Lack of an active billing have resulted in some "funny" errors (the error message displayed have nothing to do with billing but were caused by it)
2. Can you post the contents of your package.json file and/or take a look at this stackoverflow response and see if it applies to you

    ......NoCommandLine ......
https://nocommandline.com

        Analytics & GUI for 
App Engine & Datastore Emulator

[jraf · Accepted solution]

This ended up being due to a retention policy set up on the App Engine staging bucket (i.e. "staging.app-id.appspot.com"). Removing the retention policy on the bucket fixed deployment.

The policy was put in place over two years ago, so it's surprising that after all that time it only now started causing problems. With the solution now known, it's logical that a retention policy could cause problems, but odd that it didn't until recently. There was also a complete lack of error messages to indicate it. (We've suggested to Google to include more helpful logging in this case.)

Setting a retention policy on the bucket is certainly not typical. We did so during SOC2 prep based on automated readiness tools and left it in place as it hadn't caused any problems in the past. For future readers: do not set a retention policy on App Engine storage buckets and remove them if you are running into this problem.