Command "gcloud --project=myproject app deploy app.yaml" no longer works

[mark4]

I am using the latest version of gcloud:

$ gcloud version
Google Cloud SDK 452.0.1
bq 2.0.98
bundled-python3-unix 3.9.17
core 2023.10.25
gsutil 5.27

For years now I've updated my website using this command: `gcloud --project=diffpdf app deploy app.yaml`.

But now it always fails:

$ gcloud --project=diffpdf app deploy app.yaml

Services to deploy:

descriptor:                  [/home/mark/qtrac/www-qtrac/app.yaml]
source:                      [/home/mark/qtrac/www-qtrac]
target project:              [diffpdf]
target service:              [default]
target version:              [20231031t084200]
target url:                  [https://diffpdf.appspot.com]
target service account:      [diffpdf@appspot.gserviceaccount.com]


Do you want to continue (Y/n)?  
Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 261 files to Google Cloud Storage              ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...failed.                                                        
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build abfcabcb-2fd6-40e3-9520-51c0082323c8 status: FAILURE
An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/abfcabcb-2fd6-40e3-9520-51c0082323c8?project=469551747607
Full build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/abfcabcb-2fd6-40e3-9520-51c0082323c8?project=469551747607

The failure is due to an "unexpected error" by a tool that gcloud runs on my behalf and over which I have no control.

This looks to me like a GAE bug, especially since this method of updating my website has worked for years. However, perhaps the problem is due to a warning that precedes the error (see below).

The log referred to in the message ends with warnings and then an error:

Step #1 - "pre-buildpack": Checking if image us.gcr.io/diffpdf/app-engine-tmp/build-cache/default/ttl-7d:latest exists
Step #1 - "pre-buildpack": Image us.gcr.io/diffpdf/app-engine-tmp/build-cache/default/ttl-7d:latest does not exist
Step #1 - "pre-buildpack": WARNING: Failed to reuse previous cache image; will not affect current build: image us.gcr.io/diffpdf/app-engine-tmp/build-cache/default/ttl-7d:latest does not exist
Finished Step #1 - "pre-buildpack"
Starting Step #2 - "build"
Step #2 - "build": Already have image (with digest): us.gcr.io/gae-runtimes/buildpacks/google-gae-22/python/builder:python_20231023_RC00
Step #2 - "build": ===> ANALYZING
Step #2 - "build": ERROR: failed to initialize analyzer: getting previous image: getting config file for image "us.gcr.io/diffpdf/app-engine-tmp/app/default/ttl-18h:latest": GET https://storage.googleapis.com/us.artifacts.diffpdf.appspot.com/containers/images/sha256:5e3af8d7b37415e98bc7c2070577c91f7fe05b106d886981de5b2ed01963c7b8?access_token=REDACTED: unexpected status code 404 Not Found: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: us.artifacts.diffpdf.appspot.com/containers/images/sha256:5e3af8d7b37415e98bc7c2070577c91f7fe05b106d886981de5b2ed01963c7b8</Details></Error>
Finished Step #2 - "build"
ERROR
ERROR: build step 2 "us.gcr.io/gae-runtimes/buildpacks/google-gae-22/python/builder:python_20231023_RC00" failed: step exited with non-zero status: 1

All of this is happening behind the scenes outside of my control. All done by gcloud. Clearly something has gone wrong but I have no idea what to do.

One thing I've noticed in the above is that there is a "NoSuchKey" error; I don't know what that means but perhaps it is a clue:

Step #2 - "build": ERROR: failed to initialize analyzer: getting previous image: getting config file for image "us.gcr.io/diffpdf/app-engine-tmp/app/default/ttl-18h:latest": GET https://storage.googleapis.com/us.artifacts.diffpdf.appspot.com/containers/images/sha256:5e3af8d7b37415e98bc7c2070577c91f7fe05b106d886981de5b2ed01963c7b8?access_token=REDACTED: unexpected status code 404 Not Found: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: us.artifacts.diffpdf.appspot.com/containers/images/sha256:5e3af8d7b37415e98bc7c2070577c91f7fe05b106d886981de5b2ed01963c7b8</Details></Error>

Surely someone with GAE expertise can see what the problem is and advise me on what I can do to fix it?