Command "gcloud --project=myproject app deploy app.yaml" no longer works

[mark4]

I am using the latest version of gcloud:

$ gcloud version
Google Cloud SDK 452.0.1
bq 2.0.98
bundled-python3-unix 3.9.17
core 2023.10.25
gsutil 5.27

For years now I've updated my website using this command: `gcloud --project=diffpdf app deploy app.yaml`.

But now it always fails:

$ gcloud --project=diffpdf app deploy app.yaml

Services to deploy:

descriptor:                  [/home/mark/qtrac/www-qtrac/app.yaml]
source:                      [/home/mark/qtrac/www-qtrac]
target project:              [diffpdf]
target service:              [default]
target version:              [20231031t084200]
target url:                  [https://diffpdf.appspot.com]
target service account:      [diffpdf@appspot.gserviceaccount.com]


Do you want to continue (Y/n)?  
Beginning deployment of service [default]...
╔════════════════════════════════════════════════════════════╗
╠═ Uploading 261 files to Google Cloud Storage              ═╣
╚════════════════════════════════════════════════════════════╝
File upload done.
Updating service [default]...failed.                                                        
ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build abfcabcb-2fd6-40e3-9520-51c0082323c8 status: FAILURE
An unexpected error occurred. Refer to build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/abfcabcb-2fd6-40e3-9520-51c0082323c8?project=469551747607
Full build logs: https://console.cloud.google.com/cloud-build/builds;region=us-central1/abfcabcb-2fd6-40e3-9520-51c0082323c8?project=469551747607

The failure is due to an "unexpected error" by a tool that gcloud runs on my behalf and over which I have no control.

This looks to me like a GAE bug, especially since this method of updating my website has worked for years. However, perhaps the problem is due to a warning that precedes the error (see below).

The log referred to in the message ends with warnings and then an error:

Step #1 - "pre-buildpack": Checking if image us.gcr.io/diffpdf/app-engine-tmp/build-cache/default/ttl-7d:latest exists
Step #1 - "pre-buildpack": Image us.gcr.io/diffpdf/app-engine-tmp/build-cache/default/ttl-7d:latest does not exist
Step #1 - "pre-buildpack": WARNING: Failed to reuse previous cache image; will not affect current build: image us.gcr.io/diffpdf/app-engine-tmp/build-cache/default/ttl-7d:latest does not exist
Finished Step #1 - "pre-buildpack"
Starting Step #2 - "build"
Step #2 - "build": Already have image (with digest): us.gcr.io/gae-runtimes/buildpacks/google-gae-22/python/builder:python_20231023_RC00
Step #2 - "build": ===> ANALYZING
Step #2 - "build": ERROR: failed to initialize analyzer: getting previous image: getting config file for image "us.gcr.io/diffpdf/app-engine-tmp/app/default/ttl-18h:latest": GET https://storage.googleapis.com/us.artifacts.diffpdf.appspot.com/containers/images/sha256:5e3af8d7b37415e98bc7c2070577c91f7fe05b106d886981de5b2ed01963c7b8?access_token=REDACTED: unexpected status code 404 Not Found: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: us.artifacts.diffpdf.appspot.com/containers/images/sha256:5e3af8d7b37415e98bc7c2070577c91f7fe05b106d886981de5b2ed01963c7b8</Details></Error>
Finished Step #2 - "build"
ERROR
ERROR: build step 2 "us.gcr.io/gae-runtimes/buildpacks/google-gae-22/python/builder:python_20231023_RC00" failed: step exited with non-zero status: 1

All of this is happening behind the scenes outside of my control. All done by gcloud. Clearly something has gone wrong but I have no idea what to do.

One thing I've noticed in the above is that there is a "NoSuchKey" error; I don't know what that means but perhaps it is a clue:

Step #2 - "build": ERROR: failed to initialize analyzer: getting previous image: getting config file for image "us.gcr.io/diffpdf/app-engine-tmp/app/default/ttl-18h:latest": GET https://storage.googleapis.com/us.artifacts.diffpdf.appspot.com/containers/images/sha256:5e3af8d7b37415e98bc7c2070577c91f7fe05b106d886981de5b2ed01963c7b8?access_token=REDACTED: unexpected status code 404 Not Found: <?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: us.artifacts.diffpdf.appspot.com/containers/images/sha256:5e3af8d7b37415e98bc7c2070577c91f7fe05b106d886981de5b2ed01963c7b8</Details></Error>

Surely someone with GAE expertise can see what the problem is and advise me on what I can do to fix it?

[robertcarlos]

Hi @mark4,

Welcome to Google Cloud Community!

For us to determine what causes this issue, we would like to ask a few questions:

• Which App Engine environment are you currently using? (Standard or Flex)
• What programming language are you using?
• Would it be possible to provide the app.yaml code?

Looking forward for your response.

[mark4]

I don't know which App Engine environment I'm using: how can I tell?

I am using Python 3. Here is my `app.yaml`:

runtime: python312

automatic_scaling:
  max_instances: 2
  min_idle_instances: automatic
  max_idle_instances: 1
  max_concurrent_requests: 10

handlers:
# Real
- url: /try
  script: auto

- url: /autotry
  script: auto

- url: /buy
  script: auto

- url: /getkey
  script: auto

- url: /
  static_files: static/index.html
  upload: static/index.html

- url: /
  static_dir: static

Note that originally the runtime I used (and which always worked) was `python37`.

However, even when I change the runtime in `app.yaml` back to `python37` I still get the same error message. So now I can't go back or forward and can't update my website at all.

[robertcarlos]

Hi @mark4,

As of the moment, the supported Python versions are Python 3.8-3.10 based on this documentation.

Python 3.12 is still and preview/infancy and some of its functions/features may not work as intended. You may check this App Engine standard environment for Python 3 release notes for more information.

You may also refer to this documentation on Python 3 Runtime Environment for more information with regards your setup.

Hope it helps.

[mark4]

As I mentioned I also tried with both python37 and python39 runtimes and both failed with exactly the same error. I have now tried yet again this time with the python310 runtime and it gives exactly the same "NoSuchKey" error. As for the docs, they are of no help whatsoever. I also tried searching for the error but nothing that came up appeared to be related.

I've use the command

gcloud --project=diffpdf app deploy app.yaml

for many years with no problems. But now it never works and always produces the same error no matter which Python 3 runtime I use.

[thinairin]

Mark,

Did you get this resolved?  I have the *exact* same problem and do not know how to resolve it.  Thanks!

(PII Removed by Staff)

[mark4 · Accepted solution]

Hi @th ,

The problem is due to a bug in Google's software. The workaround that worked for me was to run gcloud with the `--no-cache` option.

I hope this works for you.

[thinairin]

Hi Mark.

Your suggestion fixed the issue for me as well.   Thank you SO much!  Happy New Year!

(PII Removed by Staff)