Cloud run deploy revision launch stage error despite no change to deployment config

[jaysouth]

Hello,

I'm wondering if anyone has run into this scenario or might know where start troubleshooting this. We have an existing Cloud Run service that is currently configured to the following launch-stage:

However, when we execute a job in cloud build that runs a "gcloud run deploy" command to create a new revision of this run service, we're now suddenly getting the following error:

"cloud-run-deploy": ERROR: (gcloud.run.deploy) FAILED_PRECONDITION: The feature 'Direct VPC' is not supported in the declared launch stage on resource service-app. The launch stage annotation should be specified at least as BETA. Please visit https://cloud.google.com/run/docs/troubleshooting#launch-stage-validation for in-depth troubleshooting documentation."

When you follow the URL in the error, it takes you to a troubleshooting page that recommends you have the config for launch-stage set to "BETA" which we already have set accurately in the above image. This build job was functioning perfectly fine this week and we've made no changes to it, but all of a sudden seeing this on all builds. It has been configured to use `Direct VPC` since deployment months ago so that's not a new config either. Has anyone else run into this recently?

[Klaitos]

Same issue here .. since 17 of April 17h00 UTC.

Since the issue appears, there is a permanent diff in my terraform plan.

Also the gcloud run services describe command return either null either beta flag on the `.metadata.annotations["run.googleapis.com/launch-stage"` annotation when you run it several times in a row..

 

[hari-stainless]

Also experiencing this issue. The deploy failures are intermittent, but the majority of attempted deploys appear to run into this, across a number of services. 

[jaysouth]

Same general fail rate for us as well.

[hydeta]

Experiencing the exact same thing started today and appears completely transient.  I can deploy services 1 by 1 but doing them all at once always leads to some sort of failure

[eerootsus]

Seeing the same issue. Similarly to OP, configs for run services have not changed in a while, but deploys mostly fail. And then, out of the blue, it works.

[simon3]

Seeing same behavior here. Deploys using gcloud run deploy are failing intermittently when Direct VPC is being used because of 'missing' launch-stage=BETA. Also using describe, this annotation is sometimes shown and sometimes it is <null>

❯ gcloud run services describe ${cloudrun} --format json| jq -r '.metadata.annotations["run.googleapis.com/launch-stage"]'
null
❯ gcloud run services describe ${cloudrun} --format json| jq -r '.metadata.annotations["run.googleapis.com/launch-stage"]'
BETA
❯ gcloud run services describe ${cloudrun} --format json| jq -r '.metadata.annotations["run.googleapis.com/launch-stage"]'
null
❯ gcloud run services describe ${cloudrun} --format json| jq -r '.metadata.annotations["run.googleapis.com/launch-stage"]'
BETA
❯ gcloud run services describe ${cloudrun} --format json| jq -r '.metadata.annotations["run.googleapis.com/launch-stage"]'
null

[simon3]

Created bug report here: https://issuetracker.google.com/issues/335695528

(hit the +1 button there 'I am impacted'. Maybe it helps to get more attention)

[stefanTaku · Accepted solution]

Our pipelines were also hitting this issue, but I was able to work around it by adding the `beta` flag to our `gcloud` commands.

I ran our deploy pipeline 4 times, and each run has 4 `services update` calls that all succeeded, so it seems unlikely that I just got lucky on the `launch-stage = BETA/null` coin-flip

- gcloud run services update
+ gcloud beta run services update

[jaysouth]

This appears to be working for us so far as well in my testing. Thanks for the recommendation!

[yanweiguo]

Hi,

Direct VPC feature was still a public preview feature when the issues happened to you. Make sure to set launch stage to beta when creating or updating a Cloud Run resource with public preview feature. Using `gcloud beta` will do the trick.

What has changed? The launch stage requirement for Direct VPC feature was being removed for GA launch.

Why was it working before? `gcloud deploy` does a get request first then sends an update request with modification based on the get result, if the resource with the same name exists. Before the change, Cloud Run API ensures the resource was created with launch stage beta so gcloud would always get a resource with Direct VPC enabled and launch stage set to beta as well.

Why was it not working intermittently? The change mentioned above was gradually rolled out. During the rollout, if the resource was created/updated by the API server with the change, the launch stage setting was dropped by API server since there's no public preview feature on the resource. `gcloud deploy` got the resource without launch stage setting and modified it then sent the update request. If the update request was routed to an API serve with the old version, which still required launch stage, the FAILED_PRECONDITION was thrown.

The change has been fully rolled out and the issue should be gone. As a reminder, always set launch stage if any preview feature is used.

Thanks.