Older revisions of Cloud Run service keep getting traffic even if the latest has 100% allocated

[Mike99]

Good afternoon,

Previous revisions of a Cloud Run service on our end keep on receiving traffic, while the latest revision is set at "100% (to latest)". In essence we do the following in our CI:

gcloud run services update-traffic <name of instance> --to-latest --region=<region>

(to ensure manual mode is turned off - cause it could be that someone went back a revision manually). We do this because of this issue. After that we do:

gcloud run deploy <name of instance>

Previous revisions now have 0% traffic in the UI, but still have this check mark that they are 'healthy and serving traffic'. When testing the service, traffic is still sent to these "green" previous revisions as well.

The only way that I managed to solve this is to amend the YAML and remove all the non-relevant revisions from the traffic block, but that doesn't quite fit our CLI-based approach.

Would anyone have an answer to this?

[knet]

Hi Mike,

Are you saying that if you send requests to the default Cloud Run URL, they sometimes get served by revisions that show as serving 0% traffic? That's unexpected. 

We can take a closer look if you can reply privately to me with your project ID. 

[tphillips05]

Has there been any resolution or investigation into this? Currently happening to us and is causing issues as old revisions are randomly serving traffic

[Mike99]

Hi tphilips05 ; this did the trick for me:

- not tag your revisions / deployments (i.e. no gcloud run deploy --tag). Do note: tags randomly getting traffic should not be happening; GCP is investigating this behavior.

- do 'gcloud run services update-traffic <instance-name> --to-latest --region=<region>' before you deploy (to get out of  'manual' mode; for instance if someone went back a revision manually and you are now doing your usual automated / CI deployment)

[tphillips05]

Hey Mike99,

Just confirming that when we removed our tags from the revisions that had them (we didn't fully delete the whole deploy), the traffic was immediately fixed and is now routing through our revision that we have set for 100% traffic. Thanks!