Google App Engine: manual scaling with one instance suddenly created multiple instances

[Maccer]

I have a Google App Engine application with

runtime: java17
instance_class: B4
[...]
manual_scaling:
  instances: 1

It has been running as set up with 1 instance for years. Suddenly this night (2024-03-26 around 3:15 UTC), without me doing any change, it scaled up to 4 instances, causing various issues.
How is this even possible, and is there anything I can do to stop it?

(I also posted this question on StackOverflow , but I figured I might get better help here)

Suddenly scaling from 1 to 4 instances

[knet]

Cloud Run optimizes for meeting your traffic needs without downtime. To help route requests quickly, our infrastructure is decentralized, which sometimes results in the number of instances temporarily exceeding max-instances. 

Did you have any increase in the number of requests to your instance during this time? Has the service scaled back down since?

[Maccer]

Hi! No, unfortunately, at the time of writing – around 36 hours later – the service is still stuck at 4 instances, and I haven't found a way of getting it back to one instance.
I have tried to
- stop and restart the service,
- re-deploy the same version
- deploy a new version
New deployments are set to stop and replace the old version, but the new version immediately starts up four instances (see image).

Our service is not used that frequently and does not have high loads. The ramping up happened during early morning Swedish time when there is even less activity. I can't see anything unusual from the logs that could explain it.

_ah/start and _ah/stop for prod service


I also noticed that the our development version of the same app engine service also ramped up to four instances around the same time, starting around 15-20 minutes earlier and doing it with slightly longer pauses in between (see image below). It is also still running on four instances.
Development version ramp-up to 4 instances

[Maccer]

Update: all good for now, today (2024-03-27) around 18:30-19:00 CET the services finally stepwise ramped down to one instance.
So they used four instances for 38 hours straight without any good reason (at least none I can find). Hopefully this was some rare bug from Googles side, as this has to my knowledge never happened before during the 4+ years we have had the app engine services deployed.

Prod rampdown:

Dev rampdown:

[knet · Accepted solution]

Thank you for the additional details! I checked and this was an issue on our end that is being/has been mitigated.