Greetings all. I am currently experiencing a massive slowdown in the amount of time it takes for a created job to start. This manifested in my hourly pipelines suddenly doubling or tripling in how long they take to run, and took an embarrassingly long amount of time to identify what was going on. I will be submitting a support ticket shortly, but am waiting for some permissions changes so I can actually enable support! I am hopeful someone sees this in the mean time, and the fact that there is no official issue happening is really frustrating as it sent me on a wild goose chase.
One example, at 9:00:04 AM Eastern we created a job (we create jobs using GCP workflows). It did not start until May 9, 2025, 9:14:19 AM. I am using the cloud console's 'Creation time', and 'Start time' values.
I was able to pull some logs and get a good idea of how long certain jobs are delayed vs about 10 days ago:
10 days ago: Most jobs were starting around 1.5 to 2 minutes after the top of the hour, with low variability.
Today: Jobs are starting later on average (~4 min 15 sec), and variability is much higher—some jobs even took up to ~13 min 40 sec to start.
The median time today (~2 min 41 sec) is already higher than the 75th percentile from 10 days ago, confirming a generalized delay in job starts.
Though we can modify our workflows a bit to have fewer job starts, a 14m delay is absolutely crippling.
LinkedIn
Twitter
