LinkedIn
Twitter
I'm trying to investigate a very strange issue on our GKE clusters where metrics for all of our Cronjobs mysteriously stopped working around 22:05 on the 20th April. As far as I have been able to determine so far:
- the configuration of the clusters wasn't changed at this time
- the configuration of the cronjobs also wasn't changed around this time
- the jobs do run on the correct schedule and the logs are still visible
An example of the CronJob deployment details screen for one job shows an abrupt end to the metrics and nothing in the four days since despite the job continuing to run on the same schedule:
It's the same for a completely unrelated job that runs in a different node pool in the same cluster:
Metrics for non Cronjob deployments on the same cluster still work as expected:
Alerting policies based on log counts for the Cronjobs also stop at the same time despite logs still being written by the containers:
As I mentioned the issue is affecting all CronJobs across multiple clusters with the only commonality being:
- they are in the same project
- the Kubernetes versions are the same (control plane: 1.24.10-gke.2300, node pools: 1.23.11-gke.300)
It's possible that the issue is being caused by the difference in version between the control plane and nodes, however I can't see anything in the logs that suggests the control plane was updated at that time. Any other suggestions welcome.
