GKE Security Posture issue with metrics

[carstenthiel-te]

We are experiencing issues with the image-package-extractor-cleanup cronjob created by the GKE Security Posture on a Kubernetes Cluster running v1.25.8-gke.1000.
This cluster has been consecutively upgraded over a longer time and we can not reproduce the issue on a brand new installation with the same Kubernetes version.

"Failed to export metrics to Cloud Monitoring"
"rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist)."

with stacktrace

google3/cloud/kubernetes/metrics/common/exporter/exporter.(*exporter).exportBuffer
  cloud/kubernetes/metrics/common/exporter/export.go:233
google3/cloud/kubernetes/metrics/common/exporter/exporter.(*exporter).Flush
  cloud/kubernetes/metrics/common/exporter/export.go:179
google3/cloud/kubernetes/metrics/common/exporter/exporter.(*exporter).Shutdown
  cloud/kubernetes/metrics/common/exporter/export.go:191
main.main.func2
  cloud/kubernetes/distro/containers/image_package_extractor/er_cleanup/main.go:83
main.main
  cloud/kubernetes/distro/containers/image_package_extractor/er_cleanup/main.go:95
runtime.main
  third_party/go/gc/src/runtime/proc.go:250

While the job is referring to the ServiceAccount pkgextract-cleanup-service, we can't see any difference in its configuration between the old and new clusters.
Does anyone have a better understanding of how the authentication against Google Monitoring is realised?

[ShadowK · Accepted solution]

I'm having the same issue. Did you able to resolve it?

[Willbin]

Hello @carstenthiel-te,

Welcome to Google Cloud Community!

Based on the error you posted, this error occurs if the permissions for the Ops Agent are not properly configured.

"rpc error: code = PermissionDenied desc = Permission monitoring.timeSeries.create denied (or the resource may not exist)."

 You may fix this error by enabling the Monitoring API and set the Monitoring Mteric Writer role.

[carstenthiel-te]

Thanks @Willbin ,

the metrics API is enabled. Our issue is that we are struggeling to understand which service account the GKE Security Posture is using to communicate with the API and why it's not being configured for permissions automatically when we enable it.

Carsten

[ShadowK · Accepted solution]

I'm having the same issue. Did you able to resolve it?

[garisingh · Accepted solution]

Unfortunately, this is a known issue.  The fix has been rolled out for 1.27 and later.  We are still waiting on an update for the timeline to backport to other releases.  Will update when I know more.

[ZXpower]

Is there any updates when this could be available on GKE v1.25.x? Did a recent update from v1.24.x to v1.25.x and immediately got this error.... 🙄