GCP Cloud Observability: Monitoring (Managed Prometheus): How to use user labels

[blewis]

I am trying to use metrics collected automatically by GCP Kubernetes (GKE) using the Managed Prometheus and query them via PromQL in GCP Cloud Observability: Monitoring.

I have certain node pools for different use cases and I would like to create different alerts for node CPU and RAM usage depending on the node pool to which the node belongs.

I have tried the following PromQL query, but it yields no results:
```

avg_over_time(kubernetes_io:node_cpu_allocatable_utilization{monitored_resource="k8s_node",project_id="hirundo-mvp-dev",location="us-central1",cluster_name="gke-mvp-dev-cluster", metadata_user_cloud_google_com_gke_nodepool="mvp-dev-pool"}[${__interval}])

```

The issue seems to be with the `metadata_user_cloud_google_com_gke_nodepool` label. I do not understand how else I can transform the `cloud.google.com/gke-nodepool` label value shown in the Builder UI as one of the "User Metadata Labels".

Furthermore, the example from the documentation (https://cloud.google.com/stackdriver/docs/managed-prometheus/promql#metadata-labels)

```

compute_googleapis_com:instance_cpu_utilization{monitored_resource="gce_instance",metadata_user_env="prod"}

```

Does not seem to work for me whatsoever.

What am I missing?

[kamblem]

Hi@ @blewis 

To assist us in conducting thorough investigation, we kindly request your cooperation in providing the following information regarding the reported issue:

1. Has this scenario ever worked as expected in the past?
2. It would be greatly helpful if you could attach screenshots of the output related to this issue.

Thank you 

[blewis]

Hi kamblem,

No, it has not worked in the past. I don’t know exactly what help a
screenshot would be. The results table and graph are empty 🙈.

[blewis]

@kamblem  Do you have any ideas why this does not work?