Alert policies: ${metric.label.value} null for promQL policies

[Christophe31fr]

I have several promql alert policies.

Sample request:

round(histogram_quantile(0.99, increase(run_googleapis_com:container_cpu_utilizations_bucket{monitored_resource="cloud_run_revision"}[1m])) * 100 , 0.01) > 90

The result returned is done as expected ( a list of fields + "Value" variable well defined)

On the alert policy, I have a documentation text field containing:

The cloud run CPU utilization for [${resource.label.revision_name}] has exceeded 90% for over 5 minutes. Current CPU usage is [${metric.label.value}%]

When the alert trigger is reach, the alert is created (incident creation OK).

The variable ${resource.label.revision_name} is well defined with the field value for the could run revision name as expected but the field concerning ${metric.label.value} is replaced by "Null" rather thant the promql  value that trigger the alert...

The cloud run CPU utilization for [my_ressource_name] has exceeded 90% for over 5 minutes. Current CPU usage is [(Null)%]

Several months ago, I designed about 20 promql policies like that (cpu / ram / network parameters etc...) , the variable ${metric.label.value} was always well defined with the promql value.

Since several days or weeks (i don't know exactly), the returned value for ${metric.label.value} is always "Null" even if the data is well returned by the promql request.

Can you tell me if Google has updated the way they manage the Value field for promql queries ? The GCP documentation about variables confirms that ${metric.label.value} stays the solution to put the promql "Value" field in the documentation.

Many thx for your help, I'm stuck in this since several days ! 🙂

Chris

PromQL request:

Returned data:

Documentation field defined:

[-Rhett]

Hi @Christophe31fr,

Welcome to the Google Cloud Community!

Thanks for outlining the issue in detail. Given that ${metric.label.value} is correctly configured but now returns Null, this may be due to missing time series in your PromQL expression output, rather than a problem with variable substitution itself.

As explained in this PromLabs blog post, Prometheus can return an empty result if the label combination referenced hasn't been emitted yet (e.g., if the Cloud Run revision hasn’t produced any metric samples in the alerting window). This commonly happens when a metric with custom labels hasn’t been initialized or when the associated target has not emitted data for that label combination yet.

To confirm that this is the cause, you can follow the Managed Prometheus troubleshooting steps to check if data is missing or if the query returns no series. If the metric is indeed missing, one workaround is to add a PromQL expression that provides a fallback value such as or on() vector(0), which converts null as zero.

If this seems like a regression or an undocumented behavior change, I’d recommend filing a report either via the GCP issue tracker or by opening a GitHub issue on the Prometheus Engine repo. Include steps to replicate, configuration settings, screenshots, and other details of the issue to help us investigate. While there isn’t a specific timeframe for resolution, once we've fixed an issue in production we'll indicate this and then update and close the bug.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.