Observed value for MQL Incident is not published through Webhook notification channel

[Friedrich]

I have the following setup:

• Many custom metrics in the google cloud platform
• Alerting policies reacting to these metrics
  • Some use UI components for creation
  • and some MQL
• A webhook notification channel to forward the alerts to OpsGenie

The Alert Policies created using UI components contain an "observed_value" field in the webhook call, as documented here https://cloud.google.com/monitoring/support/notification-options#webhooks

The Alert Policies configured by MQL are missing this value.

Problem: I require access to the observed value, as it is one of the most important facts about the alert. 

What do I have to do to include it also for MQL Alert Policies?

[reinc]

Hi @Friedrich, 

Welcome to Google Cloud Community!

If you want to use the webhook payload to include the "observed_value" from MQL, you'll need a script (Python, Node.js, or another language).

Script Functionality:

• Receive Webhook Payload: Capture the incoming webhook notification data from Google Cloud Monitoring.
• Extract Data: Parse the payload and retrieve information like resource name, timestamp, and alert name.
• Retrieve Stored MQL Value: Access the metric value you stored earlier (see explanation in previous response).
• Modify Payload: Add the extracted metric value to the webhook payload with a new key ("observed_value" or similar).
• Forward to OpsGenie: Send the modified payload to the OpsGenie webhook URL.

Example (Python)

import requests
import json

# Retrieve the "observed_value" from the MQL data source 
observed_value = get_observed_value(alert_name, resource_name, timestamp) 

# Modify the webhook payload
payload["observed_value"] = observed_value

# Send the modified payload to OpsGenie
requests.post("https://your-opsgenie-webhook-url", json=payload)

Important Update: Upcoming Changes to Cloud Monitoring Query Language (MQL) that may affect how you manage your dashboards and alerts.

Here's what you need to know:

Starting October 22, 2024:

• MQL is no longer recommended: We encourage you to transition to PromQL, the open-source standard for querying time series data. PromQL offers similar capabilities to MQL with a broader user base and more community support.
• Certain MQL usability features will be turned off: You may experience limitations with some of the MQL features in the Google Cloud Console.
  
  

On July 22, 2025:

• MQL will be unavailable for new dashboards and alerts in the Google Cloud Console: You will need to use PromQL for creating new dashboards and alerts within the console.
• Google Cloud customer support for MQL will end: You will no longer be able to receive support for MQL-related issues from Google Cloud.

Important: Your existing MQL dashboards and alerts will continue to function after July 22, 2025. You can also continue to create new MQL dashboards and alerts using the Cloud Monitoring API. For more information, please see the deprecation note for MQL.

If you have any questions or need assistance, please reach out to our support team.


I hope the above information is helpful.

[Friedrich]

Hi @reinc,

thank you for your response and thorough explanation.
So to make it work, I would need another Service/Cloud function in between. So the setup would be Monitoring -> Python Cloud Function -> OpsGenie

This solution sounds rather complicated for something this simple, but should work. Thank you! Do you see any way to include the observed value in the original request, as it would be, if the alert condition was defined via the UI and not MQL? 

Are there any changes to be expected in the future or does this change with PromQL?

Thanks again