Violated policy does not create alert

[charles9876]

I have a Google project where I execute and log from an Apps Script. I created a log monitoring policy to be notified if `severity>=ERROR` and in the preview during policy creation the filter lists the appropriate log entries I'd like to alert on. I also set the time interval to 5 minutes and auto resolution to 1 hour.  When a generate an error from the script a new error log entry in created but when I view the alerting summary page there are no alerts listed. 

I get the same behavior when I create an error notification from the error reporting summary. I configure a notification with an email notification channel with my email. I resolve all errors listed and then generate the error again from the Apps Script. The new error is reported but no email notification is sent. 

I've followed the trouble shooting tips from the built in Genie which includes:

1. Policy Configuration Issues:

• Snoozed or Disabled: Check if your policy is accidentally snoozed or disabled. A snoozed policy won't trigger alerts, and a disabled policy is inactive.
• Maximum Incident Limit: Policies have a limit on the number of incidents they can open simultaneously. If this limit is reached, new incidents won't be created.
• Resource State: If the resource being monitored is known to be disabled (e.g., a stopped VM), Monitoring might not create an incident. Look for a metadata.system_labels.state label on the resource.
• Incorrect Condition: Double-check the condition of your policy. Ensure it accurately reflects the threshold you want to trigger alerts.
• Label Extraction: If your policy extracts labels, make sure it's not extracting the timestamp label. This can prevent incident creation.

2. Notification Channel Problems:

• Misconfigured Channels: Verify that your notification channels (e.g., email, Slack) are correctly configured and active.
• Channel Limits: Log-based alerting policies have a daily notification limit of 20. If you exceed this limit, you might not receive notifications.

3. Log-Based Alerting Issues:

• Query Errors: If your policy is based on logs, ensure your query is correctly written and retrieves the expected log entries. Use Logs Explorer or the "Preview logs" button to validate your query.
• Insufficiently Restrictive Query: Your query might be too broad, causing alerts for more log entries than intended. Refine your query to be more specific.

4. Personalized Service Health:

• Unknown Relevance: Personalized Service Health won't send notifications if an incident has an "Unknown" relevance.

5. App Engine Logs:

• Mismatched Logs: If your policy is for App Engine logs, ensure the logs actually contain the events you're expecting.

Troubleshooting Steps:

1. Review Policy Details: Carefully examine your policy's configuration, including the condition, notification channels, and any label extraction.
2. Check Incident History: Go to the "Incidents" page in Cloud Monitoring and filter by your policy name. This will show you past and current incidents.
3. Validate Logs: If your policy is log-based, use Logs Explorer to verify your query and ensure it's retrieving the correct logs.
4. Test Notification Channels: Send a test notification to ensure your channels are working.
5. Consult Documentation: Refer to the official Google Cloud documentation for detailed information on troubleshooting alerting policies.

I'm assuming that because I can create a policy I have the right permissions in place. Any help would be appreciated!

[DamianS]

Hello @charles9876  ,Welcome on Google Cloud Community.

Would you be able to provide screenshots of your alerting policy configuration along with log-based metric config ? 

--
cheers,
Damian Sztankowski
LinkedIn medium.com Cloudskillsboost Sessionize Youtube

[charles9876]

Here's the json for the policy:

{
  "name": "projects/XXXXfoods/alertPolicies/748579912096065859",
  "displayName": "Apps Script Exception Error",
  "documentation": {
    "content": "XXXX Foods Error",
    "mimeType": "text/markdown"
  },
  "userLabels": {},
  "conditions": [
    {
      "name": "projects/XXXXfoods/alertPolicies/748579912096065859/conditions/748579912096063760",
      "displayName": "Log match condition",
      "conditionMatchedLog": {
        "filter": "severity>=ERROR",
        "labelExtractors": {
          "XXXXFoods": "EXTRACT(jsonPayload.message)"
        }
      }
    }
  ],
  "alertStrategy": {
    "notificationRateLimit": {
      "period": "3600s"
    },
    "autoClose": "3600s",
    "notificationPrompts": [
      "OPENED"
    ]
  },
  "combiner": "OR",
  "enabled": true,
  "notificationChannels": [
    "projects/XXXXfoods/notificationChannels/16995925881456324820"
  ],
  "creationRecord": {
    "mutateTime": "2024-12-09T14:36:40.638419280Z",
    "mutatedBy": "XXXXfoods@XXXXXXX.com"
  },
  "mutationRecord": {
    "mutateTime": "2024-12-11T13:39:16.768479184Z",
    "mutatedBy": "XXXXfoods@XXXXXXX.com"
  },
  "severity": "ERROR"
}

  I've obfuscated some of the data with `XXXX` and here are a couple screen shot of the policy

Thanks for taking a look at this!