Disk utilization Monitoring Policy false positives

[hyperrat]

Hi

I have a policy in a number of projects that checks for physical disks that are over 90% used. Loop disks are filtered out. 

On a weekly basis, some of the projects get open Incidents created for what appears to be no good reason. Some disks are pretty static like the EFI partition and are no where near (e.g. 5% of) the 90% threshold. These incidents won't automatically close for many days. I don't think that any of these are getting to the 90% threshold. The closest is the root partition which is normally around 50% but data is mounted on a second disk.

Now the strange part... if I fill a disk deliberately to test the policy, it triggers fine, opens an incident and when I clean up the disk it automatically closes the incident after a while. There is only one disk utilization policy in the project.

Could there be a misconfiguration in my policy ?

Here is the one disk utilization policy JSON:

{
  "name": "projects/project-a/alertPolicies/123456789",
  "displayName": "project-a low-server-disk-alert",
  "documentation": {
    "content": "A disk of a server project-a has low free disk space.",
    "mimeType": "text/markdown"
  },
  "userLabels": {},
  "conditions": [
    {
      "name": "projects/project-a/alertPolicies/123456789/conditions/987654321",
      "displayName": "VM Instance - disk utilization",
      "conditionThreshold": {
        "aggregations": [
          {
            "alignmentPeriod": "900s",
            "perSeriesAligner": "ALIGN_MEAN"
          }
        ],
        "comparison": "COMPARISON_GT",
        "duration": "0s",
        "filter": "resource.type = \"gce_instance\" AND metric.type = \"agent.googleapis.com/disk/percent_used\" AND (metric.labels.device != starts_with(\"/dev/loop\") AND metric.labels.state = \"used\")",
        "thresholdValue": 90,
        "trigger": {
          "count": 1
        }
      }
    }
  ],
  "alertStrategy": {
    "notificationPrompts": [
      "OPENED"
    ]
  },
  "combiner": "OR",
  "enabled": true,
  "notificationChannels": [
    "projects/project-a/notificationChannels/5647382910"
  ],
  "creationRecord": {
    "mutateTime": "2024-12-02T08:55:57.129430088Z",
    "mutatedBy": "someadmin.iam.gserviceaccount.com"
  },
  "mutationRecord": {
    "mutateTime": "2025-04-22T14:54:57.919785586Z",
    "mutatedBy": "someotheradmin"
  },
  "severity": "WARNING"
}

Thanks