Unable to create MongoDB alerting policies with Terraform

[LPrackiNC]

Hi,

In Cloud Monitoring, I want to create alerting policies for a MongoDB database hosted on a Compute Engine instance. While it gets created successfully via GC console, it fails to get created with Terraform.

Here's what I'm getting when applying the configuration: 

Error creating AlertPolicy: googleapi: Error 404: Cannot find metric(s) that match type = "workload.googleapis.com/mongodb.collection.count". If a metric was created recently, it could take up to 10 minutes to become available. Please try again soon.

For the sake of this post, let's take the following metric as an example: 

workload.googleapis.com/mongodb.collection.count

Alerting policy created via GUI:

{
  "displayName": "",
  "userLabels": {},
  "conditions": [
    {
      "displayName": "VM Instance - workload/mongodb.collection.count",
      "conditionThreshold": {
        "filter": "resource.type = \"gce_instance\" AND metric.type = \"workload.googleapis.com/mongodb.collection.count\"",
        "aggregations": [
          {
            "alignmentPeriod": "600s",
            "crossSeriesReducer": "REDUCE_NONE",
            "perSeriesAligner": "ALIGN_MEAN"
          }
        ],
        "comparison": "COMPARISON_GT",
        "duration": "0s",
        "trigger": {
          "count": 1
        }
      }
    }
  ],
  "alertStrategy": {
    "autoClose": "604800s"
  },
  "combiner": "OR",
  "enabled": true,
  "notificationChannels": []
}

Alerting policy created in code:

    mongodb_collections_critical = {
      name     = "CRITICAL: VM Instance - MongoDB collection count > #threshold#"
      combiner = "OR"

      thresholds = {
        threshold_value = 30
      }

      conditions = {
        condition_1 = {
          name = "CRITICAL: VM Instance - MongoDB collection count > #threshold#"
          condition_threshold = {
            comparison = "COMPARISON_GT"
            duration   = "0s"
            filter     = "resource.type = \"gce_instance\" AND metric.type = \"workload.googleapis.com/mongodb.collection.count\""

            aggregations = {
              aggr1 = {
                alignment_period   = "600s"
                per_series_aligner = "ALIGN_MEAN"
              }
            }

            trigger_count = 1
          }
        }
      }

      content = <<EOT
             ALERT DOCUMENTATION
            EOT

      pd_integration_level = "critical"
    }

Full error:

│ Error: Error creating AlertPolicy: googleapi: Error 404: Cannot find metric(s) that match type = "workload.googleapis.com/mongodb.collection.count". If a metric was created recently, it could take up to 10 minutes to become available. Please try again soon.
│ 
│   with module.monitor["XXX"].google_monitoring_alert_policy.monitor,
│   on monitor_wrapper/monitor.tf line 1, in resource "google_monitoring_alert_policy" "monitor":
│    1: resource "google_monitoring_alert_policy" "monitor" {

Here's what I tried already:

1. Ensured that the metric data is being produced (to rule out the second part of the error message). Waited more than 10 minutes to trigger the config apply action.
2. Verified the metric type names and syntax in the code. They match the alert policy code from Cloud Monitoring GUI. 
3. Verified Ops Agent's configuration according to Google's documentation.

Thanks in advance for help!