Attempting to configure a forwarding rule, seeing ...

dpkirchner · 12-15-2022 07:17 AM

I'm trying to configure a global forwarding rule via Terraform, however I'm seeing this error:

│ Error: Error creating ForwardingRule: operation received error: error code "INVALID_USAGE", message: Service not found

I've included bits from the trace output and Terraform config at the end of this post.

Some background: I'm adding a new Cloud CDN instance that points to a backend bucket and naturally I need to apply a few rules: rewriting the path, enabling CORS, and removing a few dozen x-goog-* headers. The Cloud CDN instance, URL map, HTTPS proxy, and managed SSL cert resources all seem to be configured correctly and successfully. The only piece missing is this forwarding rule and the result is that there's no frontend configured for the global LB.

According to the API docs[0] there is a "backendService" parameter you can send when creating forwarding rules, however it's only to be used for "Internal TCP/UDP Load Balancing and Network Load Balancing" and "must be omitted for all other load balancer types". I'm using an external load balancer (end users need to be able to access the content). The only other "service" parameters I could send pertain to Service Directory, which I don't use.

Is there some other parameter I'm missing? Is this error message accurate? I've been at this a few hours and I'm out of ideas.

Here's the POST request I'm making and the response I get (with private information replaced):

POST /compute/v1/projects/$PROJECT/global/forwardingRules?alt=json HTTP/1.1
Host: compute.googleapis.com
User-Agent: Terraform/1.3.3 (+https://www.terraform.io) Terraform-Plugin-SDK/2.10.1 terraform-provider-google/dev DeclarativeClientLib/0.0.1
Content-Length: 211
Content-Type: application/json
Accept-Encoding: gzip

{
  "IPAddress": "projects/$PROJECT/global/addresses/cdn",
  "loadBalancingScheme": "EXTERNAL",
  "metadataFilters": [],
  "name": "cdn",
  "portRange": "443",
  "target": "projects/$PROJECT/global/targetHttpsProxies/cdn"
}

{
  "kind": "compute#operation",
  "id": "xxx",
  "name": "operation-XXX",
  "operationType": "insert",
  "targetLink": "https://www.googleapis.com/compute/v1/projects/$PROJECT/global/forwardingRules/cdn",
  "targetId": "xxx",
  "status": "DONE",
  "user": "my-user@domain.com",
  "progress": 100,
  "insertTime": "2022-12-15T06:46:56.405-08:00",
  "startTime": "2022-12-15T06:46:56.421-08:00",
  "endTime": "2022-12-15T06:47:09.441-08:00",
  "error": {
    "errors": [
      {
        "code": "INVALID_USAGE",
        "message": "Service not found"
      }
    ]
  },
  "httpErrorStatusCode": 400,
  "httpErrorMessage": "BAD REQUEST",
  "selfLink": "https://www.googleapis.com/compute/v1/projects/$PROJECT/global/operations/operation-XXX"
}

The target https proxy "cdn" exists according to "gcloud compute target-https-proxies describe cdn".

Here's the relevant terraform config:

resource "google_compute_backend_bucket" "default" {
  name        = "default"
  bucket_name = google_storage_bucket.default.name
  enable_cdn  = true
}

resource "google_compute_url_map" "cdn" {
  name            = "cdn"
  default_service = google_compute_backend_bucket.default.id

  host_rule {
    hosts        = ["cdn.dev.example.com"]
    path_matcher = "default"
  }

  path_matcher {
    name            = "default"
    default_service = google_compute_backend_bucket.default.id

    route_rules {
      priority = 1
      service  = google_compute_backend_bucket.default.id

      header_action {
        response_headers_to_remove = [
          "x-guploader-uploadid",
          "x-goog-generation",
          "x-goog-metageneration",
          "x-goog-stored-content-encoding",
          "x-goog-stored-content-length",
          "x-goog-hash",
          "x-goog-storage-class",
          "server"
        ]
      }

      match_rules {
        prefix_match = "/some/prefix"
      }

      route_action {
        cors_policy {
          allow_origins = [
            "dev.example.com"
          ]
          disabled = false
        }

        url_rewrite {
          path_prefix_rewrite = "/"
        }
      }
    }
  }
}

resource "google_compute_managed_ssl_certificate" "cdn" {
  name = "cdn"

  managed {
    domains = ["cdn.dev.example.com"]
  }
}

resource "google_compute_target_https_proxy" "cdn" {
  name             = "cdn"
  url_map          = google_compute_url_map.cdn.id
  ssl_certificates = [google_compute_managed_ssl_certificate.cdn.self_link]
}

resource "google_compute_global_address" "cdn" {
  name         = "cdn"
  ip_version   = "IPV4"
  address_type = "EXTERNAL"
}

resource "google_compute_global_forwarding_rule" "cdn" {
  name       = "cdn"
  target     = google_compute_target_https_proxy.cdn.id
  ip_address = google_compute_global_address.cdn.id
  port_range = "443"
}

0: https://cloud.google.com/compute/docs/reference/rest/v1/globalForwardingRules

dpkirchner

It looks like the secret magic is to load_balancing_scheme to "EXTERNAL_MANAGED". Without that, you end up with "EXTERNAL", which is the "classic" load balancer. Classic doesn't support all the features documented in the REST API[0] and when you try to use those features you get either weird "Service not found" errors or something like (or similar):

Error: Error waiting for Updating UrlMap: generic::invalid_argument: Message does not pass whitelist validation. The offending field is a submessage of SimplifiedHttpRouteAction (UrlMap.SimplifiedHttpRouteAction) at field number 7 (UnknownField::Type 3)

Here's the working config:

resource "google_compute_backend_bucket" "default" {
  name        = "default"
  bucket_name = google_storage_bucket.default.name
  enable_cdn  = true
}

resource "google_compute_url_map" "cdn" {
  # TODO: use a different default service -- probably a bucket with a single
  # file? This is only used when a user hits our IP instead of hostname.

  name            = "cdn"
  default_service = google_compute_backend_bucket.default.id

  host_rule {
    hosts        = ["cdn.dev.example.com"]
    path_matcher = "default"
  }

  path_matcher {
    name            = "default"
    default_service = google_compute_backend_bucket.default.id

    header_action {
      response_headers_to_remove = [
        "x-guploader-uploadid",
        "x-goog-generation",
        "x-goog-metageneration",
        "x-goog-stored-content-encoding",
        "x-goog-stored-content-length",
        "x-goog-hash",
        "x-goog-storage-class",
        "server",
      ]
    }

    default_route_action {
      cors_policy {
        allow_origins = [
          "http://localhost:3000",
          "https://localhost:3000",
        ]
        disabled = false
      }
    }

    path_rule {
      service = google_compute_backend_bucket.default.id
      paths = [
        "/path/whatever/*"
      ]

      route_action {
        url_rewrite {
          path_prefix_rewrite = "/"
        }
      }
    }
  }
}

resource "google_compute_managed_ssl_certificate" "cdn" {
  name = "cdn"

  managed {
    domains = ["cdn.dev.example.com"]
  }
}

resource "google_compute_target_https_proxy" "cdn" {
  name             = "cdn"
  url_map          = google_compute_url_map.cdn.id
  ssl_certificates = [google_compute_managed_ssl_certificate.cdn.self_link]
}

resource "google_compute_global_address" "cdn" {
  name         = "cdn"
  ip_version   = "IPV4"
  address_type = "EXTERNAL"
}

resource "google_compute_global_forwarding_rule" "cdn" {
  name                  = "cdn"
  target                = google_compute_target_https_proxy.cdn.id
  ip_address            = google_compute_global_address.cdn.id
  port_range            = "443"
  load_balancing_scheme = "EXTERNAL_MANAGED"
}

0: To be fair, some of that is documented in the API, I just overlooked it. Seeing things like "HeaderAction is not supported for load balancers that have their loadBalancingScheme set to EXTERNAL" would have been the hint I needed.

kumards

Hi @dpkirchner ,

Glad that you were able to resolve the issue, and thank you for sharing the solution!

I agree that the "Service not found" error message is not actionable. If you don't mind, could you please report the issue at https://b.corp.google.com/issues/new?component=187245&template=1162973&pli=1 ?

To learn what happens after you report an issue, please see https://cloud.google.com/support/docs/issue-trackers#what-to-expect.

Attempting to configure a forwarding rule, seeing error "INVALID_USAGE" and "Service not found"