Getting Access Denied when getting files from Cloud CDN

[simen-andresen]

Hi,

We're using Cloud CDN with a GCS bucket as back-end. We're also using Signed URL with URLPrefix   to prevent public access. 

This mostly works fine, but now and then, some users experience 403 responses:

<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message></Error>

What's worth noticing is that these kind of errors can happen in the middle of many successful requests. E.g. a single user can have the following requests:

https://domain.com/url-prefix/file1?URLPrefix=prefixbase64&Expires=1712220069&KeyName=our-key&Signat...  - Code 200

https://domain.com/url-prefix/file2?URLPrefix=prefixbase64&Expires=1712220069&KeyName=our-key&Signat... - Code 200

https://domain.com/url-prefix/file3?URLPrefix=prefixbase64&Expires=1712220069&KeyName=our-key&Signat...- Code 403

https://domain.com/url-prefix/file4?URLPrefix=prefixbase64&Expires=1712220069&KeyName=our-key&Signat...- Code 200

I.e. only the request for file3 fails with 403. (Notice that the prefix, expiration and signature is the same for all the requests.)

Furthermore, I get the following warning in the load balancer logs for the failed request:

{
  "insertId": "ephmc7fjnrey6",
  "jsonPayload": {
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "backendTargetProjectNumber": "",
    "cacheId": "FRA-1209ea83",
    "remoteIp": "...",
    "cacheDecision": [
      "RESPONSE_HAS_CACHE_CONTROL",
      "RESPONSE_CACHE_CONTROL_DISALLOWED_CACHING",
      "RESPONSE_HAS_EXPIRES",
      "RESPONSE_HAS_CONTENT_TYPE",
      "CACHE_MODE_CACHE_ALL_STATIC"
    ],
    "statusDetails": "response_sent_by_backend"
  },
  "httpRequest": {
    "requestMethod": "GET",
    "requestUrl": "...",
    "requestSize": "249",
    "status": 403,
    "responseSize": "425",
    "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0",
    "remoteIp": "",
    "referer": "..",
    "cacheLookup": true,
    "serverIp": "...",
    "latency": "0.059912s"
  },
  "resource": {
    "type": "http_load_balancer",
    "labels": {
      "url_map_name": "some-load-balancer",
      "project_id": "...",
      "zone": "global",
      "target_proxy_name": "...-proxy-2",
      "forwarding_rule_name": "...-forwarding-rule-https",
      "backend_service_name": ""
    }
  },
  "timestamp": "2024-04-04T07:48:37.615757Z",
  "severity": "WARNING",
  "logName": "...",
  "trace": "...",
  "receiveTimestamp": "2024-04-04T07:48:38.293388621Z",
  "spanId": "574a047ae0176190"
}

however, there's nothing in the log that gives me any clue.

Any help here would be greatly appreciated.