We have two projects one for testing and another for production. On each project we have a Cloud Run deployment connected to API Gateway, then a Serverless NEG and then a Backend Service with Cloud CDN.
On CDN configuration we set them to CACHE_ALL_STATIC and also enabled Signed URLs and set a signing key. Here is the describe output for our testing backend (some fields removed or modified for security):
> gcloud compute backend-services describe --global foo-test
affinityCookieTtlSec: 0
backends:
- balancingMode: UTILIZATION
capacityScaler: 1.0
group: https://www.googleapis.com/compute/v1/projects/foo-test/regions/us-east1/networkEndpointGroups/foo-test-serverless-neg
cdnPolicy:
cacheKeyPolicy:
includeHost: true
includeProtocol: true
includeQueryString: true
cacheMode: CACHE_ALL_STATIC
clientTtl: 3600
defaultTtl: 3600
maxTtl: 86400
negativeCaching: true
requestCoalescing: true
serveWhileStale: 86400
signedUrlCacheMaxAgeSec: '300'
signedUrlKeyNames:
- key-20221031
compressionMode: DISABLED
connectionDraining:
drainingTimeoutSec: 0
creationTimestamp: '2022-10-31T10:06:32.425-07:00'
customResponseHeaders:
- 'Cache-Status: {cdn_cache_status}'
- 'Cache-ID: {cdn_cache_id}'
description: ''
enableCDN: true
fingerprint: ...
id: '...'
kind: compute#backendService
loadBalancingScheme: EXTERNAL_MANAGED
localityLbPolicy: ROUND_ROBIN
logConfig:
enable: true
optionalMode: EXCLUDE_ALL_OPTIONAL
sampleRate: 1.0
name: foo-test
port: 80
portName: http
protocol: HTTPS
selfLink: https://www.googleapis.com/compute/v1/projects/foo-test/global/backendServices/foo-test
sessionAffinity: NONE
timeoutSec: 30
We are using URLPrefix caching in order to have groups of users who can see the same cached contents according to their access levels, one path prefix for group.
It was a pain to make it cache even with propper signature (more on that latter) but it was working.
Suddenly, about 3 weeks ago (around June 12, 2023) the endpoint stopped working because it can not verify signed URL parameters. We are checking signature as suggested in CDN documentation (https://cloud.google.com/cdn/docs/using-signed-urls#security_considerations). Then we found that signed URL parameters are no longer passed to our backend. In production it is still working (thank God) but in testing all query parameters from URLPrefix forward are removed on backend request.
We removed signature verify middleware from our backend in order to get our service working again but have found that no request is being cached. All requests return "cache-status: miss" header.
As I already told it was a pain to get "cache-status: hit" before. We had to juggle with backend response headers until finally setting "cache-control: public,max-age=300" made CDN cache work. Even when documentation says only url sign parameters are needed (https://cloud.google.com/cdn/docs/using-signed-urls#customize_the_maximum_cache_time) : "Cloud CDN caches responses for signed requests regardless of the backend's Cache-Control header".
By comparing our testing CDN config against production we found that production CDN does not have localityLbPolicy option so maybe the problem has been generated by a new version of CDN on testing which is worrying because we do not know how to fix the issue and in any moment could affect production.
So here are our questions:
- Why this CDN instance is modifying query parameters send to backend? all parameters from URLPrefix forward are removed.
- Why are requests with valid url prefix parameters and signature not cached? What are we missing?
Thanks in advance for any help.
LinkedIn
Twitter
