I'm using Cloud Endpoints with ESPv2 on Cloud Run. I create the Endpoints config, build the ESP image and deploy to Cloud Run in Sydney/Australia. The Cloud Run deployment times out after 15 mins with the status:
Updating ingress traffic allowed. Resource readiness deadline exceeded.
This project is a few years old so I have reviewed the documentation and don't see any major changes. I event updated the gcloud-build-image script to no avail. My pipeline succeeded 2 months ago but fails today with no changes on my end. I have been unable to deploy for 36 hours now.
The current ESP version is 2.45.0, I tried using 2.44 which would have been the version used for my last successful deployment but same issue, also tried 2.43 to no avail.
I note there has been 2 Cloud Run releases since my last deployment.
I don't see any particularly interesting logs other than I don't use gRPC but there seem to be a lot of logs about it:
W0804 01:45:57.630 14 external/envoy/source/common/config/grpc_stream.h:201] [14][config]StreamAggregatedResources gRPC config stream to @espv2-ads-cluster closed since 92640s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination
Here's the timeout error log:
{
"protoPayload": {
"@type": "type.googleapis.com/google.cloud.audit.AuditLog",
"status": {
"code": 10,
"message": "Ready condition status changed to False for Service ***-apis with message: Updating ingress traffic allowed. Resource readiness deadline exceeded."
},
"serviceName": "run.googleapis.com",
"methodName": "v1",
"resourceName": "namespaces/***-prod/services/***-apis",
"response": {
"metadata": {
"name": "***-apis",
"namespace": "***",
"selfLink": "/apis/serving.knative.dev/v1/namespaces/***/services/***-apis",
"uid": "33f68c76-a626-4f48-9a50-ffab33f8eb39",
"resourceVersion": "AAYCDyBLuQ4",
"generation": 49,
"creationTimestamp": "2020-07-15T09:49:39.751710Z",
"labels": {
"run.googleapis.com/satisfiesPzs": "true",
"cloud.googleapis.com/location": "australia-southeast1"
},
"annotations": {
"run.googleapis.com/client-name": "gcloud",
"serving.knative.dev/creator": "***",
"serving.knative.dev/lastModifier": "***",
"client.knative.dev/user-image": "gcr.io/***-prod/endpoints-runtime-serverless:***-apis-***-ts.a.run.app-2022-06-07r1",
"run.googleapis.com/client-version": "441.0.0",
"run.googleapis.com/operation-id": "b2ec3451-85c9-43b9-a5ee-49f3466218f0"
}
},
"apiVersion": "serving.knative.dev/v1",
"kind": "Service",
"spec": {
"template": {
"metadata": {
"name": "***-apis-00049-wug",
"labels": {
"run.googleapis.com/startupProbeType": "Default"
},
"annotations": {
"run.googleapis.com/client-name": "gcloud",
"client.knative.dev/user-image": "gcr.io/***-prod/endpoints-runtime-serverless:***-apis-***-ts.a.run.app-2022-06-07r1",
"run.googleapis.com/client-version": "441.0.0",
"autoscaling.knative.dev/minScale": "1",
"autoscaling.knative.dev/maxScale": "1000"
}
},
"spec": {
"containerConcurrency": 80,
"timeoutSeconds": 300,
"serviceAccountName": "***-compute@developer.gserviceaccount.com",
"containers": [
{
"image": "gcr.io/***-prod/endpoints-runtime-serverless:2.45.0-***-apis-***-ts.a.run.app-2023-08-04r0",
"ports": [
{
"name": "http1",
"containerPort": 8080
}
],
"resources": {
"limits": {
"cpu": "1000m",
"memory": "256Mi"
}
},
"startupProbe": {
"timeoutSeconds": 240,
"periodSeconds": 240,
"failureThreshold": 1,
"tcpSocket": {
"port": 8080
}
}
}
]
}
},
"traffic": [
{
"percent": 100,
"latestRevision": true
}
]
},
"status": {
"observedGeneration": 49,
"conditions": [
{
"type": "Ready",
"status": "False",
"reason": "UpdatingIngressTrafficAllowed",
"message": "Updating ingress traffic allowed. Resource readiness deadline exceeded.",
"lastTransitionTime": "2023-08-04T01:50:49.862414Z"
},
{
"type": "ConfigurationsReady",
"status": "True",
"lastTransitionTime": "2023-08-04T01:40:45.257035Z"
},
{
"type": "RoutesReady",
"status": "False",
"reason": "UpdatingIngressTrafficAllowed",
"message": "Updating ingress traffic allowed. Resource readiness deadline exceeded.",
"lastTransitionTime": "2023-08-04T01:50:49.862414Z"
}
],
"latestReadyRevisionName": "***-apis-00049-wug",
"latestCreatedRevisionName": "***-apis-00049-wug",
"traffic": [
{
"revisionName": "***-apis-00043-hol",
"percent": 100,
"latestRevision": true
}
],
"url": "https://***-apis-***-ts.a.run.app",
"address": {
"url": "https://***-apis-***-ts.a.run.app"
}
},
"@type": "type.googleapis.com/google.cloud.run.v1.Service"
}
},
"insertId": "n1h8uqc5d4",
"resource": {
"type": "cloud_run_revision",
"labels": {
"location": "australia-southeast1",
"service_name": "***-apis",
"revision_name": "",
"project_id": "***-prod",
"configuration_name": ""
}
},
"timestamp": "2023-08-04T01:50:49.890845Z",
"severity": "ERROR",
"logName": "projects/***-prod/logs/cloudaudit.googleapis.com%2Fsystem_event",
"receiveTimestamp": "2023-08-04T01:50:50.163792374Z"
}
LinkedIn
Twitter
