Cloud Run Service unable to connect to another Cloud Run Service via gRPC: "No route to Path"

[jmkernes1234]

I have two Cloud Run services running, which I will call "backend" and "helper". The backend service is a publicly accessible HTTPS connection, running a Python server. It sends requests to the helper service (which is running a Python gRPC server) via a Python gRPC client. At the moment, it is just running the standard health check protocol.

To help isolate the problem, I have controlled for the following things. Both backend and helper have role: "roles/run.invoker" and member "allUsers", and allow INGRESS_TRAFFIC_ALL. I have also enabled HTTP/2 on the helper service. When using the Python gRPC client on the backend, I am using the secure channel, i.e. the code snippet looks like `

grpc.aio.secure_channel(addr, grpc.ssl_channel_credentials())`, where `addr` is the hostname of the embedding service as it is displayed on cloud run, i.e. https://<hostname>.

I can confirm that accessing the helper service locally, from my own computer, using the above code snippet works successfully. This only appears for the Cloud Run service.

Finally, the specific error I am receiving is 

```

AioRpcError of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B1234:1234:1234:12::12%5D:443: No route to host"

```

I suspect this is a network problem as it takes roughly 30 seconds of hanging before this error appears.

[robertcarlos · Accepted solution]

Hi @jmkernes1234,

Welcome to Google Cloud Community!

Please check if your setup is listed as known Issues for Cloud Run. Your setup may have the following:

• HTTP/2 Push is not supported 
• Using HTTP/2 with Identity-Aware Proxy is not supported 
• VPC network egress does not support IPv6 

If the issues are not mentioned, you may file a bug through this link so that our engineers could take a look at it.

Hope this helps.

[jmkernes1234]

Thank you so much, you're a lifesaver! I was able to isolate the problem, and it was tied to option (3). After switching to an IPv4 only subnet, I was able to get things working. There was one hiccup though, which I'm not sure if it's a bug or not to report. I could only get everything to work once I selected to route all egress for the backend service through the VPC. In terraform, this amounted to setting `vpc_access { network_interfaces { egress = "ALL_TRAFFIC", ...}}`