Cloud Run high latency after deploy with Direct VPC

[bestway_gg]

Hello.

I've been using Cloud run with Direct VPC (with Route only requests to private IPS option) succesfully for about 6 months. I'm using Laravel 11 with Octane and connecting to Redis instance on Compute Engine and Cloud SQL. 

I have the same setup for my 2 Cloud run services. Let's say service A and service B.

3 days ago I deployed new version of my services. Unexpectedly service A got a hight latency for every single request (about 10-20s) but service B was completely fine. 

I got high latency on 80% of my deploys on service A since.

I though that was some problem with my code or slow DB queries but after checking everything I found nothing suspicious. 

The problem with high latency on service A lasts for good 10 minutes (after a deploy) and then everything return to normal. 

I implemented Cloud Trace to both of my services to try to narrow an issue.

After checking cloud trace when this high latency happend I noticed all reqeusts to Cloud Sql and Redis are extremely slow (like 700ms to Cloud SQl and 200/300ms to Redis when normally it toks 5ms for CloudSql and 1ms to Redis).

So I think the problem is somewhere with networking. I switched from Direct VPC to Serverless VPC access and now everything seems to be OK but at the end I prefer Direct VPC. So I'm asking you to help me.

[Joy_S]

Hi @bestway_gg,

Welcome to Google Cloud Community!

It seems like the issue you’re facing is related to network scaling or configuration within your VPC network. This issue can pop up when the service starts up for the first time or after a cold start, causing slow connections to Cloud SQL and Redis. Here are some workarounds that may help resolve the problem:

• Since Service B is working fine, carefully compare the Direct VPC settings—like routes, firewall rules, and VPC peering—between the two services. Any differences might be causing the issue.
• Cloud Run uses health checks to see if a service is ready to handle traffic. If your service is slow to set up network connections, the health check might mark it as unhealthy too soon. Keep an eye on the health check settings and tweak them if needed.
• Check out the IPs in your Direct VPC setup. Make sure the IPs for your Cloud Run services aren’t causing any conflicts or delays when they start up. It’s possible that new containers are trying to use IPs that aren’t fully set up yet or are taking time to get routed properly.
• The problem might also be with DNS resolution when using Direct VPC. With Serverless VPC Access, DNS resolution is probably smoother. You might want to check if DNS timeouts or routing settings are causing delays when the service starts up.
• Look into whether cold starts or the new version of the service in Cloud Run is changing the network setup, which could be causing the delays. Cold starts might take longer in Direct VPC if it’s waiting for the network connections to settle.

If the workarounds above don't work, you can contact Google Cloud Support for a more in-depth analysis. When contacting them, please provide comprehensive details and include screenshots. This will help them better understand and address your issue.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[MatthieuC]

Hi, we have the same issue on multiple cloud run. After seeing this post we switched one of our cloud run to serverless connector and it's also solved the issue

[bestway_gg]

My Cloud Run services were using First Generation environment. But after switching to Second Gen and Direct VPC everything started to work normally. So the problem were caused by having First Gen and Direct VPC.

When using First Gen and Serverless VPC access OR Second Gen and Direct VPC there is no issue.

[MatthieuC]

Oh we also use GEN1, I will try in GEN2 thanks !

[hungig]

We are also experiencing high request P99 latency when we deploy new revision. Started few days ago though.