Unreliable Connections to Database from Cloud Run using Cloud NAT

[deguy]

Hello,

I have been experiencing an issue with our Cloud Run services for some time now when using the VPC network feature and Cloud NAT to get a consistent egress IP address. When I configure the Cloud Run service this way, there is a significant increase in request latency, and timeouts frequently occur.

The screenshot below depicts the difference in latency after I have disabled the VPC networking feature:

Turning off the feature resolves our problem, and we get excellent performance. However, this is at the cost of reducing security on the database service. The service is still new, and as a result, there is not a high volume of egress traffic, so I do not believe the latency is being caused by insufficient resources on the connector instance.

Some additional details:

Our Cloud Nat is on the Premium network tier:

Our Serverless VPC connector also shows very little load over the past 15 days:

----

Has anyone experienced this before? Is there something in my configuration that could cause this issue?

[VannGuce · Accepted solution]

Hi,

I suggest checking your connector and choosing the bigger instance type to have a higher bandwidth, also, the connector might still require sufficient CPU and Memory to handle NAT effectively.

Enable logging in your resources, monitor the time of the spike then check the logs. Make sure that the timestamp of the logs is also the same as the time of the spike. Inspecting the logs may give you an idea what is causing the issue.

Make sure that your Cloud Run services are running in regions that close your VPC network and database resource to avoid latency.