Establishing Google Cloud SQL connection is slow

Hello

We are looking the option to to use Google Cloud SQL servers, but on the initial testing phase we are strugling how the connection making itself to the instance can be so low compared to instances running on local mahcine. Currently our production servers has mysql running on same instance where apache itself is and all the comparing is made against this perspective.

We aim to run our code on Docker containers and hoping to move Cloud Run

Cloud instance is on same country (Finland) and there is 2vCPU and 8gigs ram on it. There is no difference do I use Public IP, Private IP or socket connections on when making test on Cloud Run.

Using the Public IP from my local machine is also as slow, also with or without using Auth Proxy.

I have now benchmarked making 100 mysql connections on php on for loop and timing that.

 

$page_start = microtime(true);
$dbc = new DBC();
$tot_start = microtime(true);
$before_con = microtime(true);
for($i = 0; $i<100 ; $i++)  {
    $dbc->connect();
    $after_connected = microtime(true);
    echo "Connected in time of:  ". round($after_connected-$before_con,5) ." seconds <br>";
    $dbc->close();
    $before_con = microtime(true);
}
$tot_end = microtime(true);
echo "total time: ". round($tot_end-$tot_start,5) ." seconds <br>";

 

Test Run on Cloud Run against Google Cloud SQL server instance: (part of the results)
MySQL socket connection / private ip same results

Connected in time of: 0.01384 seconds
Connected in time of: 0.01249 seconds
Connected in time of: 0.01057 seconds
Connected in time of: 0.0218 seconds
Connected in time of: 0.01704 seconds
Connected in time of: 0.01184 seconds
Connected in time of: 0.01108 seconds
Connected in time of: 0.01097 seconds

total time: 1.56648 seconds (100 x connection)

 

Test run on local docker against localhost mysql server on docker

Connected in time of: 0.00109 seconds
Connected in time of: 0.00092 seconds
Connected in time of: 0.00078 seconds
Connected in time of: 0.00086 seconds
Connected in time of: 0.00075 seconds
Connected in time of: 0.00065 seconds
Connected in time of: 0.0008 seconds
Connected in time of: 0.00055 seconds

total time: 0.06576 seconds (100 x connection)

Is this connection time just to be accepted or is there something we could do to improve it? Our product is ERP/reporting system, what makes lots of connections and with such a lag there basically is no usability when normal functions takes ages to complete. Comparing some report loading times , when on localhost report takes 1.5 seconds it takes 10-20 seconds on Cloud Run.  

 

 

 

 

0 3 1,530
3 REPLIES 3

In a cloud-based environment, the connection time between your application and your database can significantly impact performance. This is particularly relevant when using Cloud Run, a serverless compute platform on Google Cloud , and Cloud SQL, a fully managed relational database service on GCP.

Factors Affecting Connection Time

Several factors can contribute to slower connection times between Cloud Run and Cloud SQL compared to a local Docker and MySQL server setup:

  1. Network Latency: Cloud-based resources are distributed across different physical locations, introducing inherent network latency.

  2. Public Internet Routing: When using public IP addresses for connections, traffic may traverse the public internet, adding latency and potential bottlenecks.

Strategies for Improved Connection Time

To enhance the connection time between your Cloud Run service and Cloud SQL, consider implementing the following strategies:

  1. Private IP Connection: Utilize private IPs for connections whenever possible. This bypasses the public internet and often results in lower latency. Ensure both Cloud Run and Cloud SQL reside within the same VPC (Virtual Private Cloud) to leverage private IPs effectively.

  2. Connection Pooling: Implement connection pooling to minimize the overhead of establishing new connections repeatedly. This is particularly beneficial for applications with frequent database interactions. Connection pooling maintains a pool of reusable connections, reducing the need for constant connection setup and teardown.

  3. Database Proxy Use: Consider employing a database proxy like Cloud SQL Proxy to manage connections more efficiently and securely. While its direct impact on performance may vary, it can help in connection management and may provide additional security benefits.

  4. Instance Type Upgrade: If resource limitations are impacting performance, consider upgrading your Cloud SQL instance to a larger type. This should be done after careful analysis to ensure it addresses the actual bottleneck. A larger instance may provide more processing power and memory, improving performance.

  5. Region Alignment: Align the regions of your Cloud Run service and Cloud SQL instance to minimize network latency. Choose regions that are geographically close to each other to reduce network hops and improve connection speeds.

Additional Considerations

In addition to the primary strategies mentioned above, consider the following factors:

  1. Persistent Connections: Explore the use of persistent connections in your application to further reduce connection overhead in a cloud environment. Persistent connections allow for maintaining open connections between the application and the database, eliminating the need to re-establish connections for each request.

  2. Benchmarking and Monitoring: Regularly monitor and benchmark your Cloud SQL and Cloud Run services using Google Cloud's tools. This can help identify specific performance bottlenecks and track the effectiveness of any implemented optimization strategies.

  3. Optimize Queries and Schema: Review and optimize your database queries and schema for efficiency. Inefficient queries or poorly designed schemas can contribute to performance issues. Optimize queries to minimize execution time and consider schema normalization to improve data retrieval.

  4. Network Analysis: Investigate the network path and bandwidth between your Cloud Run service and Cloud SQL instance. Network configurations or issues can sometimes contribute to latency. Analyze network traffic patterns and identify potential bottlenecks that may require optimization.

By implementing these strategies and considerations, you can effectively enhance the connection time between your Cloud Run service and Cloud SQL instance, leading to a more responsive and efficient application. Remember that performance optimization is an ongoing process, and continuous monitoring and refinement can further improve your application's performance.

Could you give a link to a documentation for the point 4. Network Analysis ? How can we monitor the network latency between CloudRun and CloudSQL in the same region ?

While there's no single "network latency" metric between Cloud Run and Cloud SQL, Google Cloud offers tools to help you infer network performance and troubleshoot potential issues:

1. Cloud Monitoring

  • Focus: Request durations, execution times, network-related errors on both Cloud Run and Cloud SQL.
  • How-To:
    • Navigate to Cloud Monitoring in the Google Cloud Console.
    • Create a custom dashboard with relevant metrics from both services.
    • Look for:
      • Cloud SQL: cloudsql.googleapis.com/database/network/round_trip_latency [invalid URL removed]
      • Cloud Run: run.googleapis.com/request_latencies [invalid URL removed]

2. Cloud SQL Insights

  • Focus: Database query performance (which can be indirectly impacted by network latency).
  • How-To:
    • Access Insights from your Cloud SQL instance's page.
    • Look for slow queries, especially those that perform well locally but poorly in the cloud, as this could suggest network-related issues.

3. Custom Metrics and Logging

  • Focus: Direct measurement of database operation latency from your application's perspective.
  • How-To:
    • Add code to measure the time for database connections and query executions.
    • Send these timings to Cloud Logging.
    • Analyze logs to understand how your application experiences latency.

Additional Considerations

  • VPC Flow Logs: For advanced analysis of network traffic patterns (requires network expertise).
  • Network Intelligence Center Connectivity Tests: Mainly used to ensure basic connectivity, not latency-focused.

Key Points

  • Direct latency measurement requires a combination of these tools.
  • Optimize your application (efficient queries, connection pooling) alongside network analysis.
  • Establish a performance baseline to understand what "normal" latency looks like for your setup.