This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Slow response from BigQuery API request in python, but much faster through Cloud SQL

Posted on 04-19-2023 02:35 AM
dor
Bronze 2
Bronze 2
Reply posted on --/--/---- --:-- AM

I am experiencing slow response times when fetching data from BigQuery through an API request in python. The query takes much longer than expected to execute, and I'm not sure why. Here are some details:

  • I am using the Google Cloud Python SDK to make the API request.
  • The query is relatively simple and returns around 10,000 rows of data.
  • I have checked the BigQuery usage and billing logs, and there are no apparent issues or errors.
  • I have also tested the query in the BigQuery console, and it executes much faster there.
  • However, when I tried fetching the same data using Cloud SQL with Google Cloud Python SDK, the response time was much faster.
  • I also tried to use postman for the check

Any suggestions on what could be causing this slow response time when using BigQuery, and how I can improve it? Is there any known difference in performance between BigQuery and Cloud SQL?

Thanks 
Dor

2 2 3,940
Topic Labels
2 REPLIES 2

Posted on --/--/---- --:-- AM
christianpaula
Staff
Reply posted on --/--/---- --:-- AM

Hi @dor,

Welcome to Google Cloud Community!

the slow response time when fetching data from BigQuery through an API request in Python can be caused by factors such as data size, network latency, and API configuration. If Cloud SQL provides a faster response time for your use case, it may be a better option. To improve the response time when using BigQuery, you can optimize your query, use caching, increase the number of slots, and use a faster network.

Thank you

Posted on --/--/---- --:-- AM
limyj06
New Member
In response to christianpaula
Reply posted on --/--/---- --:-- AM

Hi @christianpaula,
Would you mind to elaborate more on the options of improving the API query speed?

I am experiencing a similar issue that a same query with API request is much slower than running it on Cloud SQL, despite the data size is small (<10,000 rows). 

The network speed I have is 100Mbps for both download and upload.

Some attempts I did:

  • use_query_cache=True (if I understand the documentation correctly, this only applies if the same query string is used)
  • use_legacy_sql=False

Any reason that the Cloud SQL would be much faster than an API request?

Top Solution Authors
View all