Solved: Bigquery Concurrency - Page 2

gsanjeevi · 08-29-2023 01:36 PM

Hello,

We are running a multi threaded python application which makes the Bigquery connection to read some data.

With the max thread of 10, we are able to run the code successfully but when we increase the threads to 30, it errors out.

"Connection pool is full, discarding connection: bigquery.googleapis.com. Connection pool size: 10"

Does that mean max connection we can make for an application is 10 and is there any way we can improve it?

lsolatorio

Hi @gsanjeevi,

Welcome and thank you for reaching out to the community for help.

It appears that the default pool size is indeed 10 as indicated in this published python document.

I found this GitHub post that I believe proves to be a good workaround for your use case, from this as reference, please try this code snippet if it will work on your end.

client = bigquery.Client()
adapter = requests.adapters.HTTPAdapter(pool_connections=128,
pool_maxsize=128,max_retries=3)
client._http.mount("https://",adapter)
client._http._auth_request.session.mount("https://",adapter)
query_job = client.query(QUERY)

Hope this helps.

View solution in original post