Bigquery having issue with concurrent queries

[nikhilparab92]

We are running concurrent queries in bigquery but getting below error,

"Resources exceeded during query execution: Too many DML statements outstanding against table lmic-prod-datahub:lmic_datahub_prod_config.dropzone_files, limit is 20"

is it better to increase quota or any other way we can resolve this?

NOTE: we are using cloud function to execute those queries and python is the langu

[nikhilparab92]

Quick reply will be appreciated

[glen_yu]

Increasing quota won't solve your problem.  As per the documentation , BQ only runs 2 concurrently and can queue up to 20, or else it will get a table fail, which is what you're encountering now.   At the bottom of that same document you will find some Best Practices which says to avoid individual updates but rather group the DML ops together when possible. 

EDIT: I don't know what your queries look like, but if there are ways you can make it more efficient/scan less tables, it'll both save you money and time on each operation you're trying to make.

[glen_yu]

One more option you can maybe look into is limiting the number of CF instances with max instances option.  If you limit your max-instances to 20 for example then you shouldn't go over your queue size since it's your CF that triggers the DML queries.  Don't know if that's a feasible option for you though.

[glen_yu]

Also this: https://cloud.google.com/blog/products/bigquery/performing-large-scale-mutations-in-bigquery <-- if you want a better understanding of why