Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

BigQuery external table limitations

I have e-commerce product details stored in CSV format in the GCS bucket. Each file contains the attributes of a single product. The number of such files is approximately 4 billion.

I am planning to use a BigQuery external table to query the data.

It worked when I tested with 10 million such files. My question is, will it work when the number of files is as high as 4 billion? Even if it takes time (15-30 mins), it is fine.

What are the other recommended options to query the data if we don't use the BigQuery external table? 

3 5 4,544
5 REPLIES 5