Compressed storage model in BigQuery

[samy]

What is compressed Storage model in BigQuery and how it is different that current bigquery billing model?

[VishalBulbule · Accepted solution]

Currently in BigQuery storage is billed on logical size , with compressed billing model billing would be based on actual physical storage which is compressed and lower in size than logical mostly. compressed storage pricing allows you to only pay for data storage after it’s been highly compressed. With compressed storage pricing, you can reduce your storage costs while increasing your data footprint at the same time.  You can see below one example from one of my table where Physical storage is way less than logical as its compressed so my cost will be less .

[ms4446]

BigQuery's Compressed Storage model is a new pricing model that charges based on the physical storage used after data compression. This model was announced at the Google Cloud Data Summit and went into General Availability on July 5th, 2023. It is available for every customer using one of the three Editions, or any exclusive on-demand customer.

The Compressed Storage model can significantly reduce your BigQuery bill if you store large amounts of data in BigQuery. During the Preview phase, some customers achieved up to 50% savings while retaining full performance. The storage price, under this model, will be based on the compressed bytes used by the tables in that dataset. Data compression is handled automatically, and users can't fine-tune or adjust it. The compression rates observed in internal tests ranged from 1:6 to 1:12, but this can vary depending on data volume and data modeling.

The Compressed Storage billing model has higher storage costs per original gigabyte, roughly double, compared to the standard logical storage billing model. However, it still results in savings as the storage reduction itself is much larger than double.

In terms of eligibility, customers who exclusively use the on-demand model, and who have completed migration of their flat-rate slot commitments to BigQuery editions are eligible to use the new physical storage billing model. If you have flat-rate slot commitments, you need to either cancel them or fully migrate to Editions to be eligible for compressed storage pricing.

This new model is different from the current BigQuery billing model, which charges based on the logical storage used. The logical storage refers to the total amount of data stored without considering any compression or optimization. In contrast, the Compressed Storage model takes into account the compression of data, which can lead to significant cost savings for users storing large amounts of data.

For more information: https://cloud.google.com/bigquery/docs/editions-intro