BigQuery Compressed/Physical Storage tier

[VishalBulbule]

Hello All,

I am exploring BigQuery dataset new storage models for one of our client. One query here

When we convery logical billing model to Physical what happenes to Long term logical storage? Will it convert to Long term physical or Active Physical?

[ms4446]

BigQuery's new storage model, known as "Compressed Storage", prices based on physical storage after compression​.The conversion from logical to physical storage in this model is automatic, and you can't adjust or fine-tune the compression process​.

When you convert from logical to physical storage (now known as Compressed Storage), your Long Term Logical Storage will likely become Compressed Storage. However, the exact nature of the conversion depends on a number of factors, including the specific characteristics of your data and how well it compresses. The compression rate can vary significantly depending on data volume and data modeling​.

The cost for Compressed Storage is higher compared to the previous standard logical storage billing model. However, due to the compression, which can achieve ratios up to 1:6 to 1:12, the overall storage cost can be significantly less than double the original cost​.

It's also important to note that when you switch to the physical storage model, time travel storage (storing previous versions of data for a certain period to allow recovery) is billed separately and is part of the physical storage used​.

[VishalBulbule]

Thanks for quick response !! Coming back to main query,  Compressed model we have Active Physical and Long term physical as we can see in table details , if we convert long term logical to Physical will it become Active Physical or Long term Physical?

[ms4446 · Accepted solution]

When you switch your BigQuery dataset storage from logical to the compressed storage model (formerly known as physical), the storage type (i.e., "Active" or "Long Term") should be preserved. Therefore, if you have data in "Long Term Logical Storage" and you switch to the compressed storage model, it should become "Long Term Compressed Storage".

This is inferred from the fact that the new compressed storage model is simply a different way of calculating storage costs, and it's based on the actual physical storage used after data compression. This is not directly related to the concept of active versus long-term storage, which is more about how frequently data is accessed.

For example, active storage typically refers to data that has been modified in the last 90 days, while long-term storage refers to data that hasn't been modified for over 90 days. These categories should remain the same regardless of whether you're using the logical or compressed storage billing model.

[VishalBulbule]

Thank you for covering all doubts now , I was looking for this clarification !!