Storage bucket class A and B operations file versioning

[coryschwartz]

TL;DR:

After switching to a different google storage client software, I was met with a surprise bill and I suspect it's caused by a difference in the kind of operations, but it's not 100% clear to me this is the case. I want to know whether file versioning counts as a Class A operation or what kind of operation it is to update an existing file with more data. I was unable to find clear documentation about this, so linking me to documentation would be wonderful as well. Do file versions increase the storage bill (I assume they do)

Ultimately, I am trying to decide whether to continue using new client software. Is this increase in the bill a one-time abnormality that I can mitigate by changing a setting on our storage bucket?

details:

We use google cloud storage as well as a few other cloud storage providers using some adapter software we wrote ourselves. Essentially, our software would receive files over an API, encrypt the file, and then store the file in GCS. This results in -- I believe -- one Class A operation for every file as files are uploaded, and we had a predictable bill for this use.

Last month, we decided to experiment with another adapter software called "rclone". rclone allows the user to mount GCS as a fuse filesystem and use regular file utilities, and encryption and a few other things like this. As an experiment, we copied files from our old software into a new bucket into a new bucket. We anticipated that this experiment would cause our storage bill to double during the experiment. However, we were shocked to discover that our bill was more than 70x for the same stored data.

One observation I made is that on the "observability" tab, it looks like our rclone experiment bucket has about 4x stored than the size of the dataset. My assumption as to why this might be is file versioning. The storage bucket does have file versioning enabled. However, this additional storage only accounts for a small fraction of the billing increase. I don't yet have billing details available yet, but I am suspicious that the increase is from an increase in class A operations.  Observability tells me there are a large number of "soft deleted" objects, and I suspect rclone may be creating temporary files and then deleting them, or updating existing large files.

Thank you in advance. Any documentation you can point me to would be greatly appreciated.

[alexmoore]

Hi!

This one is a tricky one to answer without more visibility, but to answer your specific question here is the reference that shows which operations fall into Class A and B:

https://cloud.google.com/storage/pricing#operations-by-class

Do you have versioning enabled on the bucket? Also what class of storage do you have, because if you are deleting objects early from tiers that have a minimum storage duration there can be early delete charges.

Best,

Alex

[coryschwartz]

Interesting, that might be the case, I'm not sure.

The data is mostly archival, so it's in archival storage class. There is versioning turned on, but this is an oversight, we don't need it.

I count myself fortunate that we did this experiment on a small dataset. The data we used in the experiment was only about 20 TB, and was costing tens of dollars per month. for the last few months. We uploaded the same content into a new bucket using rclone+rsync and we were unknowlingly incurring a huge charge every day. Billing projections for this GCP project are up 76,061% (76 thousand percent). It's such an incredible surprise. 

It's really difficult to figure out what's causing the issue. The billing portal just lists "Storage"  as the major cost API, but it doesn't break it down any further than that.

[alexmoore]

You should be able to get more clarity on the source of the cost.  In the Billing section of the console, click into "Reports" on the left hand side, it will give you a billing breakdown, on the right hand side under "Filters" you should see an option "Group by", by default it will show "Service", but if you pick "SKU" it should break it down further.