TL;DR:
After switching to a different google storage client software, I was met with a surprise bill and I suspect it's caused by a difference in the kind of operations, but it's not 100% clear to me this is the case. I want to know whether file versioning counts as a Class A operation or what kind of operation it is to update an existing file with more data. I was unable to find clear documentation about this, so linking me to documentation would be wonderful as well. Do file versions increase the storage bill (I assume they do)
Ultimately, I am trying to decide whether to continue using new client software. Is this increase in the bill a one-time abnormality that I can mitigate by changing a setting on our storage bucket?
details:
We use google cloud storage as well as a few other cloud storage providers using some adapter software we wrote ourselves. Essentially, our software would receive files over an API, encrypt the file, and then store the file in GCS. This results in -- I believe -- one Class A operation for every file as files are uploaded, and we had a predictable bill for this use.
Last month, we decided to experiment with another adapter software called "rclone". rclone allows the user to mount GCS as a fuse filesystem and use regular file utilities, and encryption and a few other things like this. As an experiment, we copied files from our old software into a new bucket into a new bucket. We anticipated that this experiment would cause our storage bill to double during the experiment. However, we were shocked to discover that our bill was more than 70x for the same stored data.
One observation I made is that on the "observability" tab, it looks like our rclone experiment bucket has about 4x stored than the size of the dataset. My assumption as to why this might be is file versioning. The storage bucket does have file versioning enabled. However, this additional storage only accounts for a small fraction of the billing increase. I don't yet have billing details available yet, but I am suspicious that the increase is from an increase in class A operations. Observability tells me there are a large number of "soft deleted" objects, and I suspect rclone may be creating temporary files and then deleting them, or updating existing large files.
Thank you in advance. Any documentation you can point me to would be greatly appreciated.
LinkedIn
Twitter
