How does Cloud Storage pricing with operations work?

[adrianom]

I'm trying to get an understanding of how Cloud Storage costs work. I have a Cloud Run service that downloads an item to a bucket, processes that item, then uploads that item to a bucket. I created about 2 TB worth of data doing this and my buckets are standard storage. I then downloaded the data (all 2 TBs worth) to my local machine. I noticed in my Billing Report that I used about $311 worth of Cloud Storage in a day, but I am unsure why it would be this high. It's only been a day, so I doubt this has to do with the standard storage (per GB per month) metric. Reading the Cloud Storage documentation, I noticed a section on Operation charges. Does copying files to my Cloud Run services (and the downloading of the 2 TB of data to my local machine) fit in with the Class A operations listed here in the documentation? If so, what does Class A operations (per 1,000 operations) mean? I used this command: 

gsutil -m cp -r gs://my-bucket .

to download the 2 TBs to my machine. I use a similar command in my Cloud Run service to download data as well. Is this what is incurring the costs so fast?

[kolban]

What I would suggest is that you go to your Cloud Billing page in the Google Cloud Console.  Visit the Reports tab.  Run a report showing:

Group By: SKU
Services: Cloud Storage

That should produce a report showing your daily spend broken down by the underlying SKUs that are being billed.  Think of "Cloud Storage" as a service you are consuming ... but the billing for that service is composed of multiple SKUs that, on aggregate, constitute that service.   If we can see what proportion of your spend corresponds to which SKUs, that will go a long way to answering the questions.

[adrianom]

Thanks so much for always responding so quickly, kolban! Looking at the report, things are starting to make more sense (screenshot of the report attached below). So after I finished running everything I wanted to in Cloud Run, I downloaded about 2.1 TB of the data to my local machine. That has to be what the Download Worldwide Destinations (excluding Asia & Australia) refers to. Copying of data from a Cloud Run service to a bucket shouldn't be involved with this SKU, right?  Even the usage matches up with around the amount I downloaded to my local machine (the report shows 2,177.6 gibibyte, which lines up with the 2.1 TB I copied over) but isn't this considered more of an egress fee? I didn't know that downloading data from a bucket to a local machine costs this much money. EDIT: I found documentation on this particular SKU here.  Am I correct when I say that this charge was due to downloading data from the bucket to my local machine?

As for the Regional Standard Class A Operations I think that is due to using the gsutil command in my Cloud Run services to copy data to each of my buckets.  I guess I'm also confused with how computing Class A operations works (link to documentation on operation charges here). Since I am using the following command to copy data to buckets:

gsutil -m -o GSUtil:parallel_composite_upload_threshold=150M cp ./{} gs://{}

does it count each file copied as an "operation"? This command copies files in a parallel fashion. Also, do you know if mounting buckets to Cloud Run services to write data to bypasses Class A operations?

[kolban]

When I look at the pricing docs ... I seem to see that moving data within the same region is free.  So if your bucket were in us-central1 and your Cloud Run were in us-central1 there should be no charge.  However, there are charges if your permutations are other ... for example a Cloud Storage bucket located in a US Multi Region and your Cloud Run were located in a different region.

We see the list pricing by looking up the SKU here.   The prices seem to apply to "General Network Usage" ... which is where you are egressing Cloud Bucket storage over the Internet.

Your Regional Standard Class A operations seems a lot. 11.7 million class A operations.   28 days = 28 * 24 * 60 * 60 seconds = 2.4M seconds.   That is approx 4.9 operations per second CONSTANTLY.   The prices match ... 11.7M class A at $0.005/1000 does equal $58.

What is in your buckets?  Do you have millions of small files or a few large files ... or something else?  I would expect mounting buckets would NOT reduce the number of class A operations ... in fact, I could imagine you'd have to be cautious there.  If I had a Linux mount using something like GCSFUSE and ran a Linux process that "busy polled" the bucket, that would likely be an AWFUL idea and could easily rack up class A operations.

Think of a class A operation as performing a "listing or examination" of a file object in the bucket.