Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Cloud Storage deletes across multiple buckets seem to share throughput

We have 3 different Google Cloud Storage buckets from which we are continuously deleting objects, based on an object-specific retention period (our own business logic). One bucket ("bucket 1") is multi-region (us), and the other two ("buckets 2 and 3") are regional (us-central1). Right now, we are averaging around 18 deletes per second on bucket 1 (multi-region), 30 deletes per second on bucket 2, and 220 deletes per second on bucket 3. These deletes are performed by Node.js workers running in GKE pods, also running in us-central1. 

Our bucket 3 deletes are obviously the highest-throughput. They are on their own, separate 5 GKE nodes. However, at times, bucket 3 deletes will pause, and when that happens, the rate of deletes for buckets 1 and 2 will jump to about 90 deletes/second.

We have also seen similar correlation between delete and write/upload throughput, although it's more complicated to compare, since uploads are customer-driven.

Given these are 3 entirely separate buckets, it doesn't seem like they should bottleneck each other with deletes. Also, none of these buckets are exceeding the 1,000/s API limit for write/upload; the closest is bucket 3 which sometimes peaks around 850/s, but doesn't reach 1,000.

Are there shared resources that could be causing these throughputs to be shared? Are there ways for us to decouple these delete processes so that they won't interfere with each other?

Screenshot 2024-10-29 at 2.35.00 PM.png

 

0 2 582