I'm in need to upload a small chunk of files, around 70k of them, using the google cloud storage client for python. The size of the files are no larger than 1kb. In the beginning of the upload process I managed to upload about 20 files per second, but suddenly I started getting 429. After implementing measures to retry the upload waiting for some seconds between each request if I get 429, it is now suddenly always using more than 10 seconds for each upload.
I tried checking see if we were hitting the quota, but nothing is indicating to me that there is something wrong with the setup. Any suggestion on what could be causing the issue?