Composer uses very high amount of ListObjects operations

[rafael2]

Hi,

I have noticed my GCS costs increasing for the past few months and decided to investigate. I used Metrics Explorer on a specific bucket and am seeing over 22 ListObjects operations per second. The number of operations has climbed every week since mid January.

I have enabled the Data Access Audit Logs and I can see the API requests come from the service account associated with my composer environment. 

Up until this morning, I had enabled the auto composer snapshot to be saved to this bucket every friday, but I have disabled this option and did not see any improvement.

I have also disabled every single DAG on the Airflow webserver and nothing has changed.

Since it is not a faulty DAG, what else could it be?

Thank you,

Rafael.

[rafael2]

I have also removed the "Storage Object User" role of the Composer Service account, so it shouldn't even be able to list the buckets, but the operations remain on the same level.

Here is the monitoring chart showing the ListObjects operations on this specific bucket.

I have no idea what's causing this.