Solved: count no of objects in a bucket/folder - custom me... - Page 2

sandeepguptha9 · 05-24-2023 07:28 AM

Hello,

I am really trying to create a custom metric to "count the no of objects present in a bucket/folder/folder"

Once I create a metric I want to export the results to the big-query table via sink with the parameters (project id, bucket name, folder name, no of objects, and timestamp)

Could anyone tell me how to make it?

ex: my project - X, bucket name: Y, subfolder: Z or maybe bucket has many sub-folders but I need to count from one of the sub-folders

KirTitievsky

GCS object count metrics can be grouped by bucket, but not by object prefix (e.g. "folder/folder"). I wonder if you might have a better time using storage insights (https://cloud.google.com/storage/docs/insights/inventory-reports) or BigQuery object tables ( https://cloud.google.com/bigquery/docs/object-table-introduction) to get all object metadata into BigQuery and then create your metrics as queries over that data in BigQuery.

Note that Inventory Reports are shipped as GCS objects. You can use them in BigQuery either by defining external tables in BigQuery or just running a regular BigQuery load job.

View solution in original post

count no of objects in a bucket/folder - custom metric