Dataproc APIs requests but there are no clusters

[domaaoo1]

Hey,

I'm trying to isolate the projects that use Dataproc service. I did this by querying gcloud so that it would show which ones have dataproc.googleapis.com enabled and then checked to see if there were any requests.

What raised my concern was that on several projects for which we have requests in for “Cloud Datastore API” in the service itself there are no clusters. How is this possible? Is my reasoning correct?

[nikacalupas]

Hi domaaoo1,

Welcome to the Google Cloud Community!

Here are few possible reasons why you might notice Dataproc API requests in some projects without any active clusters:

• Clusters are created and deleted very quickly (most common).
• Cluster creation might be failing due to issues with quota, IAM, or configuration.
• Another service, such as Dataflow, could be indirectly using Dataproc.

To investigate, you can check the Google Cloud Audit Logs. Focus on the data access logs, which record API calls to Dataproc resources. These logs provide detailed information about every Dataproc API call, who made them, and whether they succeeded or failed. Sometimes the audit logs can have a slight delay before they appear.This method is much more reliable than just looking for active clusters. Additionally, verify your service accounts' permissions and quotas, look for any automations, and if needed, ask the project owners for more details.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.