Solved: How to activate Dataproc Job metrics - Page 2

dextras · 05-26-2024 11:54 PM

Hi all,

I've been struggling with it for the past 2 weeks, going through GCP documentation and looking of the solution online. No luck. I simply do not know how to turn on (activate/enable) the Dataproc Job metrics?

When I look into Metric Explorer I see that Job metrics are inactive:

But when I check the documentation it states that by default Cluster AND Job metrics are enabled bye default and free of charge:

https://cloud.google.com/dataproc/docs/support/troubleshoot-monitor#metrics

Is there something I am missing? Is there any parameter I need to use while starting my Dataproc cluster in order to activate Job metrics? Did any of you had similar problem?

dextras

To be honest I am still not sure what exactly is the reason.
I did nor mention in original question though that I am not running the jobs directly but using ILUM, I assume that this is the reason why the do not show up in the GCP console. Also ILUM has an open request for integration with GCP https://roadmap.ilum.cloud/boards/feature-requests/posts/google-dataproc-integration where it states that there are a missing features :

visibility of the Dataproc jobs on GCP UI
managing access with Cloud IAM
logging with Stackdriver
etc.

I will stop my analysis here. Thanks @DamianS for taking time to answer!

What you wrote in your answer is also true 🙂 Just not in my case.

View solution in original post