newbie
I am unable to debug my spark jobs. They fail with an OOM exception. The only tool I have to debug are log files. I am having a heck of a time with dataproc spark logging. My my job completes I get a msg on the job details tab . Unfortunately the I can not debug my driver and work logs, the files are missing
Google Cloud Dataproc Agent reports job failure. If logs are available, they can be found at: https://console.cloud.google.com/dataproc/jobs/a222e057148b4878b2f516f07ca368b9?project=dataprocspar...ion=us-central1 gcloud dataproc jobs wait 'a222e057148b4878b2f516f07ca368b9' --region 'us-central1' --project 'dataprocspark-328421' https://console.cloud.google.com/storage/browser/anvil-gtex-v8-hg38-edu-ucsc-kim-lab-spark/google-cl... gs://anvil-gtex-v8-hg38-edu-ucsc-kim-lab-spark/google-cloud-dataproc-metainfo/2aada858-aa7d-4727-8547-2b119e5bcef4/jobs/a222e057148b4878b2f516f07ca368b9/driveroutput
Today I started using https://cloud.google.com/dataproc/docs/guides/logging?authuser=1#job_logs_in
When I create my cluster I set
--properties dataproc:dataproc.logging.stackdriver.job.driver.enable=true,dataproc:dataproc.logging.stackdriver.job.yarn.container.enable=true
I also set
--max-idle 10m
If the cluster is up I can get to the Dataproc Job driver and YARN container logs are listed under the Cloud Dataproc Job resource.
Once the cluster terminates. This no longer works. For the life of me, I can not figure out how to use log explorer https://cloud.google.com/logging/docs/view/building-queries?_ga=2.93830367.-351085001.1615423034
I create a cluster for each job. There are only a hand full of logs.
- it is super slow so as to be unusable. My job fails in about 30 mins.
- I want to look at my driver and yarn/work log files individually
- I want to select warnings and errors in chronologic order
- at this point, I think it would be faster and easier if I could just download the logs and use grep
I really wish there was an easy to to get to the log files from the dataproc job details page
Any suggestions would be greatly appreciated
Kind regards
Andy
LinkedIn
Twitter
