Dataproc spark job execution error (ERROR ApplicationMaster: User class threw exception)

[srinidevops95]

23/02/09 09:48:12 INFO JobExecutor: Starting a new execution for job-20230209094543-08db8f0d
23/02/09 09:48:12 INFO JobExecutor: Connecting to wba-performance-dataproc-cluster-m-0:9001
23/02/09 09:48:12 INFO JobExecutor: Connected to wba-performance-dataproc-cluster-m-0:9001
23/02/09 09:48:12 INFO JobExecutor: Obtaining job assignment for job-20230209094543-08db8f0d
23/02/09 09:48:13 ERROR JobExecutor: It seems that the launched executor in charge of running execution job-20230209094543-08db8f0d, doesn't have any assigned job. This can happen if the job was already assigned to other executor.
23/02/09 09:48:13 INFO JobExecutor: Closing connection to job manager server
23/02/09 09:48:13 ERROR ApplicationMaster: User class threw exception: com.mobileum.aip.sjm.runtime.executors.JobNotFoundException: Job job-20230209094543-08db8f0d doesn't have any work assigned
com.mobileum.aip.sjm.runtime.executors.JobNotFoundException: Job job-20230209094543-08db8f0d doesn't have any work assigned
at com.mobileum.aip.sjm.runtime.executors.JobManagerClient.getJob(JobManagerClient.java:73) ~[__app__.jar:?]
at com.mobileum.aip.sjm.runtime.executors.JobExecutor.run(JobExecutor.java:53) ~[__app__.jar:?]
at com.mobileum.aip.sjm.runtime.executors.JobExecutor.main(JobExecutor.java:186) ~[__app__.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:739) ~[spark-yarn_2.12-3.3.0.jar:3.3.0]
23/02/09 09:48:13 ERROR ApplicationMaster: Uncaught exception:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301) ~[spark-core_2.12-3.3.0.jar:3.3.0]
at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:511) ~[spark-yarn_2.12-3.3.0.jar:3.3.0]
at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:275) ~[spark-yarn_2.12-3.3.0.jar:3.3.0]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:926) ~[spark-yarn_2.12-3.3.0.jar:3.3.0]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:925) ~[spark-yarn_2.12-3.3.0.jar:3.3.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at javax.security.auth.Subject.doAs(Subject.java:423) ~[?:?]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878) ~[hadoop-client-api-3.3.3.jar:?]
at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:925) ~[spark-yarn_2.12-3.3.0.jar:3.3.0]
at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) ~[spark-yarn_2.12-3.3.0.jar:3.3.0]
Caused by: com.mobileum.aip.sjm.runtime.executors.JobNotFoundException: Job job-20230209094543-08db8f0d doesn't have any work assigned
at com.mobileum.aip.sjm.runtime.executors.JobManagerClient.getJob(JobManagerClient.java:73) ~[?:?]
at com.mobileum.aip.sjm.runtime.executors.JobExecutor.run(JobExecutor.java:53) ~[?:?]
at com.mobileum.aip.sjm.runtime.executors.JobExecutor.main(JobExecutor.java:186) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:739) ~[spark-yarn_2.12-3.3.0.jar:3.3.0]

End of LogType:stderr

[Roderick]

Hey @srinidevops95,

Did you have a specific question about this error code? This error message indicates that the job manager server could not find any work to assign to the executor that was trying to run the job. This can happen if the job was already assigned to other executors, or if the job was not properly submitted.

To troubleshoot this issue, you can check the job history to see if the job was properly submitted. You can also check the executor logs to see if there were any errors during the execution of the job.

If you are still having trouble, you can contact the job manager server administrator for assistance.