org.apache.spark.shuffle.FetchFailedException in Serverless Dataproc

[prabhaA]

I am getting the below error with serverless Dataproc (Batches) for a job.

"org.apache.spark.shuffle.FetchFailedException"

This is due to autoscaling removing executors, Serverless data proc doesn't support EFM; is there a way to handle this issue? Any help/pointer is appreciated.

Thanks in advance.

[osvaldolopez]

Try with these Stackoverflow posts:

1. FetchFailedException or MetadataFetchFailedException when processing big data set.
2. Downscaling problem and how to mitigate it.