Solved: Selected software image version '2.1.24-s8s-spark'... - Page 2

jeyob · 11-27-2023 09:09 AM

Hi,

I have a dataproc serverless batch job using the 2.1 Runtime version.
Looking to use the new autoscaling version on my job (i.e.

"spark.dataproc.scaling.version": "2") but when submitting the job it fails instantly with:

"Selected software image version '2.1.24-s8s-spark' does not support control plane decommissioning"

I can also mention that I am using a custom image (according to instructions on the docs)

Any idea what might be causing this?

ms4446

I'm glad to hear that the new patch version 2.1.27 resolved the control plane error for your Dataproc Serverless setup.

Regarding the verification of the autoscaling version:

Observe Scaling Behavior: The most practical way to infer the autoscaling version is by observing how your jobs scale. Autoscaling version 2 should show more efficient and responsive scaling, especially in reducing resources when they're not needed.
Check Documentation and Release Notes: Keep an eye on Google Cloud's documentation and release notes for any updates or tips on verifying autoscaling versions.
Conduct Experiments: If possible, test with controlled workloads to see how the autoscaler responds under different conditions.

Staying updated with Google Cloud's releases and monitoring your system's performance will help you make the most of Dataproc Serverless.