Batch: Managing Instance Count and Tasks per Instance

[masony]

Is there a limit on the number of tasks that can be running on a single VM associated with a batch job? Is it 20? I have been attempting to run tasks with 1 runnable on N2D machines (n2d-highmem-48) with an allowance of 1 core and 8 GB of memory per task. However, I noticed I consistently only get about 41.7% CPU usage on each machine, and more machines are launched than I would expect if I was to get 48 individual tasks running on each machine as expected. For example, I do get full utilization with jobs launched that specify e2-highmem-16 machines.

[bolianyin · Accepted solution]

Yes, the number of tasks per VM is capped at 20 now, which might be increased in the future?  Is it a hard requirement to run more than 20 Tasks for your workload?

[masony]

It's not a hard requirement, but it would be convenient (and perhaps a bit more efficient from a resource utilization standpoint) if we could run more tasks per VM.