Re: Batch job on pre-existing vm

JuliaFaltech · 11-03-2022 12:05 PM

Hi,

Is it possible to run a new job on a pre-existing VM? What I would like to do is to use configurations I've used in the past, for script jobs that I have been routinely using. I would like it to scale to 0 when not in use, and prefferably have a gpu. Is this possible with any method on GCP? I've been trying to use batch jobs but it starts a new vm every time.

Shamel

Currently, this is not possible. Each job submitted to Batch will create a new managed instance group with new VMs. Although tasks within a job can share VMs, VMs cannot be shared across different jobs. Are there specific reasons why using a pre-existing VM is preferred for your use case? By scale to 0 do you mean to delete the VM after the job completes?

JuliaFaltech

That makes sense, thank you.

The reason why I would need a pre-existing vm is to skip out on setup time. Ideally I would be able to use the same script without needing to reset the vm, using the same setup every time. What I mean by scaling to 0 is turning the vm off and using less resources while the script isn't running.

JimmyPinks

This would also be a very desirable feature for me. In my experience startup time is on the order of 1-2 minutes. This can be too expensive when scheduling several jobs in a sequential manner. Machine re-use is useful for billing and for job time purposes. If the jobs started in 0-30 seconds it would be a different story.

Shamel

@JimmyPinks @JuliaFaltech - we have recently launched a new feature in preview that would allow reusing warm VMs for multiple jobs. If you are still interested, feel free to direct message me and I can share more details on how to enable this for your project.

rmmh

I'm interested too, is there public guidance on enabling the feature in preview? I don't see an option to direct message on this forum.

thiago-oliveira

I have interest as well. How can I have access to this preiew feature?

renatoff

I'm interested in this too. How do I enable it? What is the name of the feature?

Shamel

This feature, called node pools, is currently in private preview and will go public soon. If you have a Google Cloud account representative, please have them reach out to the Google Batch team about this feature, and we can share more details.