GCP Batch On GKE

[nitishukg]

Hi I am using GCP batch to provision thousands of batch job at a time as it is running on multi tenant so i cannot have task parallelism, now the issue arises is the allocation of IP and other resources. Is there any provision of having Batch in GKE like that AWS has ??

[bolianyin]

Batch does not currently support running jobs in GKE. Could you elaborate the issue you have about Batch regarding to IP and other resources?

[nitishukg]

Hi,

We have can have upto 10K or more batch job running in parallel so it requires same number of IP address. Moreover provision 10K jobs will take too much time if it has been in GKE we just needed to have pods provisioned there.

[bolianyin]

Are those 10K jobs share containers?  If so, running one Batch with multiple tasks could help. Batch supports up to 100K tasks in a job. This way, you can also have multiple tasks run on one node/VM.  In addition, if some jobs don't need external IP, the no_external_ip_address  field in the job can be set to save IP addresses.  Will that help?

[nitishukg]

No they are different jobs right now its per event batch job because of the system requirement.  Is there for batch to natively use GKE ? since GKE can handle pods creation and node creation.
and in batch node spawning takes 2 minute roughly time in case of e2 medium 🙂

[nitishukg]

Hi I have a set of task that I want to execute in a order 

1. T1.1, T1.2,T1.3

2. T2

3. T3

I want to have batch 1 Execute in parallel ,  Batch 2 after batch 1 , and Batch 3 after batch 2.

Within a Batch job how should i achieve this ? 

[bolianyin]

@nitishukg Batch jobs don't run on GKE. That is something we consider to support in the future. The 2 mins delay mostly due to creating new VMs for the Batch jobs. We are working on a new feature where jobs could use existing VMs created by other jobs in certain cases. This should significantly improve the startup latency.

In your 3-job example, the 3 tasks in job 1 should run in parallel by default. Batch does not support job dependency. However, you can use GCP Workflows to achieve  the desired result by submitting a Batch job after the previous one finishes.  Here is a tutorial on using Workflows with Batch although it is about the dependency case.

[nitishukg]

Hi @bolianyin ,

For the example shared above its 3 task within the same job how can we get that within one job?

As for  Job Dependency  what does it do?

[bolianyin]

The "Create a basic job" section in Guides contains instructions to create a job with multiple tasks.  The Job dependency in the v1alpha API is not yet implemented.

[nitishukg]

@bolianyin within a  job how the task dependency can be made?

[bolianyin]

Yes, but in a very limited way for now. You can specify tasks in the same job to run sequentially.

[nitishukg]

Hi @bolianyin 

Suppose I have T1,T10 tasks I have set parallelism to 2 and scheduling policy to "IN ORDER" what will be task execution plan??

(t1,t2), (t3,t4) etc...

[bolianyin · Accepted solution]

IN_ORDER only supports running tasks one by one for now (parallelism=1).