Batch: TaskEnvironments Array has a max length of 200 items

[jacksonwb]

The TaskEnvironments array is the best way to specify task specific input when launching batch jobs. But this field is limited to supporting 200 tasks. 

Is there a strong technical reason for this? Can this be increased?

Wanted to flag that it would be extremely helpful for me if this limit could be increased.

[bolianyin]

Hi @jacksonwb, the limit is out of practical reason. It can be increased if really needed, but we suggest you consider the alternative of reading per tasks files (or per task portion of a shared file) based on BATCH_TASK_INDEX first.

We were balancing an easy way to set a small number of per task env vars and special handling inside tasks. Is your job spec written manually or generated by some tool?  Is it easy to maintain if you set all these vars in the job spec vs storing in separate files?  In addition, you may hit the create job request size limit if the environment section is too large.

[jacksonwb]

Job yamls are generated using a tool. 

The TaskEnvironments is the easiest interface to specify each task's work as it allows us to compute the work relevant components (i.e. which input file / input file range) outside of the task and submit together, vs. having to have every task for every job have a way to compute the relevant work inside the task from the BATCH_TASK_INDEX.

We can do that but it would just be nice to be able to use the per task Env vars.

[Shamel]

We noticed this thread was not updated, but would like to share that the taskEnvironment limit of 200 has been removed.