Google Cloud Run Job taskCount vs parallelism logical clash

[alizaidi]

For Google Cloud-run jobs, we can deploy a job to run X number of tasks and the max value here is 10,000, while the number of parallel executions for these tasks is between 0 and 100.

We can override the tasks count while enqueuing a job via rest API, gcloud command and through the available SKDs as well.

The issue is that the parallelism value must be less or equal than task count, and it can not be overridden while enqueuing the job like the task count can be.

I have a use case where I need to dynamically set the number of tasks, which I can do via the SDK, but since I can not change the parallelism for that specific execution I get the following error

description: must be less or equal than task count
field: spec.template.spec.parallelism

IMO this defeats the purpose of task count customization since the parallelism property doesn't allow full customization of this property.

I have a hacky workaround in my mind but for that, I would need to deploy around 100 job definitions each with separate parallelism properties, and then invoke the desired one based on the task count.

Is there a better way of achieving this, and why not allow a higher number of parallelism than the task count?

[knet · Accepted solution]

Hi, 

The default behavior is that if you don't specify a parallelism, Cloud Run will run as many tasks as possible in parallel. The only reason to set parallelism is if you want to limit how many tasks run in parallel - for example, if you have a backing resource like a database that doesn't do well with too many tasks accessing it in parallel.

So my advice would be to not set the parallelism at all, unless there's a need for it.

[alizaidi]

I see, I was expecting something like if task count is less than parallelism then set parallelism to task count kind of behavior.

But not setting parallelism property did the trick, thanks @knet for the prompt response.

[TheIK]

Although already answered, I would like to add a few words on the topic.

The behaviour initially suggested by @alizaidi is not only the most logical one (from user point of view) but it also should have been extended further to the scope of all job executions. For example, by introducing a parameter like "total_concurrency" or "job_parallelism" . Then you would have in total three parameters:

1. Number of tasks T (defined for each execution)
2. Parallelism P (defined for job, applies for every single execution)
3. Concurrency C (defined for job, applies for all running executions of a job)

Then, a hard requirement follows: P <= C. It is essential as parallelism and concurrency are properties of the compute resource itself which always has its limits.

However, T is defined by inputs to this resource. Inputs vary by definition - they come outside. Having 100% of your compute resource utilised is not only a bad idea, it's also notoriously hard to keep at the same level 24/7. It's a hard problem of science and it's not solved. Why Google Cloud expects its customers to know their expected load beforehand? And no, it's not always possible to "run all of them at once", as we live in the world where dozens of systems communicate to make one thing happen. They all have their rate limits.

Running executions with lesser number of tasks should have been made possible - it doesn't lead to any unintended consequences nor creates a potential for it. I can't think of any decent reasons behind current design. As for now, I resort to creating ugly copies of the same job for each value of parallelism I need.

[knet]

That's good feedback, thank you. We'll be working on fixing this in the near future and will allow higher parallelism than task count.