Solved: How do I specify resorceHints when running a Dataf... - Page 2

Zanuah · 11-06-2023 12:06 PM

Hello all,

We are trying to deploy our dataflow job as a classic template. The job is Apache Beam Java and when we run the job using maven, we can specify --resourceHints to increase the min_ram (--resourceHints=min_ram=64GB), but I have been unable to figure out how to set this when running the classic template using 'gcloud dataflow jobs run'.
Any help is appreciated.

Thank you!

ms4446

While the gcloud dataflow jobs run command doesn't directly support specifying resourceHints for Classic Templates, you can consider the following options:

Option 1: Hardcode in the Template Creation:

Modify your Java Beam code to include the desired resourceHints during the pipeline construction. This means that the resourceHints will be part of the template itself.

PipelineOptions options = PipelineOptionsFactory.create();
// Set the resource hints directly in the options
options.as(DataflowPipelineWorkerPoolOptions.class)
    .setResourceHints(ImmutableMap.of("min_ram", "64GB"));

When you create the template, the resourceHints will be embedded within it.

Option 2: Use Flex Templates:

Convert your Classic Template to a Flex Template, which allows you to specify resourceHints at runtime. This involves:

Modifying your pipeline to be compatible with Flex Templates.
Packaging your pipeline as a Docker image.
Specifying resourceHints in the Flex Template launch command.

For example:

gcloud dataflow flex-template run "your-job-name" \
--template-file-gcs-location="gs://your-template-bucket/path-to-your-template-spec.json" \
--parameters="resourceHints=min_ram=64GB,..."

Flex Templates offer more flexibility and are the recommended approach if you need to specify dynamic resource requirements.

For the most accurate and up-to-date information, always refer to the official Google Cloud Dataflow documentation.

View solution in original post

How do I specify resorceHints when running a Dataflow classic template