How do I specify resorceHints when running a Dataflow classic template

[Zanuah]

Hello all,

We are trying to deploy our dataflow job as a classic template.  The job is Apache Beam Java and when we run the job using maven, we can specify --resourceHints to increase the min_ram (--resourceHints=min_ram=64GB), but I have been unable to figure out how to set this when running the classic template using 'gcloud dataflow jobs run'.
Any help is appreciated.

Thank you!

[ms4446 · Accepted solution]

While the gcloud dataflow jobs run command doesn't directly support specifying resourceHints for Classic Templates, you can consider the following options:

Option 1: Hardcode in the Template Creation:

Modify your Java Beam code to include the desired resourceHints during the pipeline construction. This means that the resourceHints will be part of the template itself.

PipelineOptions options = PipelineOptionsFactory.create();
// Set the resource hints directly in the options
options.as(DataflowPipelineWorkerPoolOptions.class)
    .setResourceHints(ImmutableMap.of("min_ram", "64GB"));

When you create the template, the resourceHints will be embedded within it.

Option 2: Use Flex Templates:

Convert your Classic Template to a Flex Template, which allows you to specify resourceHints at runtime. This involves:

1. Modifying your pipeline to be compatible with Flex Templates.
2. Packaging your pipeline as a Docker image.
3. Specifying resourceHints in the Flex Template launch command.

For example:

gcloud dataflow flex-template run "your-job-name" \
--template-file-gcs-location="gs://your-template-bucket/path-to-your-template-spec.json" \
--parameters="resourceHints=min_ram=64GB,..."

Flex Templates offer more flexibility and are the recommended approach if you need to specify dynamic resource requirements.

For the most accurate and up-to-date information, always refer to the official Google Cloud Dataflow documentation.