I am having the issue with dataflow flext temaplete , while lauching the jobs from vertex ai kfp pipeline getting,two jobs are triggering when I lauched the dataflow jobs from flextemplate run command one that one is taking the job name from the flextemplate run command and other one is from main.py code, provide the steps to launch the dataflow code from dataflow flex template and I making the docker container like this
# Use the official Apache Beam Python container as base
FROM gcr.io/dataflow-templates-base/python3-template-launcher-base
WORKDIR /app
# Configure the Template to launch the pipeline with a --requirements_file option.
# See: https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/#pypi-dependencies
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="/app/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="/app/src/main.py"
COPY . /app
# Set PYTHONPATH environment variable
ENV PYTHONPATH="/app"
RUN pip install --no-cache-dir -r requirements.txt
# Set the entrypoint
#ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"]
LinkedIn
Twitter
