LinkedIn
Twitter
Hey everyone,
I'm currently working on scheduling a Dataflow pipeline, and I'm facing a challenge in parameterizing the source file path based on the latest received file in our Google Cloud Storage (GCS) bucket.
Every day, we receive a new source file with the naming convention "SALES_YYYY-MM-DD". Since the date suffix changes daily according to the current date, I need to dynamically select the appropriate file for processing in the pipeline.
I've tried to set up parameters as shown in the screenshot, but I keep encountering an "object not found" error.
Could anyone provide guidance on the correct approach for this situation? I'd greatly appreciate any help or insights you can offer!
Thanks in advance,
Murali
