Solved: Error in Dataflow pipeline - reading json and writ... - Page 2

ammar_hanif · 06-26-2023 02:54 AM

Hi everyone,

I'm using a basic ETL pipeline to run in dataflow. Python script is in my default cloud shell VM from where I'm running it. I have defined the arguments as mentioned below,

python3 my_pipeline.py --project centon-data-pim-dev --region us-central1 --stagingLocation gs://sample_practice_hrp/staging --tempLocation gs://sample_practice_hrp/temp --runner DataflowRunner

link to the sample file used,
https://github.com/GoogleCloudPlatform/training-data-analyst/blob/master/quests/dataflow_python/1_Ba...

while the other options are as below,

input = 'gs://sample_practice_hrp/events.json'.format(opts.project)

output = '{0}:logs.logs'.format(opts.project)

Unfortunately, I'm receiving some error, causing the pipeline to fail (picture attached)

ms4446

For diagnosing such issues, you might consider the following strategies:

Use a structured approach to error solving: Start with the error messages you receive. Even if they seem cryptic at first, they often provide a starting point for your investigation. Then, check the prerequisites for your pipeline (like whether the required datasets and tables exist, whether the data is in the expected format, etc.).
Check permissions: Ensure that the Dataflow service account has the necessary permissions to both the source (GCS) and the destination (BigQuery). It requires storage.objects.get and storage.objects.list for GCS, and bigquery.dataEditor and bigquery.jobUser for BigQuery.
Examine the logs: Cloud Logging is an invaluable tool for understanding what's happening in your pipeline. The logs for Dataflow jobs can be lengthy, but they're often the best place to find detailed error messages. You can filter the logs by the job ID or other parameters to narrow down the volume of logs you need to examine.
Validate your data: Make sure the input data is in the expected format and schema. Any discrepancies in data format or schema can cause your pipeline to fail.
Use error handling in your code: Consider adding try/except blocks around parts of your code where errors may occur. This can provide more detailed error messages and make it easier to identify the cause of the issue.

View solution in original post

Error in Dataflow pipeline - reading json and writing to BQ