DATAFLOW -Operation ongoing for over 2437.47 seconds in state process-msecs in step Move-ptransform-

[davidregalado25]

Hello

I have a dataflow job that:

1. Reads a GSC bucket
2. Decompress n tar files with m images. 2Gb of size per each tar file.
3. Upload these files  to another GCS bucket

The DAG looks like this:

But the job keeps failing because of this error: Operation ongoing for over 2437.47 seconds in state process-msecs in step Move-ptransform-

I've tried with different machine types including: n1-standard-2, n1-standard-8, n1-standard-96, n1-highmem-8, n1-highmem-96.

How can I accomplish this using Cloud Dataflow? 

--
Best regards
David Regalado
Web | Linkedin | Twitter