This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Error on Data Flow Job - The Dataflow job appears to be stuck because no worker activity

Posted on 06-19-2022 06:03 PM
Nery
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

My DataFlow Job was working correctly every day until June 8th, the only message I can see is 

Nery_0-1655686678796.png

There was no change on my code, there is no error message on Job log. I am using 

tensorflow-transform==1.7.0
apache-beam[gcp]==2.37.0
tensorflow==1.15.5

Does anyone knows is there any change on gcp or some dependencies of these libraries were deprecated?

0 1 1,706
Topic Labels
0 Likes
1 REPLY 1

Posted on --/--/---- --:-- AM
Eduardo_Ortiz
Staff
Reply posted on --/--/---- --:-- AM

This situation can happen due three major reasons:

a) Some tasks take more than an hour to process.

b) Some tasks got stuck processing.

This is generally caused by transforms that take too long to process, or enter a blocking state.

Best way to debug is to check for previous logs and see if there were any errors, or unexpected state. Apparently you have tried to re-run the job and it still failed. In this case you can add extra logs to the step that got stuck and see which data it got stuck at.

c) There is a failure on the Apache Beam / Dataflow side. If so, you could create a support ticket.

0 Likes