LinkedIn
Twitter
Hello!
I'm working with Google Dataflow in Apache Beam Python 3.8 SDK 2.37.0. The issue I'm facing is that one of my DoFns is partially executing. Or at least, it seems that way since, according to the lifecycle of DoFns, they go through the following states:
- __init__
- setup
- process
- start_bundle
- finish_bundle
- teardown
Out of these methods, only __init__, setup, and process are being executed. Any idea?
More context: I'm processing only 1 document from MongoDB and each of the previous steps executes successfully. I can even see my log.info for the process method in the logs. But not for the teardown where my insert to Elasticsearch takes place..
This is my DAG:
I've tested with the following machine types:
n1-standard-1: a machine too slow so I can see the results before the teardown method finishes. Or at least, that was what I thought.
n1-highcpu-96: was it a machine too fast for 1 document?
UPDATE: I tried the same on COLAB and it worked!
