Dataflow Job Workers

[yogocik]

Hi, I am Yogic, Data Engineer. I have built a pipeline for migrating data from Datastore to Bigquery (every 6 hours, 7 days). Today, I have found some VM have been created with it's name is looks like come from Dataflow template (showed in VM Instance but not in VM Instance Dashboard). I have read the documentation and found this on FAQ,
https://cloud.google.com/dataflow/docs/resources/faq#is_it_possible_to_access_my_jobs_worker_machine...

At that time, I was relieved that was background action (by Google) but after couple of hours, there is still uncleaned VM until new cycle is started. I am affraid that this will cost us more than we expected (due to unclean VM by Google system). Is this normal behavior or something wrong happened?

[josegutierrez · Accepted solution]

For "Compute page shows incorrect/old VM with status started" sometimes UI takes time to show correct status of VM.

However, if you are concerned about VM billing charges then I would request you to please share the VM name or you can also check when VM was stopped by using the following documentation/ query . 

Once the VM is deleted you will not be charged anymore. To view billing reports you can use the following documentation.