I have two Data Fusion pipelines that replicate data from MySQL table to BigQuery tables. The second pipeline executes right after the first pipeline is completed. For time-saving and scalability purposes, I want to reuse the Dataproc cluster provisioned for the first pipeline. I have created an autoscaling Dataproc compute profile with the correct settings to reuse the clusters as specified here: https://cloud.google.com/data-fusion/docs/how-to/reuse-clusters.
When triggering the pipelines it always provisions and creates a second cluster when executing the second pipeline. I have made sure both pipelines are configured exactly the same on the customizable compute config settings.
LinkedIn
Twitter
