MONGODB TO BIGQUERY - Connection and Transform

[rrotter]

I'm new to data engineering and using MongoDB, so I'd like some help, please.

I have a MongoDB database that needs to be migrated to BigQuery. The goal is to:

1. Send the data to BigQuery;
2. Periodically update the data that has already been migrated and add new data;
3. Transform this data (partition and cluster).

I initially implemented the Dataflow batch template MongoDB to BigQuery template, but it is WRITE_APPEND and this caused the target tables to have millions of repeated records.

Later I tried to use the MongoDB to BigQuery template (Stream), but it gave an error:

Failed to read the result file : gs://dataflow-staging-southamerica-east1-287781428749/staging/template_launches/2025-04-22_06_30_53-16695916874292364341/operation_result with error message: Calling GetObjectMetadata with file "/bigstore/dataflow-staging-southamerica-east1-287781428749/staging/template_launches/2025-04-22_06_30_53-16695916874292364341/operation_result/": cloud.bigstore.ResponseCode.ErrorCode::OBJECT_NOT_FOUND: No such object: dataflow-staging-southamerica-east1-287781428749/staging/template_launches/2025-04-22_06_30_53-16695916874292364341/operation_result/ [google.rpc.error_details_ext] { message: "No such object: dataflow-staging-southamerica-east1-287781428749/staging/template_launches/2025-04-22_06_30_53-16695916874292364341/operation_result/" details { [type.googleapis.com/google.rpc.DebugInfo] { stack_entries: "com.google.net.rpc3.client.RpcClientException: APPLICATION_ERROR;cloud.bigstore (...)

Questions:

1. What would be the best approach to the problem using open source tools? Use the Debezium connector, for example?
2. Does the 'MongoDB to BigQuery template (Stream)' template need a connector with MongoDB to work?
   1. Would this template solve my problem? Or part of it?
   2. I would also like to confirm: it is also WRITE_APPEND, correct?

Any help is welcome, as I have been looking for solutions for a few weeks now.

Thank you.