Hi @rrotter thanks for sharing your case , this is actually a pretty common scenario when working with MongoDB and BigQuery. Let me walk you through your questions step by step:
1. Does the streaming template need a connector?
Yes, the MongoDB to BigQuery (Stream) template requires a real-time change data source — like a MongoDB change stream. This means your MongoDB instance needs to be set up with change streams enabled (which requires a replica set), and Dataflow must be able to access that stream of changes.
2. What’s causing the "No such object..." error?
That error usually happens due to a failure during pipeline initialization or due to permission issues with the Dataflow staging bucket. A few things to check:
3. Does the template use WRITE_APPEND?
Yes , both the batch and streaming templates use WRITE_APPEND by default. So if you want to avoid duplicates, you’ll need to implement deduplication logic — either directly in the pipeline (e.g., using GROUP BY in Dataflow or filtering with BigQueryIO.Write.Method.STREAMING_INSERTS) or by writing to a staging table and using a MERGE statement in BigQuery.
4. Is using Debezium a good alternative?
Absolutely. Debezium with Kafka Connect is a solid choice, especially if you already have the infrastructure or can spin it up using Docker or Kubernetes. It lets you capture CDC (change data capture) events from MongoDB and send them to a Kafka topic, which you can then consume and write to BigQuery using a custom logic (e.g., with deduplication via MERGE).
5. Is there a simpler, faster way to do this?
If you're looking for something quicker to set up and less operationally heavy, you could consider a solution like Windsor.ai. It can connect MongoDB to BigQuery, handle both the initial load and incremental updates, and gives you control over deduplication — without the need to manage Kafka or complex Dataflow pipelines.
Hope this helps!
LinkedIn
Twitter
