Integration between MongoDB and BigQuery

[happyhooter]

Hi all, I am very new to GCP and BigQuery and that's why I want to ask the question how to connect MongoDB with BigQuery?

As I currently understand I can use Dataflow to connect them. But just wanted to ask whether there are any other alternative methods?

[gowthamnagarajn]

Hi @happyhooter ,

If I understand correctly, you requirement is to read the entry from BigQuery and add it to Mongodb. For this use case, you can create an Application Integration Workflow with BigQuery and Mongodb connector tasks to achieve this. Adding the relevant docs below.

https://cloud.google.com/integration-connectors/docs/connectors/mongodb/v2/configure

https://cloud.google.com/integration-connectors/docs/connectors/bigquery/configure 

[happyhooter · Accepted solution]

Actually a bit different: On the daily basis I want to read the whole table from MongoDB and send it to the BigQuery table in the append mode. But I think it's quite easy to achieve this with Dataflow pipelines using MongoDB to BigQuery template.

[Snoshone07]

Hi @happyhooter You're absolutely right. Dataflow is one of the most common methods for connecting MongoDB with BigQuery, especially when using the MongoDB to BigQuery template. However, there are alternative approaches depending on your specific needs:

BigQuery Data Transfer Service (BQ DTS)

While BQ DTS doesn’t natively support MongoDB, you can use third-party connectors that integrate with this service.

Cloud Functions or Cloud Run

You can create a Cloud Function that triggers whenever there’s a new update in MongoDB, then writes the data to BigQuery using the BigQuery API.

Custom ETL with Python

Another option is to use Apache Beam on Dataflow or a simple Python script with pymongo and pandas to extract data from MongoDB and load it into BigQuery using the google-cloud-bigquery package.

Third-Party ETL Tools

If you prefer a low-code or no-code approach, tools like Windsor.ai offer seamless connectors for MongoDB and BigQuery, simplifying data integration without requiring infrastructure management.

If your priority is automation and scalability, Dataflow or an ETL tool would be the best choice. However, if you're looking for a quick and easy setup, exploring ETL platforms like Windsor.ai could save you significant time. Let me know if you need more details.

Hope this helps!