This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Azure Cosmos DB - BigQuery Integration

Posted on 11-24-2022 03:05 PM
fbermeoq
Bronze 4
Bronze 4
Reply posted on --/--/---- --:-- AM

Hi All

We have a prospect lead which have their JSON documents saved on Azure Cosmos DB. They require a data integration solution which extract data from Azure Cosmos DB and ingest it into Cloud Storage as raw layer. Then, take that some business information contained into the JSON and save it on BigQuery

ยฟSomeone have an idea about how to build that integration pipeline?

Thanks in advance for your guidance

0 1 1,091
Topic Labels
0 Likes
1 REPLY 1

Posted on --/--/---- --:-- AM
Roderick
Staff
Reply posted on --/--/---- --:-- AM

Sure, here are a few ideas on how to build that integration pipeline:

  1. Use a cloud data integration tool like Matillion or Fivetran. These tools can connect to Azure Cosmos DB and Cloud Storage and extract data from both sources. They can also transform the data into a format that can be loaded into BigQuery.
  2. Use a cloud data warehouse like Google BigQuery or Amazon Redshift. These warehouses can connect to Azure Cosmos DB and Cloud Storage and load data from both sources into their own storage. You can then use SQL queries to extract the data from the warehouse and load it into BigQuery.
  3. Use a cloud serverless compute platform like Google Cloud Functions or Amazon Lambda. These platforms can connect to Azure Cosmos DB and Cloud Storage and extract data from both sources. You can then use code to transform the data and load it into BigQuery.

The best approach for you will depend on the specific needs of your client's application. If you need to extract data from multiple sources and transform it into a format that can be loaded into BigQuery, then a cloud data integration tool is a good option. If you only need to extract data from one source and load it into BigQuery, then a cloud data warehouse or a cloud serverless compute platform may be a better option.

Once you have extracted the data from Azure Cosmos DB and loaded it into BigQuery, you can then use SQL queries to analyze the data and extract insights. You can also use BigQuery's machine learning capabilities to build models that can predict future trends.