Replicating Oracle data in BigQuery for real-time and automated insights

[fguan]

Hello BQ users!

We recently published a step by step guide (with sample codes), which walks through how to set up Change Data Capture from Oracle database in BigQuery. If you're interested in automating the process for faster insights, take a look at the tutorial and feel free to post any questions in this post!

https://cloud.google.com/blog/products/data-analytics/unlock-real-time-insights-from-your-oracle-dat...

[DataEngineer_1]

Thanks for sharing this!

Could you help show how the records look like on BigQUery side? I would also like to understand how the UPDATES & DELETES are being handled? I was unable to find DataFlow job templates which could support streaming data loads with UPDATES & DELETES. It seems that the standard templates only supports INSERTS.

Looks like the streaming buffer in BQ does not allow changes other than inserts. Any suggestions on how to handle the changes on records sitting in BQ Streaming Buffer?

[caugusto]

Hi. Please find more information about the Dataflow template for Datastream here: 

https://cloud.google.com/dataflow/docs/guides/templates/provided-streaming#datastream-to-bigquery

 The template inserts all changes into a "<TABLE>_log" table first and then performs a merge into the final table. The merge will perform upserts to keep the end table updated in BigQuery.  

[cloud-jake]

Check out the PREVIEW of Datastream for BigQuery

This new option really simplifies CDC from Oracle to BigQuery!!