Datastream from MySQL to BigQuery: issues with update

[iAbdullah80]

Hello,
I’m encountering an issue with Datastream that’s set up to replicate data from a MySQL 8.4 source into BigQuery. I'm using GTID based replication and the stream write mode is merge. The stream correctly captures and reflects inserts and deletes in BigQuery, but updates to any record never show up. Also btw if i do initial backfill the updates show up,  I’d appreciate any suggestions on how to resolve this.

Thank you.

[mcbsalceda]

Hi @iAbdullah80,

Welcome to Google Cloud Community!

It sounds like your issue might be related to one of the limitations of GTID-based replication in Datastream. Since this feature is still in Preview, it has some constraints—such as no support for stream recovery and occasional data discrepancies during schema changes. Given that updates show up during a backfill but not in real-time, the problem could stem from how Datastream handles schema or event processing for updates in GTID mode.

Double-check if any recent schema changes might have occurred or if the table structure includes unsupported elements like spatial data types or JSON columns with restricted values. If none of that applies, it might be worth reaching out to Google Cloud support for assistance since the feature is still evolving.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[iAbdullah80]

Thanks for the reply. I wanted to mention that my setup uses Oracle HeatWave MySQL 8.4 which is a managed service,  does that raise an issue?

[Tanmay953]

Facing similar issue as of @iAbdullah80, @mcbsalceda do we need to check any mysql parameter which is blocking the updates to be fetched by Datastream or is it that Datastream GTID doesn't support updates yet?