schemas not streaming after instance recover

[M_GCP]

have an SQL instance called production, we stopped this instance for about two months, after I started it again, the google cloud console, told me that due to stopping the instance for a long period you (me) have lost the log file and log file position, so he gave me 4 choices, and I selected the second choice which was to skip for now and start from the last log file and data, and then, the instance started and back to work again, But the problem I'm facing right now is that the Datastream tables and schemas became not streaming, and I don't want to checkbox every schema manually, before stopping that instance and when I open the Google Cloud Datastream and click on "Edit source configuration" I always had some schemas always checked and streaming by itself, now they don't.

plz help me to make these selected schema streaming again to the BigQuery without i put them in the objects to include.

[ibaui]

Hi @M_GCP,

Welcome to Google Cloud Community!

The loss of log file position, due to the long downtime and the selected recovery option ("skip for now and start from the last log file and data"), directly impacted Datastream's ability to reliably replicate data. This choice caused Datastream to resume from a new point, losing the historical context of changes made during the downtime, which is why the previously selected schemas and tables are no longer streaming. Since Datastream relies on transaction logs for Change Data Capture (CDC), the loss of log file position stopped the stream, and the schemas that were automatically selected for streaming are no longer included. You can check this document, especially regarding the loss of the log file and log file position.

With Datastream losing its continuous log position, this likely caused the internal tracking of active schemas to become invalidated or reset. As a result, you will need to manually list the schemas you wish to include in the “objects to include” field. Additionally, consider initiating a backfill to ensure data consistency between the source and destination.

If the above doesn’t work and the stream is too broken to recover, you might need to recreate it.

If the issue still persists, I recommend reaching out to Google Cloud Support for further assistance. They have the tools, access to more detailed diagnostics, and expertise to delve deeper into the problem and can provide tailored guidance for your specific solutions.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.