This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Datastream Error

Posted on 03-02-2025 11:00 PM
NishinThattil
Bronze 4
Bronze 4
Reply posted on --/--/---- --:-- AM

how to know what cause the sql server to bigquery datastream failed with the following error: 

The stream is unable to read the event in LSN "xxxx" because the transaction log seems to be truncated.

0 3 159
Topic Labels
0 Likes
3 REPLIES 3

Posted on --/--/---- --:-- AM
mcbsalceda
Staff
Reply posted on --/--/---- --:-- AM

Hi @NishinThattil,

This issue happens because the transaction logs on the source SQL Server database have been truncated or no longer exist. Since Datastream relies on these logs for Change Data Capture (CDC) ingestion, it can't reliably replicate data to BigQuery when the logs are missing.

To diagnose and resolve this, start by confirming which CDC method is being used. If the stream is using the transaction log method, ensure the necessary configuration steps have been followed, such as adjusting polling intervals and setting up a truncation safeguard as outlined in “Configure a source SQL Server database”.

If log truncation is a recurring issue, switching to the Change Table CDC method may be a better option. This method offers better data reliability and avoids log retention problems.

If possible, try recovering the stream by reconfiguring or restarting it after verifying that the required logs are available. Making these adjustments can help prevent future failures due to log truncation.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

0 Likes

Posted on --/--/---- --:-- AM
NishinThattil
Bronze 4
Bronze 4
Reply posted on --/--/---- --:-- AM

Datastream  was streaming properly until the timestamp "2025-02-18". However, after that, at "2025-02-18", a CDC fetch started for logs between start LSN: xxx and end LSN: xxx.
The process was not completed, and at timestamp "2025-02-20", the stream's state changed to "FAILED_PERMANENTLY" with the error :
"The stream is unable to read the event in LSN 'xxx' because the transaction log seems to be truncated."

0 Likes

Posted on --/--/---- --:-- AM
NishinThattil
Bronze 4
Bronze 4
Reply posted on --/--/---- --:-- AM

The fetch was not completed. After two days, the stream failed because transaction log truncation after 24 hours was enabled. As a result, the error occurred: 'The stream is unable to read the event in LSN 'xxx' because the transaction log seems to be truncated.' As expected, after two days, the transaction log from 24 hours earlier was automatically truncated.

0 Likes