Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Datastream unsupported events

RC1
Bronze 4
Bronze 4

So I have been using MySQL to GCS Datastream pipeline. When I was inspecting the job , I found out that as shown in the pic

 

 

There are 105 events that were unsupported and were not loaded within last 7 days.

 

 

RC1_0-1669195945979.png

In the help section , I found out the following note 

 

 

The number of events DataStream was not able to process in the last 7 days. 
These events are not loaded to the destination, 
and you are not charged for them. 
More details about unprocessed events can be found in the logs.

 

 

When I checked out logs I did not find any logs about unprocessed events. Mostly I found out logs like CDC fetch failed: An unexpected error occurred. Contact Google Support...  (which is mostly a recoverable error). There was almost no info about the unprocessed event. Any way or method via which I can get details of these unprocessed events or any suggestion on how these events can be avoided ?

cc: @etaim 

1 3 1,693
3 REPLIES 3

As this would be hard to reproduce, I just want to share with you some GCP documentations and other resources [1,2,3,4,5] that could help you with audit logging and system logging for MySQL to GCS Datastream.

[1] https://cloud.google.com/datastream/docs/audit-logging
[2] https://cloud.google.com/logging/docs/audit#system-event
[3] https://cloud.google.com/datastream/docs/faq
[4] https://cloud.google.com/sql/docs/mysql/replication#bin-log-impact
[5] https://dev.mysql.com/doc/refman/5.6/en/binary-log.html

@RC1 What do you mean by "which is mostly a recoverable error"?
I'm having the error: "CDC fetch failed: An unexpected error occurred. Contact Google Support.." when I started the stream. Do you know why this is happening?

I'm having a similar error "Fetch failed: An unexpected error occurred. Contact Google Support.." and validation tests all passed ok. Has anyone found a solution?