So I have been using MySQL to GCS Datastream pipeline. When I was inspecting the job , I found out that as shown in the pic
There are 105 events that were unsupported and were not loaded within last 7 days.
In the help section , I found out the following note
The number of events DataStream was not able to process in the last 7 days.
These events are not loaded to the destination,
and you are not charged for them.
More details about unprocessed events can be found in the logs.
When I checked out logs I did not find any logs about unprocessed events. Mostly I found out logs like CDC fetch failed: An unexpected error occurred. Contact Google Support... (which is mostly a recoverable error). There was almost no info about the unprocessed event. Any way or method via which I can get details of these unprocessed events or any suggestion on how these events can be avoided ?
cc: @etaim
As this would be hard to reproduce, I just want to share with you some GCP documentations and other resources [1,2,3,4,5] that could help you with audit logging and system logging for MySQL to GCS Datastream.
[1] https://cloud.google.com/datastream/docs/audit-logging
[2] https://cloud.google.com/logging/docs/audit#system-event
[3] https://cloud.google.com/datastream/docs/faq
[4] https://cloud.google.com/sql/docs/mysql/replication#bin-log-impact
[5] https://dev.mysql.com/doc/refman/5.6/en/binary-log.html
@RC1 What do you mean by "which is mostly a recoverable error"?
I'm having the error: "CDC fetch failed: An unexpected error occurred. Contact Google Support.." when I started the stream. Do you know why this is happening?
I'm having a similar error "Fetch failed: An unexpected error occurred. Contact Google Support.." and validation tests all passed ok. Has anyone found a solution?