Hello ,
I have setup datastream to push events from a GCP Cloud mysql instance to google storage.
With no load on the database, If I change data In the mysql database , it takes approx 2 mins until a json file is created in google storage.
In the destination config:
file_rotation_mb = 200
file_rotation_interval = "15s"
For example in the resulting json I have
"read_timestamp":"2024-05-02T17:16:16.529Z",
"source_timestamp":"2024-05-02T17:15:02.000Z",
Is this latency expected?
I would love to get a 1-5 secs Data freshness when testing the system. But maybe this is not possible?