Hello ,
I have setup datastream to push events from a GCP Cloud mysql instance to google storage.
With no load on the database, If I change data In the mysql database , it takes approx 2 mins until a json file is created in google storage.
In the destination config:
file_rotation_mb = 200
file_rotation_interval = "15s"
For example in the resulting json I have
"read_timestamp":"2024-05-02T17:16:16.529Z",
"source_timestamp":"2024-05-02T17:15:02.000Z",
Is this latency expected?
I would love to get a 1-5 secs Data freshness when testing the system. But maybe this is not possible?
Hi @jackmann,
Welcome to Google Cloud Community!
Based on this documentation on monitoring Datastream, temporary spikes in latency are to be expected when there's a significant increase in event throughput and it would also decrease gradually as the backlog of event is processed.
Hope this helps.
Am having the same setup and similar issue but more latency. am seeing a sawtooth pattern when freshness rises almost to 4hrs and then sharply drops and repeats. I've tried to go through the docs for optmization tips but no luck. my rotation thresholds are 20MBs/60s. Any tips?