Datastream latency issue

[NishinThattil]

We are facing 80 seconds of latency in Datastream Mysql to Bigquery.

Reducing the binlog size configuration will  help if transactions are too big since binlog files are still very big. So this  workaround provided will only work for smaller transactions since smaller binlogs means Datastream can scale and read different files concurrently to support higher loads.

Is there any other workaround reduce the latency?

[-Rhett]

Hi @NishinThattil,

Possible causes for high latency in Datastream include:

• Configuration of the Datastream source
• Configuration of the data destination
• Configuration of the network Datastream uses

Here are a few troubleshooting steps you can try out:

• Check if network conditions or ISP performance might be contributing to the high latency
• Diagnose high latency when streaming from a MySQL source
• Follow best practices when using Datastream
• Refer to the detailed answer on Datastream latency provided by ms4446

As a workaround, you can also consider recreating the stream using our Datastream to BigQuery template to check if the high latency persists.

If these steps didn’t resolve the issue, feel free to ask a question on the Datastream Stack Overflow or open a ticket on the Datastream issue tracker. Include steps to replicate, configuration settings, screenshots, and other details of the issue to help us investigate. While there isn’t a specific timeframe for resolution, once we've fixed an issue in production we'll indicate this and then update and close the bug.

I hope this helps!