BigQuery Streaming Buffer Manipulation

[DataEngineer_1]

Hi All,

I am using BigQuery as a target for CDC implementation where I am using Debezium to pull the CDC records from a source database. For all inserts we are using the BigQuery streaming insert feature as it is very fast in comparison to the standard insert methods.

I found out that Streaming Inserts load records into a Streaming Buffer where the records reside until the time the buffer is flushed into the regular BQ storage (Buffer lifetime is around 90 minutes). This poses issues with our CDC record ingestion process in case a change is made on records which have been recently created/inserted.

Is there a way to manually flush or force flush the streaming buffer into the BQ storage?

Any other options to manage record changes (Updates/Deletes) for records which are sitting in the streaming buffer?

Thanks in advance for your help !

Cheers !