Solved: Real time streaming of analytics table in BQ to Pu... - Page 2

chittemp · 01-02-2024 08:21 AM

Hi,

Am currently working on a POC. We are leveraging Google analytics for our website and capturing the traffic in GA4. We setup a connection to stream the real time data from GA4 to BQ. Now we would like to stream the same data from BQ to Pub/Sub in real time.

I tried enabling the CDC events over the BQ Analytics table to publish an event to pub/sub topic and created a cloud function with event trigger on analytics table with append rows. Nothing is worked.

@ms4446

The idea is to stream the data from GA4 ---> BQ Table --> Pub/Sub in real time. Need assistance to accomplish this @ms4446

chittemp

Hi Praneeth,

Finally, I've materialized this approach due to the limitations with GCP for real time streaming.

1. Created a cloud scheduler to send a message to Pub/Sub topic for every 1 minute.

2. The cloud function is triggered when a new event is published to Pub/Sub Topic.

3. In the cloud function, am running the query on the GA4 streaming table to read the rows where the start time & end time falls within the minute.

4. The identified rows are streamed to the downstream application.

Please let me know if it helps and really appreciate the suggestions.

View solution in original post

Real time streaming of analytics table in BQ to Pub/Sub