Datastream analysis bigquery

[marcelocastro]

Hello, good morning,

I was searching for some information about Datastream and BigQuery and did not find anything about my doubt.

My doubt is: I created a streaming Datastream -> BigQuery and my base was 130GB. But when I saw BQ Analysis it told me that 29TB was processed. Why did this happen? In this project I do not have anything work with BQ.

Thanks,

[Aris_O]

Hi @marcelocastro 

Welcome back to Google Cloud Community.

Any configuration change leads to the creation of a new revision. Subsequent revisions will also automatically get this configuration setting unless you make explicit updates to change it.

For Cloud Run services, you can set memory limits using the Google Cloud console, the gcloud command line, or a YAML file when you create a new service or deploy a new revision

This reference might help you:
https://cloud.google.com/run/docs/configuring/memory-limits

[budiwaskita]

Hi Aris, I don't understand how configuration changes on CloudRun relate to the question: 

"Why does Datastream usage causes excessive BigQuery analysis charges?"

[marcelocastro · Accepted solution]

Good morning, my problem is in staleness configuration, because BQ always do a query before insert. Thanks,

[budiwaskita]

Hi @marcelocastro , thanks for the hint! I was having a similar issue and BigQuery analysis usage indeed went down as I increased the max_staleness settings.