Data at Google Cloud Next '24!
Answer this question in the comments to earn a special badge: In your journey, what's one overlooked data skil...
Forum Posts
Influence Looker’s Roadmap and join the BI Customer Council (BICC)
Would you like to influence Google Cloud’s BI roadmap more? I’d like to invite you to join the Google Cloud BI...
Insert/Update to GCP Bigquery using Spring boot
Hi @, I need to insert/update data received from another saas based application to GCP bigquery, could you sug...
DataFlow job converted into pipeline scheduled not running
Dear All,I'm quite new to DataFlow, I've build a job to read data from SQL Server and write into BigQuery. Whe...
Replicating data from MySQL to BigQuery using Dataflow template
Hello BQ users! We recently published a step by step guide, which walks through how to set up data replication...
Big Query missing 3 days data from GA4
Hello all,Would like to seek for anyone's kind help here for an issue encounted. We have an established linkag...
I want to publish my app but I HAVE to have a login screen. Is there a workaround?
Greetings,Not sure if this is the right forum to ask this, so please, if my question doesn't fit the forum, in...
Dataform Incremental Backfill
Hi guys,I am trying to create an incremental table for a web and app event tracker but because I have a limit ...
Near Realtime with Oracle 12C OCI and DataStream ¿Is it feasible?
Hi allA prospect lead needs to extract data as near realtime from their Oracle 12C 1.0.2 instance hosted in Or...
Convert a GCP Dataflow Successful Job into Pipeline?
Dear GCP community, I have run few DataFlow jobs (MongoDB to BigQuery) and they successfully completed with th...
Missing Days from GA4 BigQuery Export
HiMy GA4 daily export has various missing days. It seems like its happened to various users however Ive not se...
No default table is created and no data is exported in bigquery
On my firebase console I have linked my project and did the necessary changes on the app side so that the metr...
DATAFLOW -Operation ongoing for over 2437.47 seconds in state process-msecs in step Move-ptransform-
HelloI have a dataflow job that:Reads a GSC bucketDecompress n tar files with m images. 2Gb of size per each t...
Error in Dataflow pipeline - reading json and writing to BQ
Hi everyone,I'm using a basic ETL pipeline to run in dataflow. Python script is in my default cloud shell VM f...
Trying to access Analytics project using http post with service account
We are able to access our Analytics data through postman (using Oauth 2.0). However, when using Postman we get...
Issue with dataflow template "Text files on gcs to pubsub"
I got a error message with SDK version 2.48.0 while everything is normal in 2.47.0:Show More Error message fro...
apache beam error
Hi community,I have been trying to merge an unbounded pcollection and a bounded pcollectionThe unbounded pcoll...
Where can I get support for Cloud data fusion that I didnt use
I tested out some of googles demo projects and one of them used cloud data fusion. I shut down the deployment ...
Issue running healthcare data harmonization pipeline
Hi - Was wondering if anyone could offer some help/guidance on an issue I'm hitting. New to GCP and Dataflow. ...
DATAFLOW: what is the difference between jsonPayload.portability_worker_id and jsonPayload.worker in
HiI wanted to know how many threads were in use by a specific job. When exporting the job logs, I saw these fi...
Dataflow memory usage never stops increasing
Hi Folks! I would like to know if anyone as any idea of what could be happening here. I have a streaming job w...
Huge delays in GA4 linked BigQuery data
Hi,We're using GA4 in our app, and linked it to BigQuery so that we collect raw data there.Normally, previous ...
How can i run multiple queries in dataflow template?
I am performing MSSql to bigquery data pipeline job using jdbc to bigquery template in dataflow.Since, I'm wor...
DataFlow template MongoDB to Bigquery (CDC)
HiI am using mongodb to bigquery cdc template and I am able to create a successful job.However when the mongod...
Triggering multiple requests when editing a pipeline in Dataflow
I've been getting the following message when creating/editing a new Dataflow pipeline in GCP:Quota exceeded fo...
Dataflow job machine type and estimated cost
I'm deploying a Python pipeline to Dataflow and specifying a machine type of e2-small. According to published ...
DATAFLOW: Weird graph in Estimated Cost feature from Dataflow
Hello people!I love the Estimated cost feature since its a quick way to, not only get an estimated of the cost...
DATAFLOW UI should show the information in the same time zone
Hello everyone!The logs and the minicharts are displaying different timezones. Can we have the same timezone i...
DATAFLOW: teardown method doesn't execute. Please help.
Hello!I'm working with Google Dataflow in Apache Beam Python 3.8 SDK 2.37.0. The issue I'm facing is that one ...
DATAFLOW: 0 elements added to the output after Flatten
I'm testing dataflow for 1 MongoDB document. It moves through the different steps of the pipeline. In the UI I...
Adding labels to Dataflow
Hi, I am trying to add labels to a Dataflow job that I'm creating from a template.Under the "optional paramete...
DATAFLOW: Where in the Dataflow UI can I see my Parameters?
Hi!Where in the dataflow UI can I see my Parameters? In the Job Info section, I can only see:Job Info:Job name...