Influence Looker’s Roadmap and join the BI Customer Council (BICC)
Would you like to influence Google Cloud’s BI roadmap more? I’d like to invite you to join the Google Cloud BI...
Data Analytics
Want more from your data? Have questions about products like BigQuery, Dataflow, or Pub/Sub? You're in the right place. Join conversations around data warehouse modernization, streaming analytics, business intelligence, and more.
Focused on Looker? Engage with the Looker Community here.
Focused on Looker? Engage with the Looker Community here.
Forum Posts
Apache Spark program not receiving messages from Google Cloud Pubsub
I have a simple Java program that reads messages from a Google Cloud Pubsub topic and prints them. It works co...
Dataflow HttpError 429 when requesting to trigger a dataflow template
Hi all, We created a dataflow template and running dataflow pipelines from it since a month or two. Today we r...
How to configure Log severity in Dataproc Batches
I am submitting a job to Dataproc batches (Serverless) , We are charged thousands of dollars for clod logging....
Should I limit the size of the bigquery table raw_changelog table?
I linked it with Google data studio using "Stream Collections to BigQuery," which is a firebase extension.One ...
In Looker can I rename the "Totals" row that results from checkbox
See screenshot below. When I check off the Totals box, it produces a new row that sums up the values of the co...
How to check the data present in a Pub/Sub topic.
How can check whether a message is present in a Pub/Sub topic without using a subscriber or listener?
Is there a way to label multiple Bigquery tables at the same time using SQL?
As tables within a Dataset do not inherit the Dataset labels, I am looking for ways to label multiple tables.I...
Avro library in dataproc
I am using a Dataproc cluster with hadoop 3.2 and spark 3.1. I have a python code to read avro files from GCS....
Unable to connect Cloud Data Fusion to Kafka on GCE
Hello Everyone,I am reaching out here for guidance as am unable to create a successful connection between the ...
No pipelines needed. Stream data with Pub/Sub direct to BigQuery
Great feature for #BigQuery Community, kudos #GoogleCloudCommunity.https://cloud.google.com/blog/products/data...
Cloud Data Fusion: Multiple MongoDB collections to BigQuery
This is a repost of something I wrote on Stack Overflow a few days ago, but without any takers. Guess there's ...
AWS SNS equivalent in GCP
I have my data pipelines on GCP dataproc and want to implement notification in it. Is there any AWS SNS equiva...
Logs Explorer to Big Query
I have an API hosted on GCP, I am wondering if there is a way to do a one time dump of Logs Explorer data into...
How to compare current row values with next row values in BigQuery Stored Procedure
Hi,In Big Query stored procedure, I need to compare the current row values with next row values. How can I ach...
How to increase limit of Storage Write API?
This page:https://cloud.google.com/bigquery/quotas#write-api-limitslists a 10MB limit on Storage Write API req...
How to get job_id inside a pyspark job running on dataproc cluster ?
Is there any provision or any way to get the job_id of the pyspark job inside the pyspark code while running o...
Data Fusion - Preventing Duplicated from sinking to Bigquery & Denormalise
Please assist? I would like to prevent duplicates entering into my Google Bigquery db. I have used the Distinc...
Bigquery premature EOF
I have a table in bigquery with around 32 million rows and 85 columns with size of 22 GB. (It is a physical ta...
BigQuery Console UI un-nesting all structs
Something changed in the way results are displayed in the UI. All structs and arrays are automatically un-nest...
Big query alter table add column after a specific column
Is it possible to alter table and add a column after a specific column in Big query ? something like Alter tab...
transfer a file from a container to another
I'm trying to find the best way to communicate between 2 containers (cont1 and cont2) whenever a file is uploa...
Extract Telemetry Message Content and Respond With IOTCore Command to Device
Hello,I'm new to GCloud and am looking for a way to extract telemetry messages sent by devices to IOTCore, and...
com.google.cloud.bigquery.BigQueryException: Error getting access token for service account: Error w
We've run into a very weird issue while using the bigquery java client library. The method com.google.cloud.bi...
⭐️ Google Cloud Data Heroes Series: Meet Tomi, a data engineer based in Germany 🇩🇪
Hi Google Cloud Data Analytics Community! I'm Grace, your community manager flying in with the fourth Google C...
How do you orchestrate and schedule Dataproc serverless batch jobs ?
I have multiple pyspark jobs with dependencies. I want to orchestrate and schedule it. How it is possible to d...
Google Ad Manager: Big Query
How to transfer data transfer files to database? I understand that data transfer files from Google Ad Manager ...
Bigquery automated store proc for ddl creation
Hi,Do we have store proc to create complete ddl for bigquery
👉 Don't miss the Data Engineer Spotlight on July 20!
Join the Data Engineer Spotlight to connect with your fellow data engineers, learn best practices of Google Cl...
GCP Datastream new source connections
Hi guys , I need to get CDC from databases which are of (Mysql , Mongodb , Postgres) type and store it in GCS ...
I can not run any query on BigQuery anymore (transient error n°4233314)
I can no longer query nor preview any table nor run any job in my BigQuery (region europe-west2). When I run a...
Articles from Product Experts and Googlers
Next-Gen Data Pipelines: Airflow 3 Arrives on Google Cloud Composer
We are thrilled to announce that Google is the first hyperscaler to provide selected customers with access to Apache Airflow 3 on our fully managed Cloud Composer 3 service. Discover how powerful new features like DAG Versioning, a modern React-based UI, and scheduler-managed backfills can help you innovate faster and more reliably.
Configuring GKE rollout sequencing
You can manage the order of automatic cluster upgrades across Google Kubernetes Engine (GKE) clusters in multiple environments using rollout sequencing. For example, you can qualify a new version in pre-production clusters before upgrading production clusters.
MCP x ADK - Engineering Agents That Execute, Not Just Predict
Unlock the power of AI agents by connecting them to real-world tools and data. This guide explores the Model Context Protocol (MCP) for seamless integration. Follow the code-centric tutorial to build a Financial Advisory Agent with Google's ADK. Learn to equip your agents with external capabilities for complex, real-world tasks.
