Influence Looker’s Roadmap and join the BI Customer Council (BICC)
Would you like to influence Google Cloud’s BI roadmap more? I’d like to invite you to join the Google Cloud BI...
Data Analytics
Want more from your data? Have questions about products like BigQuery, Dataflow, or Pub/Sub? You're in the right place. Join conversations around data warehouse modernization, streaming analytics, business intelligence, and more.
Focused on Looker? Engage with the Looker Community here.
Focused on Looker? Engage with the Looker Community here.
Forum Posts
Transfer csv file from BigQuery to GCS without loading it into memory
Consider the following code: #Initialize GCS client gcs_client = storage.Client() # Define the destination URI...
GCP Integration with Dynatrace
I would like to integrate GCP with Dynatrace monitoring service to analyze the logs and metrics from gcp. For ...
Firestore -> BigQuery
Hi Folks,My company has a mobile app with lots of data stored in Firestore collections. We'd like to run data ...
Equivalent of Insert Overwrite strategy in DataForm
Hello, i'm testing DataForm instead of DBT, and i'm missing a feature we use a lot. In DBT incremental mode we...
Stream data availability
Data is available for real-time analysis using GoogleSQL queries immediately after BigQuery successfully ackno...
Dataflow Provisioning time and optimization
We are using dataflow for batch workloads which are small in future roadmap we want to enable streaming worklo...
Data Engineering vs Cloud Architect
Hi, I’m currently studying to become a data analyst for a telecoms provider in the UK. Just working on my PDP ...
BigQuery Temporary Tables - Storage & Billing
1) BigQuery Temporary tables are scoped within the session / multi statement query. Will its storage as a data...
Access to street number metadata in google maps?
Hi,When I'm viewing a location on google maps, I can see the street numbers overlaid on building locations. Do...
UDF error: java.lang.NoSuchMethodException: No such function
Hi,I'm using dataflow, and I've added UDF which I uploaded in Cloud Storage bucket. The compute service accoun...
Release config ignores custom branch
Not sure what I am doing wrong here. I have a `test` workspace that is pushed to `test` branch in Github. When...
BigQuery datasets containing detailed usage cost data
Hi,"For BigQuery datasets containing standard usage cost data or detailed usage cost data, if I configure the ...
DataStream connection to Google CloudSQL timing out with no error
I am attempting to set up a DataStream from a CloudSQL to CloudStorage. I enter the network and authentication...
The caller does not have permission Cloud Composer
Hello, so currently I want to create cloud composer in service project using shared VPC from host project.I al...
Why does the viewed BigQuery data change after a certain period of time?
The January data I viewed in February is different from the January data I viewed in March.The query is the sa...
"Include advertising identifiers for mobile app streams" is ticked but no identifiers in BigQuery
We've ticked "Include advertising identifiers for mobile app streams" while linking Firebase (GA4) to BigQuery...
Have a dynamic parameter as a JS function
Hello everybody,We are trying to start using Dataform to orquestate and versionize our BigQuery queries but in...
Unable to start de Dataflow job due to : [UNAVAILABLE] HTTP_503
We are having problems to start a new JOB using gcloud commands.After successful creation , the job hangs with...
Dataflow module not found error while using Jupyter Notebook
Team,I just tried to create dataflow job using jupyter notebook available under workbench on Dataflow. If i wa...
Ingest Data from GCS to Big Query. Error - google.api_core.exceptions.Forbidden: 403 POST
I am trying to ingest data from GCS to Big Query. I can do it when I run it locally but when I put my function...
FAILED_PRECONDITION: request failed: Failed to convert table schema to Avro record schema
Hello community,I'm facing below error while trying to read BQ table data using storage read apiFAILED_PRECOND...
How to prevent a Dataform workflow in case a pre defined condition is not met.
Hello Community,I have the following scenario:1. Ingest daily multiple .csv files into designated BQ tables (e...
Access denied while retrieving DDL tables from BigQuery
I am trying to retrieve DDL objects from BigQuery but encountered a failure on line 18 with the code ddl_set =...
Dataform CI CD
Hello Community,I have the following question:In a GCP Dataform workspace lets say you have an .sqlx "operatio...
how I can get the last compilation result id dataform?
Hello, We are migrating from legacy dataform to GCP dataform. Our existing scheduling (in Legacy dataform) is ...
GCP Dataproc class not found error
Hi, I have a pipeline set up, it creates a cluster perform an ingestion and delete a cluster. In success logs ...
BigQuery Time Travel & Fail Safe for Partitioned Tables With Expiration Options
1) How does time travel physical bytes and fail safe physical bytes work for Partitioned BigQuery Tables with ...
Transform a Bigquery table to list of json on dataform
Hello, I have a script on definition/query.sqlx: with data as (SELECT array_agg(struct(obj,val)) as x FROM tab...
live agent call recording
currently our recordings work like this --> virtual agent call is recorded and written to storage bucket once ...
Google Analytics Universal raw data export?
Hi, I was wondering how I would be able to complete a raw data transfer of all of my Google Analytics data to ...
Articles from Product Experts and Googlers
Deeper Insights with knowledge engine & dataset-level metadata in BQ
In my previous blog From Data to Metadata: Unlocking Insights with Gen AI in BigQuery, we explored how BigQuery's “Insights” feature can automatically generate metadata for your tables and columns helping to uncover hidden patterns and trends within your data. We are now taking it a step further with metadata and insights generation at the dataset level. This new capability not only describes the dataset but also provides an interactive entity relationship map that visually reveals connections between different entities.
A developer's guide for building with Anthropic’s Claude 4 models on Vertex AI
We recently announced the general availability of two new models from the latest generation of Anthropic’s Claude model family on Vertex AI: Claude Opus 4 and Claude Sonnet 4. In this blog, we’ll guide you through building with the new Claude 4 models on Vertex AI.
Introducing DeepSeek R1 Model-as-a-service on Vertex AI Model Garden
Deploying Deepseek-R1-0528, a 671B parameters model, typically necessitates at least 8x H200 GPUs for a single API request. Such extensive resources are not accessible to everyone.To make Deepseek R1 more accessible to developers, we're excited to announce that DeepSeek R1 is now available as a fully managed API on Vertex AI in Preview.
