Data at Google Cloud Next '24!
Answer this question in the comments to earn a special badge: In your journey, what's one overlooked data skil...
Forum Posts
Influence Looker’s Roadmap and join the BI Customer Council (BICC)
Would you like to influence Google Cloud’s BI roadmap more? I’d like to invite you to join the Google Cloud BI...
Spark Streaming with PubSub Lite
I want to create a Spark Streaming application in DataProc that has a subscription to PubSub Lite or PubSub (I...
google busiiness pub sub
Hello I'm trying to be notified when a new review is added on my Google Business Profile or updated profilleA...
Nested Pubsub message to Bigquery
Is it possible to flatten the nested pubsub payload and insert to bigquery?Eg payload : {"data": [ { "field1":...
Issue testing PubSub to BigQuery deadletter settings
Hello! I am working with PubSub and BigQuery and was trying to test out deadletter settings to ensure messages...
Bigquery streaming buffer taking too long (90 minutes each time)
I am streaming data from pub/sub to bigquery using dataflow templates via this code [1]. Data is typically ava...
Can I run the BigQuery notebook on a Pub/Sub message or CRON?
Hi there,Is it possible to run the BigQuery notebook on a Pub/Sub message or CRON? I presume that i should be ...
PubSub permissions error while publishing using google cloud dependencies
Hello community!We are switching from spring based pubsub dependencies to Google cloud's pubsub dependency by ...
Http request call to PubSub asks for "content-length" in header, but when provided times out
Hi there,I am able to run http requests to my PubSub project no problem from python scripts. I am now testing ...
PubSub to BQ - Dataflow Standard Templates to upsert the records
Would like to know whether we have standard pubsub to BQ dataflow template to insert the record if new and upd...
Event Filter on "id" attribute of google.cloud.storage.object.v1.finalized event
Hi!I am looking to add an event filter to a cloud event trigger that filters on the id attribute of the google...
Dynamic configuration management service in GCP
Hello Team,is there any Dynamic configuration management service in GCP similar to AWS AppConfig to manage and...
NET client SubscriberServiceApiClient.Pull method to return immediately if no messages on topic
Using the NET client API, how is it possible to achieve the same functionality with SubscriberServiceApiClient...
Getting location info updates from google business using pub/sub
I've setup the Pub/Sub in order to receive update from Google My Business.So far, I've experienced decent resp...
Pub/Sub to BigQuery in different project
Hello,I am attempting to transfer data from a Pub/Sub subscription in Project A to BigQuery in Project B. I gr...
What can cause pub/sub stops sending message to subscriber?
Sorry if this wrong place to ask. I have async golang microservice that subscribe to pub/sub topic i use, afte...
Potential solutions for exporting GA4 raw data from BQ to Snowflake(When GA4 raw data loaded in BQ)?
Hi there,I am looking for some advice here. I thought of the following solutions for exporting GA4 raw data fr...
Can we increase PubSub subscription filter max length of 256 bytes.
Hi,Can we increase PubSub subscription filter max length of 256 bytes ?As per below link it looks like max all...
Cloud function error Cannot read properties of undefined (reading 'jobCompletedEvent')
Hi there, I got an error when the Cloud Function executed. run_scheduled_query_ga4_daily_master TypeError: Can...
Cloud PubSub Subscription Exactly once delivery not wokring
hi allI am using Pub/Sub and my Subscription settings are as followsAcknowledgement deadline : 60 secondsExact...
Pubsub BigQuery Subscription Incompatible type for enum field
I'm attempting to stream a PubSub topic to a BigQuery table using the PubSub BigQuery subscription. Here's wha...
pubsub emulator: how to set up dead-letter queue
Hi, "dead-letter queues" have been supported by the emulator since July 2021: https://cloud.google.com/sdk/doc...
Pub/sub subscription not sending out all messages at once
Hi all, I have a question re: pub/sub.My set up is as follows- I have a publisher creating about ~3000 message...
Can Pubsub Schema refer to previously declared schema
Hi,If we have Department schema as below and Employee has Department.So can we use and refer to Department sch...
GCPPub
Our application is using GCP PubSub.We have setup the consumer asspring.cloud.gcp.pubsub.function.definition= ...
Insert/Update to GCP Bigquery using Spring boot
Hi @, I need to insert/update data received from another saas based application to GCP bigquery, could you sug...
Issue on Automating Snowpipe for Google Cloud Storage
When my trying to create pipe via below command ,I am getting error Command: CREATE OR REPLACE PIPE GCS_SNOWPI...
Does Google Config Connector Support PubsubSchema Evolution ?
Does Google Config Connector Support PubsubSchema Evolution ?If yes then can you update the ConfigConnector do...
Pubsub Schema creation from .avsc file
Can we maintain schema registry for Pubsub in GCS ? If possible please share example or documentation link for...
Simplify data ingestion to BigQuery
Hello BQ users! We recently published a step by step guide, which walks through how you can simply extract and...
Facing issue while publishing Instant.now() (in java code) on google pubsub topic
Hi Team,Please let me know workaround for this or let me know if I am missing anything. { "eventDate": "2023-0...