PubSub to BigQuery Sink - Has Anyone Been Able to Make It Work?

[JPowell]

Hello,

Maybe I'm missing something, but I cannot get PubSub Subscription to BigQuery sink to work using directions such as https://cloud.google.com/pubsub/docs/bigquery.

- Created a topic that has a Service Account SA
- Created a dataset and a couple of tables that are shared at the dataset level to SA
- Created a subscription with multiple configs that match the table schemas
- Added IAM permissions for the SA (BQ Editor, Metadata Viewer)

All topics and subscriptions are in the green state and there are no log messages, and there is no data in the tables after submitting messages manually, even after waiting. [We tried everything in the troubleshooting page: https://cloud.google.com/pubsub/docs/bigquery-troubleshooting].

Any ideas?

Best,

Jon

[Aris_O]

Hi @JPowell,

Welcome back to Google Cloud Community.

Here are some additional checking you may try to perform.

• Verify that the subscription is actually receiving messages by looking at the subscription's metrics in the Cloud Console. Select your subscription in the Cloud Console's Pub/Sub section before choosing "Metrics" from the left-hand menu. If no messages are being received, your publisher code may be the problem.
• Verify whether the Cloud Functions are working properly: whether you're using Cloud Functions to handle the messages and write them to BigQuery, check sure they are deployed and configured correctly. Examine the logs for any mistakes or problems.
• Verify that the BigQuery table's schema and the message schema that are being published to the topic match. The messages won't be posted to the table if the schema of the messages being published conflicts with the schema of the BigQuery table.
• Verify the Cloud Logging for issues: You can look for any errors relating to the Pub/Sub Subscription to BigQuery sink in the Cloud Logging. Search for the sink name or the subscription name in the Cloud Console's Logging section. This is where any errors should be recorded.

[jldupont_sora]

@Aris_O I am having an issue with BigQuery subscription and reading & following your 4 points above I triple checked Cloud Logging: there is no trace of an anomaly. Is there a setting somewhere for this ? I could not find a way to enable Cloud Logging as you elude to. 

Thanks for your help.