Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Dataflow templates - Pub/Sub topic to BigQuery

I am trying to create the streaming pipeline using Dataflow templates (used cases in GCP). The pipeline is created using pub/sub topic but the result is not reflecting the BigQuery table and there are no errors.

0 1 286
1 REPLY 1

As mentioned in the documentation, your pub/sub messages should be in a JSON format. As an example, messages formatted as "k1":"v1", "k2":"v2" would need to be inserted into BigQuery in two different columns, k1 and k2, with string data type. And also the output table must exist before running the pipeline; furthermore, the table schema would need to match the input JSON objects.