Is there a way to apply labels to a BigQuery query issued from a Dataflow job?

We are running Apache Beam on Google CLoud Dataflow. One of our jobs SELECTs data from BigQuery and inserts each row onto a Google Cloud Pub/Sub topic. We do this using org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.readTableRows().fromQuery(). Here is a snippet of our code:

 

 

  public static PipelineResult run(Options options) {

    Pipeline pipeline = Pipeline.create(options);

    pipeline.apply("Read from BigQuery query",
        BigQueryIO.readTableRows().fromQuery(options.getBigQuerySql()).usingStandardSql().withoutValidation()
            .withTemplateCompatibility())
        .apply("Map data to PubsubMessage",
            ParDo.of(new MapQueryToPubsub(options.getBigQueryTargetDataset(),
                options.getBigQueryTargetTable())))
        .apply("Write message to PubSub", PubsubIO.writeMessages().to(options.getOutputTopic()));

    return pipeline.run();
  }

 

 

 

The query that we issue against BigQuery can be a substantial one and its one we will be issuing regularly hence I'd like to monitor the cost of that query. The way to monitor the cost of BigQuery queries (as with everything in GCP) is to apply labels to things hence I would like to apply labels to this query however I don't know how to do so via the Apache Beam API(s). Is it possible to attach labels to this query and if so, how?

0 1 2,172
1 REPLY 1

It is possible to add labels via a bq command, or an API call, as per these instructions.

Lower on the same page there is a guide on adding tags, which may also be useful.