PII + Dataform in BigQuery – Anyone make this work securely?

[Dovile]

Trying to leverage BigQuery Data Protection features (policy tags, dynamic masking) with Dataform, but hitting two major issues:

1. Policy Tags: Dataform can’t apply policy tags. So if a table is dropped/recreated, tags need to be re-applied separately (e.g., via Cloud Function). Feels brittle and risky.

2. Service Account Access: Dataform execution SA can be selected by anyone in the project. If that SA has access to protected data, users can bypass masking by choosing it. 

Has anyone successfully implemented a secure setup? Would appreciate any insights.

[ruthseki]

Hi @Dovile,

Welcome to Google Cloud Community!

You can set BigQuery policy tags on table columns in the config block in a table definition SQLX file. You may refer to this documentation that shows you how to add BigQuery policy tags to table columns with Dataform core. Also, I found an article that you may find useful which is about utilising Dataform’s config blocks with partition expiry to separate test logic and get billed less at the same time.

With regard to securing service account access, you may apply the Principle of Least Privilege, restrict service account permissions. The Dataform execution service account should have only the permissions it needs to perform its tasks. Do not grant it broad access to all PII data.

• Grant specific IAM roles:

  • roles/bigquery.dataEditor (for writing data)

  • roles/bigquery.dataViewer (for reading data – only to non-PII data)

  • roles/bigquery.jobUser (for running queries)

  • roles/storage.objectViewer (if accessing data from Cloud Storage)

  • Crucially, do not grant roles/bigquery.dataOwner or broader roles unless absolutely necessary.

You may also try to create different service accounts for different Dataform environments (e.g., development, staging, production) with varying levels of access. A development service account might have broader access for testing, while the production service account should be strictly limited.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.