Dataform - Custom Service Account Billing Issue Across Multiple Projects

[datauser222]

Hello,

I am currently using Google Cloud Dataform and would like to run workflows with different custom Service Accounts, specifying a different Service Account for each workflow execution. However, I’ve encountered an issue: even when using a non-standard Service Account, the billing project defaults to the project where Dataform is hosted, instead of the billing project associated with the SQL resources accessed by each Service Account.

Here’s what I’ve tried so far:

• Created multiple Service Accounts with specific permissions for different datasets and projects.
• Specified the Service Account directly in the workflow configuration.
• Attempted to define execution parameters to control billing dynamically.

Despite these efforts, the billing consistently defaults to the Dataform host project, not the intended project for each Service Account.

In my setup, I have a primary project where I manage the Dataform repository and configuration, and I also have 10 different client projects. My goal is to have the query costs billed to the respective client projects based on the specific query and its configuration for each execution. This would allow each client project to be billed accurately depending on the usage.

Is there a way to dynamically set the billing project based on the Service Account or another method to achieve this? If this isn’t currently possible, I’d like to suggest this as a feature enhancement to support more flexible billing configurations (add billing project id to InvocationConfig).

Thank you for any guidance or suggestions on this issue!

[caryna]

Hi @datauser222,

Welcome to Google Cloud Community!

Upon reviewing the situation, I can confirm that this issue was previously submitted as a feature request and has already been addressed by our Engineering Team. You can review the suggested solution to help resolve the issue.

If this approach doesn't resolve your case, you can still file an issue/feature request regarding this. You may also submit your suggested enhancement to add a billing project id to InvocationConfig so that our Engineering Team can investigate further. Before filing, please take note on what to expect when opening an issue.

I hope the above information is helpful.