Unable to Execute Dataform Workflow due to "IAM permission denied for service account"

[so0710]

Hello

I've been facing an issue trying to execute a workflow in Dataform. Every time I attempt to run it, I'm met with an "IAM permission denied for service account your-service-account-name" error.

Here's a summary of the steps I've taken to troubleshoot:

• Ensured the following permissions have been granted: BigQuery Data Editor, BigQuery Data Viewer, BigQuery Job User, and BigQuery Data Owner. Still facing the issue.
• Tried granting BigQuery Administrator and Dataform Administrator roles.
• Attempted to run the default template provided during repository initialization post-creation, but the same error persisted.
• Verified that I have Owner permissions on my own account, yet the problem remains.

I'm at a bit of a loss here, and would greatly appreciate any insights or advice from anyone who may have encountered a similar issue or has any suggestions to offer.

Thank you in advance for your assistance.

[ms4446]

Here are a few things to check:

• Ensure Correct Service Account Role:   - Make sure that the service account has the correct role. The Dataform service account needs the BigQuery Data Editor, BigQuery Data Viewer, BigQuery Job User, and BigQuery Data Owner roles to execute workflows. Verify the service account's role in the IAM & Admin > Roles page in the Google Cloud console.

• Verify Dataset and Table Permissions:   - The service account must have read and write permissions on the BigQuery dataset and table it's trying to access. Verify these permissions on the BigQuery page in the Google Cloud console.

• Review VPC Service Controls:   - If using VPC service controls, ensure the Dataform service account has access to the BigQuery API. Verify this in the Network Services > VPC service controls page in the Google Cloud console.

• Test with a Different Project:   - Try running the workflow from a different project to determine if the issue is related to the project or the service account.

• Service Account Verification:   - Ensure the service account is active and not disabled.

• Check Organization Policy:   - Review any organization policies that might be affecting service account permissions.

• Examine Audit Logs:   - Review the audit logs for additional information about the permission denied error.

• Ensure Valid Service Account Key:   - Make sure the service account key used by Dataform is valid and not expired or revoked.

Additional Troubleshooting Tips:

• Review Dataform Logs:   - The Dataform logs may contain more information about the error. View the logs in the Google Cloud console on the Logging page under the "Logs Explorer" tab.

• Run Workflow in Debug Mode:   - Enable debug mode to step through the workflow and identify where the error is occurring. Set the DEBUG environment variable to true.

• Execute Workflow from Command Line:   - This may help identify the specific IAM permission that is being denied. Use the following command:       
   $ dataform run workflow.yaml

[so0710]

Thank you for the reply.

As you advised, when I executed it via the Command Line, it worked without any issues. I used the service account's JSON for the df-credentials.json.

However, when I try from the dashboard, I still get denied due to permissions. I suspect it might be something with the project's settings, so I'll review that.

Thanks for your help.

[ms4446]

I'm glad to hear that you were able to get your Dataform workflow to work by running it from the command line. This indeed suggests that the problem might be with the permissions granted to the service account when used from the dashboard.

Here are a few things to check in the project's settings:

• Ensure Correct Service Account Role: Make sure that the service account used by the dashboard is granted the correct roles (BigQuery Data Editor, BigQuery Data Viewer, BigQuery Job User,and BigQuery Data Owner) to execute workflows. Verify this in the IAM & Admin > Roles page in the Google Cloud console.
• Verify Dataset and Table Permissions: The service account should have read and write permissions on the BigQuery dataset and table it's trying to access. Confirm these permissions on the BigQuery page in the Google Cloud console.
• Review VPC Service Controls: If using VPC service controls,ensure the Dataform service account has access to the BigQuery API. Check this in the Network Services > VPC service controls page in the Google Cloud console.
• Dashboard and Browser Issues:Ensure the dashboard is using the correct service account and check for any browser-related issues. Try using a different browser or clearing the cache.

If you've checked all of the above and you're still having problems, run the following command for a more detailed log:

$ dataform run --debug workflow.yaml

This will output a more detailed log of the workflow execution, which may include information about the specific permission that is being denied.

If issues persist, feel free to provide more information about your Dataform workflow and the error message you're receiving. I'm here to help troubleshoot the issue further, and don’t hesitate to reach out to Google Cloud Support or Dataform support for additional assistance.

Additional Troubleshooting Tips

• Try restarting the Dataform dashboard.
• Check the Dataform logs for any additional information about the error.
• Try running the workflow from a different project.
• Try running the workflow from a different user account.

[so0710]

Thank you for your response.

Ensure Correct Service Account Role: Make sure that the service account used by the dashboard is granted the correct roles (BigQuery Data Editor, BigQuery Data Viewer, BigQuery Job User,and BigQuery Data Owner) to execute workflows. Verify this in the IAM & Admin > Roles page in the Google Cloud console.

Yes, I checked, and all the necessary permissions were granted.

Verify Dataset and Table Permissions: The service account should have read and write permissions on the BigQuery dataset and table it's trying to access. Confirm these permissions on the BigQuery page in the Google Cloud console.

They were granted as well. The BigQuery administrator permissions were also in place.

Review VPC Service Controls: If using VPC service controls, ensure the Dataform service account has access to the BigQuery API. Check this in the Network Services > VPC service controls page in the Google Cloud console.

I wasn't using VPC services.

Dashboard and Browser Issues: Ensure the dashboard is using the correct service account and check for any browser-related issues. Try using a different browser or clearing the cache.

I cleared the cache and tried with both Chrome and Firefox, but the result was the same.

I apologize for my lack of experience with Dataform, but I haven't created a workflow.yaml file. Additionally, when I tried running the command you provided, I got an error:

$ dataform run --debug workflow.yaml Dataform encountered an error: Unknown argument: debug

The steps I followed are as below:

1. Issued a key with the service account and renamed the issued json file to df-credentials.json.

2. Set the environment variable using

$ export GOOGLE_APPLICATION_CREDENTIALS="/path/to/.df-credentials.json"​

3. When I execute `dataform run`, it completes without issues. However, when I check the job execution history in BigQuery, it appears as though the job was executed with my company account, not the service account.

4. I then logged out using `gcloud auth revoke`.

5. After that, when I ran `dataform run` again, the job history showed that it was executed by the service account.

[ms4446 · Accepted solution]

It seems like you have done a thorough job checking the permissions and trying different browsers. Your steps for executing via the command line are correct, and it's good to see that it works with the service account when you revoke your personal credentials.

Regarding the error with the --debug flag, it seems there might be a version difference or it might not be supported in the command you are using. You can check the documentation or use dataform --help to see the available options for debugging.

For the issue with the dashboard, consider the following:

1. Dashboard Service Account Configuration:

   • Double-check if the dashboard is configured to use the correct service account. Sometimes, the dashboard might use a different service account or credentials, leading to permission issues.

2. Project Settings in Dashboard:

   • Review the project settings within the Dataform dashboard. Ensure that the project is linked to the correct Google Cloud project and that the service account configuration in the dashboard matches the one you are using in the command line.

3. Additional Permissions:

   • There might be additional permissions or roles needed when executing from the dashboard. Check the documentation of Dataform to ensure all necessary permissions are granted.

4. Dataform Support:

   • Since the issue is specific to the dashboard, reaching out to Dataform support might provide more insights. They can help verify the dashboard configuration and ensure it is set up correctly to use the service account.

5. Debugging in Dashboard:

   • Look for any debugging or log options within the Dataform dashboard. It might provide more detailed error messages or information about the permission denied error.

By exploring these areas, you might be able to identify and resolve the issue with executing workflows from the Dataform dashboard.

[so0710]

In the end, I was able to confirm that it works with the default service account on the dashboard! I'm not sure why the service account I created was denied permissions...

For now, I'll consider this issue resolved.

Thank you very much for your assistance.

[pfilourenco]

Wasn't this configuration missing? :
"Additionally, you need to grant the default Dataform service account Service Account Token Creator(roles/iam.serviceAccountTokenCreator) access to any non-default service accounts that you want to use in Dataform."

[frans-snyders]

in the end what worked for us to solve the error "IAM permission denied for service account"

default service account

[dataform service agent]
roles/secretmanager.secretAdmin
roles/iam.serviceAccountViewer
roles/iam.serviceAccountOpenIdTokenCreator
roles/iam.serviceAccountTokenCreator

custom service account

roles/bigquery.dataOwner (this can be reduced)
roles/bigquery.jobUser
roles/dataform.admin
roles/secretmanager.secretAccessor
roles/iam.serviceAccountUser

[otassel]

After several tests, we just need to :

• add roles/iam.serviceAccountTokenCreator to the default service account
• configure the custom service account with roles/bigquery.jobUser, roles/bigquery.dataViewer (or roles according to your objectives)
  
  

[Paggie]

This helps:

Grant token creation access to a non-default service account 

[lekeshkumar]

I might be too late but I solved this issue by providing the Default Dataform Service Account the role "iam.serviceAccountTokenCreator". This allows the default SA to create token for our preferred service account in order to perform the workflow related tasks.

Refer: https://cloud.google.com/dataform/docs/required-access#dataform-required-roles