Solved: Re: 'iam.serviceAccounts.getAccessToken' Permissio...

noll_6451 · 07-04-2024 05:09 AM

Issue Summary:

I'm trying to create a Google Cloud project using Terraform Cloud and Workload Identity. While terraform plan succeeds, terraform apply fails with the following error:

Error: failed pre-requisites: failed to check permissions on billing account "billingAccounts/<billingAccounts>": Post "https://cloudbilling.googleapis.com/v1/billingAccounts/ <billingAccounts>:testIamPermissions?alt=json&prettyPrint=false": oauth2/google: status code 403: { "error": { "code": 403, "message": "Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).", "status": "PERMISSION_DENIED", "details": [ { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "reason": "IAM_PERMISSION_DENIED", "domain": "iam.googleapis.com", "metadata": { "permission": "iam.serviceAccounts.getAccessToken" } } ] } }

Confirmed Checks:

The Billing Account is active, and the necessary APIs (Cloud Resource Manager API, Cloud Billing API) are enabled.
The service account used in Terraform Cloud has the following permissions:
- Project Creator (at the folder level)
- Billing Account User (at the billing account level)
- Service Account Token Creator
- Workload Identity User
- Billing Project Manager
The Workload Identity Pool and Provider are correctly configured and linked to the Terraform Cloud workspace.
The Terraform configuration file (main.tf) is as follows:

Terraform

resource "google_project" "main" {
  name            = var.project_name
  project_id      = var.project_id
  folder_id       = var.folder_id
  billing_account = var.billing_account
  auto_create_network = false
}

Workload Identity federation is enabled in the Terraform Cloud workspace settings.
Terraform is running the latest version.

Troubleshooting Attempts:

Reviewed Terraform Cloud execution logs and Google Cloud audit logs, but no useful information was found.
Double-checked Terraform Cloud workspace settings, but no issues were found.
Retried terraform apply after some time, but the error persisted.

Question:

Given this information, what are the possible causes and solutions for this error?

Providing this additional information would be helpful in getting more precise advice.

noll_6451

Shows how it was resolved

It was a mistake to set the Workload Identity User at the project level.

https://cloud.google.com/iam/docs/workload-identity-federation?hl=ja#impersonation

As it says here, giving a service account a Workload Identity User at the project level will result in borrowing the privileges of the service account.

If you set up both direct access to the resource and borrowing of the service account, the borrowing is applied first, and then the service account borrows its own privileges, which is what I think happened.

Here is my case

Shows the permissions of the service account when it is wrong

- Project Creator (at the folder level)
- Billing Account User (at the billing account level)
- Service Account Token Creator (at the project level )
- Workload Identity User (at the project level )
- Billing Project Manager (at the folder level)

Principals with access to the service account in case of mistakes (only where relevant)

- principal://iam.googleapis.com/projects/PROJECTID/locations/global/workloadIdentityPools/POOLNAME/* (Workload Identity User)

In this case, the correct service account permissions

- Project Creator (at the folder level)
- Billing Account User (at the billing account level)
- Service Account Token Creator (at the project level)
- Billing Project Manager (at the folder level)

If you find any mistakes, please let us know.

View solution in original post

noll_6451

I have resolved this.

Workload Indetity user was given in the wrong place.

The service account itself was borrowing privileges from the service account.

salvadorlopez

Hi, thanks for posting this, I am seeing the same problem, but I don't completely understand your solution,

I have setup the workload identity federation to work with terraform cloud(HCP) and it works, I can create resources on GCP after executing my workspace in HCP, the problem is that now I need to create a google project and a firebase project, and after running terraform, I always get the same error, this is the resource I am trying to create

resource "google_project" "default" {
  provider = google-beta.no_user_project_override
  name       = var.google_project_name
  project_id = var.google_project_id
  # Required for any service that requires the Blaze pricing plan
  # (like Firebase Authentication with GCIP)
  billing_account = var.billing_account
  # Required for the project to display in any list of Firebase projects.
  labels = {
    "firebase" = "enabled"
  }
}

service account:dev-hcp-service-account@PROJECTID.iam.gserviceaccount.com

Organization Level Permissions	Billing Account User Project Billing Manager Project Creator
Billing Account Level permissions	Billing Account Administrator

I have tried many different changes but none of those work, I always get the same error

failed to check permissions on billing account

not sure what change I need to make, or where to make the changes.

could You please detail a little bit on how you solved this ? thanks in advance.

AndyClausen

Hey, this sounds like a completely different issue. It _might_ be that your workload identity service account doesn't have access to the specified billing account though.

noll_6451

What I want to do in your case and mine is different, so it could possibly be different,

Are you granting Workload Identity User to the principal of the pool you are linking to?

AndyClausen

I feel like the response here is a bit ambiguous, so I'll post what worked for me when connecting to OIDC in Vercel:

Go to IAM > Service Accounts and click the Service Account you've linked with Workload Identity Pool
Go to Permissions and edit the Workload Identity Pool principal(s)
Ensure roles/iam.serviceAccountTokenCreator is on the list (or custom role with similar permissions)

Hope this helps 🙂

noll_6451