PipelineJob using a dedicated service account

[mans123]

When I submit my pipelinejob (submit)

job = PipelineJob(...)
job.submit(
    service_account='<PII removed by staff>'
)

I get the following error message:

grpc_message:"You do not have permission to act as service_account: <PII removed by staff>. (or it may not exist)."

I am the owner of the project and when I go through the console to create a pipeline job, there is no issue in specifying another service account to run as. It all works if I remove the service_account parameter, so the default account is used. Any idea what the issue can be?

Thanks!

[Joevanie]

Make sure the service account has the appropriate roles and permissions to run your PipelineJob. You can check this by going to IAM & Admin > IAM in Google Cloud Console and verifying that the service account has at least one of these roles: Vertex AI Admin, Vertex AI User, or Vertex AI Custom Code Service Agent

[hpatani]

@Joevanie We are also facing similar issue. The service account does have Vertex AI User permission.

@mans123 was this issue resolved?

[uttasso]

Any update on this? we're facing the same issue here.

[mans123]

Actually, this did work! Fetching service account from the Airflow Variables. Works both in production and when Airflow is running via the personal accounts. 

The accounts that runs Airflow on production (which is also the account that we want to be used in VertexAI) has the following permissions:

[qingvincentyin]

The reason @mans123 solution above worked for him is because the principal has the crucial Editor role shown in his screenshot.

My experiments show that the principal which Python uses needs to have the iam.serviceAccounts.actAs permission to impersonate the SA (let's call it SA #1). Now, when "you" are launching Python from a GCP VM (e.g., a Vertex AI Workbench Instance), "you" are NOT your user principal (i.e., your email address); rather, "you" have impersonated the VM's underlying SA (let's call it SA #2). So, SA#2 must have the iam.serviceAccounts.actAs permission to impersonate SA #1.

Add another twist: if SA #1 and #2 are the same SA, then that SA needs iam.serviceAccounts.actAs permission on itself. Such a need is evidenced from this informational Cloud Logging record:

{
  "protoPayload": {
    ...
    "authenticationInfo": {
      "principalEmail": "my-vertex-ai-workbench-sa@my-xxx-prj.iam.gserviceaccount.com"
    },
    ...
    "methodName": "iam.serviceAccounts.actAs",
    ...
    "request": {
      "@type": "type.googleapis.com/CanActAsServiceAccountRequest",
      ...
    },
    ...

Without granting the "self-impersonating" permission, I got the following error. As you see, the authenticationInfo.principalEmail (SA #1) is trying to impersonate the 

request.pipelineJob.serviceAccount (SA #2) -- in my case, the 2 SAs are the same SA.

{
  "protoPayload": {
    "status": {
      ...
      "message": "You do not have permission to act as service_account: my-vertex-ai-workbench-sa@my-xxx-prj.iam.gserviceaccount.com. (or it may not exist)."
    },
    "authenticationInfo": {
      "principalEmail": "my-vertex-ai-workbench-sa@my-xxx-prj.iam.gserviceaccount.com",
      ...
    },
    ...
    "methodName": "google.cloud.aiplatform.v1.PipelineService.CreatePipelineJob",
    ...
    "request": {
      "pipelineJob": {
        "serviceAccount": "my-vertex-ai-workbench-sa@my-xxx-prj.iam.gserviceaccount.com"
      },
      ...
    },
    ...

So, the solution is simple:

• Grant the IAM role Service Account User (roles/iam.serviceAccountUser) from SA #1 to SA #2. If they are the same SA, then grant to itself (yeah, a bit twisted.)