403 error on kubeflow custom training pipeline when adding Experiment support

[katadam]

My setup:

I have a kubeflow pipeline that I have used to run custom training jobs in Vertex before. I am using a custom service account with aiplatform.user permissions. 

I am trying to add support for metric logging via Vertex Experiments, but the pipeline fails on the call to aiplatform.init(experiment="name") inside my training script.

The error:

google.api_core.exceptions.Forbidden: 403 GET https://us-central1-aiplatform.googleapis.com/v1/projects/alist-staging/locations/us-central1/metada...: Request had insufficient authentication scopes. [{'@type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'ACCESS_TOKEN_SCOPE_INSUFFICIENT', 'domain': 'googleapis.com', 'metadata': {'service': 'aiplatform.googleapis.com', 'method': 'google.cloud.aiplatform.v1.MetadataService.GetMetadataStore'}}]

What I have tried:

• I have confirmed that my service account has aiplatform.user permissions, which allegedly covers metadataStore. I have added storage.Objects.* permission as well, and confirmed that the project's default compute account has the same permissions.
• I tried to use the same service account to run the job from Google Colab, and was successful! But as soon as I try to replicate this in my kubeflow pipeline, I get permission errors.

My leading theory is that something is wrong with how I am authenticating with aiplatform within my training script, but I can't find any documentation on how to set this up correctly in kubeflow. Please help me, I am so stumped! 

[cassandramae]

Hi @katadam,

Welcome to Google Cloud Community!

It seems that you have insufficient authentication scopes in your service account. As mentioned in the documentation, you need to add https://www.googleapis.com/auth/cloud-platform scope to your custom service account to grant access to other resources.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[grees]

@cassandramae I have read through the documentation you linked and can't find anything about who to add the scope you listed to the service account. 

I can see it says it should be doable so long as you use a Custom Service Account with the relevant permissions (and I am):

Note: If you want your custom training code to obtain an OAuth 2.0 access token with the https://www.googleapis.com/auth/cloud-platform  scope, then you must use a custom service account  for training. You can't give this level of access to the Vertex AI Custom Code Service Agent.

But Im still getting this error trying to log metrics inside Vertex/KFP Pipelines from a CutomTrainingJobOp. 

How can one add the required scope to the custom service account? The docs dont say and google only seems to cover adding scopes to VMs and not the account itself (Vertex pipelines doesnt seem to allow adding it to the CustomTrainingJobOp)?