Vertex ai batch prediction job within a Workflows

[G_rubio]

Hi! We are having trouble launching a gemini-flash Vertex ai batch prediction job within a Workflows step. We tried granting different roles to the service account associated to it, and different ways to specify the GCP project and location. If anyone has some experience using vertex ai services and Workflows, or you have any suggestions on how to proceed it would be of great help 😀

Error we receive:
google.api_core.exceptions.PermissionDenied: 403 Permission 'aiplatform.batchPredictionJobs.create' denied on resource '//aiplatform.googleapis.com/projects/PROJECT_ID/locations/LOCATION_ID' (or it may not exist). [reason: "IAM_PERMISSION_DENIED"

Granted roles:
Vertex AI Batch Prediction Service Agent
Vertex AI Custom Code Service Agent
Vertex AI Service Agent
Vertex AI User

Workflow definition:
main:
    params: [args]
    steps:
    - init:
        assign:
     - projectId: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
     - location: ${sys.get_env("GOOGLE_CLOUD_LOCATION")}
    - assignMktList:
        assign:
            - mkt_list:
                - aus
                - aut
                - bel
                - can
    - predictTopicsNewMotivations:
        call: http.get
        args:
            url: '${args.url + "/items/predict_topics_new_motivations/?month=" + args.month}'
            auth:
                type: OIDC
            timeout: 1800
        result: res_predict_topics_new_motivations

[ibaui · Accepted solution]

Hi @G_rubio ,

Welcome to Google Cloud Community!

The error message 403 Permission 'aiplatform.batchPredictionJobs.create' denied indicates that the service account used by your Workflows execution does not have permission to create a batch prediction job in Vertex AI. Even though you've granted several Vertex AI-related roles, the specific role/permission needed for the aiplatform.batchPredictionJobs.create action is likely missing or not correctly configured.

Here are some possible ways to address your issue:

• Identify the Correct Service Account: Determine which service account is running your Workflow. It's likely not the one you've granted roles to, but rather the Workflow service account itself (it can be a default account or one you set).
• Grant the Necessary IAM Role: Give the correct service account the roles/aiplatform.admin (for full access) or roles/aiplatform.batchPredictionJobUser (for specific access to batch jobs) role on your Vertex AI project.
• Verify Project and Location: Ensure that the project and location are being correctly passed to the Vertex AI service when creating the prediction job.

You can also read the following documentation for more details:

• Create a batch prediction job: This documentation explains how to create a batch prediction job in Vertex AI using different interfaces, including the API, and highlights the required parameters.
• Vertex AI API Reference: You can browse the API reference to find the BatchPredictionJob.create method and all the parameters you need to correctly call the API from your workflow.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[G_rubio]

Thank you very much for the response! Indeed, the issue was that we were using cloud run to launch the batch job using python (orchestrated by Workflows). The roles were ok, but we had to give permission to the cloud run service account, not the workflows service account, as it wasn’t directly executing the batch job. After that, it worked flawlessly. Marking as solved!