How to send http request to discovery engine API within a Google Cloud Run app?

[a-techt]

Hi,

I am using vertexai agentbuilder to parse pdf files and get processed chunks. For this  I need to send HTTP request to discoveryengine (v1 alpha) API. It is not possible to use a python client for this. Here is the code I am using:

curl -X GET \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    "https://discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/global/collections/default_collection/dataStores/DATA_STORE_ID/branches/0/documents/DOCUMENT_ID/chunks:getProcessedDocument?processed_document_type=CHUNKED_DOCUMENT"

Link: https://cloud.google.com/generative-ai-app-builder/docs/parse-chunk-documents#list-chunks

This code works well when I use it locally. Here is the code I use for this:

        credentials = service_account.Credentials.from_service_account_file(
            find_dotenv('my-google-service-key.json'),
            scopes=['https://www.googleapis.com/auth/cloud-platform']
        )
        credentials.refresh(Request())
        print("Credentials loaded from local file")
        
        # Make the request with params
        headers = {
            'Authorization': f'Bearer {credentials.token}',
            'Content-Type': 'application/json'
        }
               
        response = requests.get(url, headers=headers, params=params)

I couldn't manage to use the same code if used within an fastapi app deployed on GoogleCloudRun. I upload my service account key json file to SecretManager. And then within GoogleCloudRun app I use the following code to do that same thing:

       client = secretmanager.SecretManagerServiceClient()
        name = f"projects/{project_id}/secrets/ServiceAccountKey/versions/latest"
        response = client.access_secret_version(request={"name": name})
        credentials = service_account.Credentials.from_service_account_info(
            json.loads(response.payload.data),
            scopes=['https://www.googleapis.com/auth/cloud-platform']  # Added required scope
        )
        try:
            credentials.refresh(Request())
            credentials_error="No error for credentials"
        except Exception as e:
            credentials_error=e
            print(f"Error refreshing credentials: {e}")
        headers = {
            'Authorization': f'Bearer {credentials.token}',
            'Content-Type': 'application/json'
        }
        response = requests.get(url, headers=headers, params=params)
        

But this gives me the following error:

"detail": "Error: API request failed: 401 - {\n  \"error\": {\n    \"code\": 401,\n    \"message\": \"Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.\",\n    \"status\": \"UNAUTHENTICATED\",\n    \"details\": [\n      {\n        \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n        \"reason\": \"ACCESS_TOKEN_TYPE_UNSUPPORTED\",\n        \"metadata\": {\n          \"method\": \"google.cloud.discoveryengine.v1alpha.DocumentService.GetProcessedDocument\",\n          \"service\": \"discoveryengine.googleapis.com\"\n        }\n      }\n    ]\n  }\n}\n"

How can I successfully send HTTP request to discovery engine API when the app is deployed on GoogleCloudRun?

[dawnberdan · Accepted solution]

Hi @a-techt,

Welcome to Google Cloud Community!

The error "Request had invalid authentication credentials. Expected OAuth 2 access token..." suggests that while you're retrieving the service account key, it isn't being used correctly in the Cloud Run environment to generate a valid OAuth 2 access token for the Discovery Engine API. Even though you're refreshing the token, the issue likely lies in how the credentials are applied within Cloud Run.

Your app correctly fetches the service account key, but the behavior of the google.oauth2.service_account library differs in Cloud Run compared to your local setup. It may not be recognizing the necessary environment variables or metadata for proper authentication. The "ACCESS_TOKEN_TYPE_UNSUPPORTED" error points to a problem with how the token is being passed to the Discovery Engine API, rather than with the token itself.

The most secure and recommended solution is to use Workload Identity Federation. This approach eliminates the need to manage service account keys in your application, significantly reducing security risks. Here's how you can implement it:

1. Grant Permissions: In the Google Cloud Console, navigate to the IAM page for your project. Create a new service account. Grant this service account the necessary role to access the Discovery Engine API.
2. Enable Workload Identity Federation: In your Cloud Run service's settings (in the Cloud Run console), enable Workload Identity Federation. Configure it to use the service account you created in step 1. This essentially links your Cloud Run service's default service account to your Discovery Engine service account.
3. Modify your FastAPI code: After enabling Workload Identity Federation, you should be able to remove almost all of your authentication code within your FastAPI application. Cloud Run will handle authentication automatically. You'll just make the request as if you were already authenticated.
4. IAM Roles: Ensure you've assigned the least privilege necessary. Don't give your service account unnecessary permissions.

By implementing Workload Identity Federation, you may solve the authentication problem securely and simplify your code significantly. The 401 error will disappear because the authentication will be handled automatically and correctly by the Cloud Run environment.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[MarkedSunholo]

The solution works, but I think workload identity federation is for when your service is off GCP such as AWS etc.

It’s the same steps (adding permission to the service account running Cloud Run) but this all happens for free using default application logins AFAIK. 

a small detail but I thought I would ask for clarification since it may confuse other answers.