Encountered 403 Forbidden : Service Account calling Google APIs using AWS Lambda

[Tonanate]

I'm new to Google APIs so My question deals with creating an AWS lambda python integration that will call Google Campaign Manager 360 DFA Reporting scope. It will take the report data and place it in an AWS S3 bucket for SA analysts to run BI and Qlik reports  (I am not on Google Workspace so I can't do delegations).

Since this is an integration app calling a service, I decided to use Service Account because its used for  server-to-server interactions.

1. I Set up Service Account giving my self owner privileges and the account has user privileges.
2. Copied the service account Json file to be used by Python and Google  API.

I'm not sure what to do at this point and what could be wrong. Thanks much for your help. Let me know if you need any more images like my service account setup.

Here is my error message and code for the lambda python integration where execution fails at line 20 (I would expect an issue with OAuth2.0 credentials but not with Service Account. I'm new to all of this) :

{

    "error": {

        "code": 403,

        "message": "Request had insufficient authentication scopes.",

        "errors": [

            {

                "message": "Insufficient Permission",

                "domain": "global",

                "reason": "insufficientPermissions"

            }

        ],

        "status": "PERMISSION_DENIED",

        "details": [

            {

                "@type": "type.googleapis.com/google.rpc.ErrorInfo",

                "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",

                "domain": "googleapis.com",

                "metadata": {

                    "service": "dfareporting.googleapis.com",

                    "method": "google.ads.xfa.dfareporting.op.v4.DfareportingCampaigns.List"

                }

            }

        ]

    }

}

import json
import os
from google.oauth2 import service_account
from googleapiclient.discovery import build
import googleapiclient.discovery

SCOPES = ['https://www.googleapis.com/auth/dfareporting', 'https://www.googleapis.com/auth/ddmconversions']
delegate = os.environ['DELEGATE_EMAIL']
profile_id = os.environ['PROFILEID']
def lambda_handler(event, context):
   
    SERVICE_ACCOUNT_FILE = 'service.json'
    print(SERVICE_ACCOUNT_FILE)
    # Implement Credentials objects from the service account and scopes
    credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
    print('<< CREDENTIAL >> ', credentials)
    #delegated_credentials = credentials.with_subject(delegate)
    gcm360_client = build('dfareporting', 'v4', credentials=credentials)
    request =gcm360_client.campaigns().list(profileId=profile_id)
    print('<< REQUEST >> ',request)
    response = request.execute()
    #print('<< RESPONSE >> ',response)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

When I test ran it, I got the following

Error on Http.Request

[Roderick]

Hi there,

Based on the errors you referenced, the issue you're facing seems to be related to the authentication process using the Google Cloud service account. Here are a few things you can check and adjust in your code:

1. Verify the Service Account JSON File: Double-check that the service.json file is correctly located in the same directory as your Python script. Confirm the file name and extension are accurate.

2. Check Environment Variables: Ensure that the DELEGATE_EMAIL and PROFILEID environment variables are correctly set and accessible within your Lambda function. Make sure they match the expected values for the delegate email and profile ID you want to use.

3. Adjust the SCOPES: Verify that the scopes you have specified (https://www.googleapis.com/auth/dfareporting and https://www.googleapis.com/auth/ddmconversions) are appropriate for your use case. Make sure they align with the permissions you have set for your service account.

4. Delegate Access (Optional): If you don't have access to delegate permissions within Google Workspace, you can remove the line related to delegated credentials (delegated_credentials = credentials.with_subject(delegate)).

5. Enable Required APIs: Ensure that you have enabled the required APIs (Campaign Manager 360 and DDM Conversions) in the Google Cloud Console. This step is necessary for accessing the respective services.

6. Handle Dependencies: Make sure you have the necessary dependencies installed for the google.oauth2 and googleapiclient.discovery modules. You can include them in your Lambda deployment package or use a package management system like pip to manage dependencies.

7. Error Logging: Consider adding error logging statements to capture any potential exceptions or error messages. This can help you identify the specific issue causing the execution failure.

By checking and adjusting these aspects, you can troubleshoot and determine the root cause of the issue you're facing with the authentication and service account setup.

[Tonanate]

HI Roderick_G. Thanks for the response. I did find out something after
performing these checks:

1. I verified that the service.json file was located in the same
directory as my python script.
2. I checked my environment variables but I really don;t need the
Delegete_Email variable because I don't have delegate access.
3. The Delegate Access statement was already commented out.
4. I made sure that the scopes are all enabled because both of the two
mentioned are under the campaigns 360 API for campaigns method.
5. I tested the Profile ID when testing the Google Explorer for the
desired APIs for listing campaigns and that brought in data.
6. I should have all the google-api-python-client libraries available.

Here's what I did to see if I could verify accessing my scopes:

1. Set them to "readonly" (https://
www.googleapis.com/auth/dfareporting.readonly)
2. Reran the script and verified that I now saw this error:

"errorMessage": "('No access token in response.', {'id_token':
'eyJhbGciOiJSUzI1NiIsImtpZCI6Ijg1YmE5MzEzZ.......'})",
"errorType": "*RefreshError*",
Q) Could it be that the server was trying to refresh the token and failed?
Q) Is there something that I'm missing concerning refreshing of tokens for
Google Service Accounts or has my current code accounted for that but I may
be missing something logically?

Thanks

[Tonanate · Accepted solution]

After unsuccessfully trying to use the Service Account method, I had to use a mostly non-Google Rest-based alternative that incorporates some Google.

---

@Tonanate wrote:

I'm new to Google APIs so My question deals with creating an AWS lambda python integration that will call Google Campaign Manager 360 DFA Reporting scope. It will take the report data and place it in an AWS S3 bucket for SA analysts to run BI and Qlik reports  (I am not on Google Workspace so I can't do delegations).

Since this is an integration app calling a service, I decided to use Service Account because its used for  server-to-server interactions.

1. I Set up Service Account giving my self owner privileges and the account has user privileges.
2. Copied the service account Json file to be used by Python and Google  API.

I'm not sure what to do at this point and what could be wrong. Thanks much for your help. Let me know if you need any more images like my service account setup.

Here is my error message and code for the lambda python integration where execution fails at line 20 (I would expect an issue with OAuth2.0 credentials but not with Service Account. I'm new to all of this) :

{

    "error": {

        "code": 403,

        "message": "Request had insufficient authentication scopes.",

        "errors": [

            {

                "message": "Insufficient Permission",

                "domain": "global",

                "reason": "insufficientPermissions"

            }

        ],

        "status": "PERMISSION_DENIED",

        "details": [

            {

                "@type": "type.googleapis.com/google.rpc.ErrorInfo",

                "reason": "ACCESS_TOKEN_SCOPE_INSUFFICIENT",

                "domain": "googleapis.com",

                "metadata": {

                    "service": "dfareporting.googleapis.com",

                    "method": "google.ads.xfa.dfareporting.op.v4.DfareportingCampaigns.List"

                }

            }

        ]

    }

}

import json
import os
from google.oauth2 import service_account
from googleapiclient.discovery import build
import googleapiclient.discovery

SCOPES = ['https://www.googleapis.com/auth/dfareporting', 'https://www.googleapis.com/auth/ddmconversions']
delegate = os.environ['DELEGATE_EMAIL']
profile_id = os.environ['PROFILEID']
def lambda_handler(event, context):
   
    SERVICE_ACCOUNT_FILE = 'service.json'
    print(SERVICE_ACCOUNT_FILE)
    # Implement Credentials objects from the service account and scopes
    credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
    print('<< CREDENTIAL >> ', credentials)
    #delegated_credentials = credentials.with_subject(delegate)
    gcm360_client = build('dfareporting', 'v4', credentials=credentials)
    request =gcm360_client.campaigns().list(profileId=profile_id)
    print('<< REQUEST >> ',request)
    response = request.execute()
    #print('<< RESPONSE >> ',response)
    return {
        'statusCode': 200,
        'body': json.dumps('Hello from Lambda!')
    }

When I test ran it, I got the following

Error on Http.Request

---

I was able to pull data using a get request and load that data into an S3 bucket using the google API urls for Campaign Manager 360. However, I had to constantly use Postman to give me an access token. This code is going to run daily and I need to have a way to generate them for my Service Account. 

Now, I need to generate tokens for my Service Account. So we requested from our client to give us domain-wide delegation to run this. I know that Google indicates that there are three approaches but the other 2 are the least desirable at this point.

It seems to use info from the Google Page here:

https://developers.google.com/identity/protocols/oauth2/service-account#jwt-auth

[DataSavvy]

Hi @Tonanate It looks like you're on the right track with setting up your AWS Lambda function to call the Campaign Manager 360 API using a service account. The error you're encountering, "Request had insufficient authentication scopes", indicates that your service account lacks the necessary permissions to access the Campaign Manager 360 API.

Here's how you can resolve this issue:

1. Verify Scopes:

   • Ensure that the scopes you’re using (https://www.googleapis.com/auth/dfareporting and https://www.googleapis.com/auth/ddmconversions) are correctly defined and match the permissions needed for your API calls.

2. Service Account Permissions:

   • Confirm that the service account you're using has been granted the appropriate permissions within Google Cloud IAM. It should have access to the Campaign Manager 360 API with roles like "Campaign Manager 360 Read Only."

3. Delegate Account (Optional):

   • If your integration requires acting on behalf of another user, make sure that the delegate email you've specified has the necessary permissions in Campaign Manager 360.

4. Testing:

   • As a first step, test the service account credentials directly from your local environment to ensure that everything works as expected. Once confirmed, migrate the setup to AWS Lambda.

Additional Suggestion:

If you find the integration process challenging or if you'd like a more streamlined way to access and analyze your marketing data, you might want to consider using a tool like Windsor.ai. It simplifies the process of aggregating data from multiple sources, including Campaign Manager 360, and could save you time on complex integrations.

Hope this helps!