Airflow DAG cannot connect to project with credentials

[cmcrawford2]

I am using Airflow in Docker on my laptop to create an external table in BigQuery using data that I've uploaded to cloud storage. I've done this before succesfully, but now I keep getting the message "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential."
I created a project; successfully uploaded files to cloud storage in the project using Airflow, but for step 2 I want to create an external table in BigQuery, and there's something wrong with authentication.

I had another project set up that I used to carry out the homework assignments, and that had no problem, I could create external table just fine and complete the homework. For the final assignment, I created a new project in GCP. I used Airflow to upload a ton of data into cloud storage, and that worked properly. The trouble started when I got to step 2. Nothing worked. I had not created a service account for the second project, so I did that, generated a key that was saved to a json file that I put in my credentials directory, and I set everything to point to that file. When I examine the Docker file, the right credentials are there. So I think something is wrong with how I set up the service account.

A couple of people have tried to help me but no luck. I followed these steps:

• Now create a service account and generate keys like shown in the videos
• Download the key and put it to some location, e.g. .google/credentials/google_credentials.json
• Set GOOGLE_APPLICATION_CREDENTIALS to point to the file

export GOOGLE_APPLICATION_CREDENTIALS=~/.google/credentials/google_credentials.json

Now authenticate:

gcloud auth activate-service-account --key-file $GOOGLE_APPLICATION_CREDENTIALS

I did all this and got a confirmation from the last step. But the problem persists.

[junlu]

Hi, 

Google Groups are reserved for general product discussion, and you shall consider going to StackOverflow for technical questions. 

To get a better support you should post to the relevant forum, thus please read the Community Support article for better understanding. 

[carl24k]

I found the answer in my case, it was adding location to my operator.  So first, check the dataset information if you are not sure the location. Then add it as a parameter in your operator. For example, my dataset was in us-west1 and I was using an operator that looked like this:

    check1 = BigQueryCheckOperator(task_id='check_my_event_data_exists',
                       sql="""
                        select count(*) > 0
                        from my_project.my_dataset.event
                         """,
                       use_legacy_sql=False,
                       location="us-west1")   # THIS WAS THE FIX IN MY CASE