PubSub Pull in GKE: ACCESS_TOKEN_SCOPE_INSUFFICIENT

[magenti]

I am implementing a PubSub-Pull in GKE but when running, the  pod throws this error:

File "main.py", line 35, in <module> streaming_pull_future.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result raise self._exception google.api_core.exceptions.PermissionDenied: 403 Request had insufficient authentication scopes. [reason: "ACCESS_TOKEN_SCOPE_INSUFFICIENT"

After deploying from yaml, I launch these IAM bindings:

gcloud iam service-accounts add-iam-policy-binding $GSA \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$PROJECT_ID.svc.id.goog[$NAMESPACE/$KSA]"

gcloud projects add-iam-policy-binding $PROJECT_ID \
--member "serviceAccount:$GSA" \
--role roles/pubsub.subscriber

What s the problem? I can't get it.

Thanks for helping

[HectorArturo]

Hello magenti,

Could you please share if you followed any available guides or tutorials?

[magenti]

Hello HectorArturo, 
First of all thank you for your help.
I am trying to implement the Autopilot PubSub Consumer as explained in this tutorial:
https://www.youtube.com/watch?v=DaopYyIhqJA&list=PL4RrzxkjhEKuZJKCjXrtzbECfnZQlp397&index=16&t=27s

Starting from the GitHub code I do the following steps:

gcloud config set compute/region $REGION
gcloud config set project $PROJECT_ID

gcloud pubsub topics create autopilot-test-topic
gcloud pubsub subscriptions create autopilot-test-sub --topic autopilot-test-topic

gcloud container clusters create-auto $CLUSTER_NAME \
--region=$REGION \
--project=$PROJECT_ID

gcloud container clusters get-credentials $CLUSTER_NAME \
--region=$REGION \
--project=$PROJECT_ID

gcloud artifacts repositories create autopilot-pubsub-consumer \
--project=$PROJECT_ID \
--repository-format=docker \
--location=$REGION

gcloud builds submit

gcloud iam service-accounts create $KSA
gcloud iam service-accounts create $cm_KSA

gcloud iam service-accounts add-iam-policy-binding $GSA \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$PROJECT_ID.svc.id.goog[$NAMESPACE/$KSA]"

gcloud projects add-iam-policy-binding $PROJECT_ID \
--member "serviceAccount:$GSA" \
--role roles/pubsub.subscriber

gcloud projects add-iam-policy-binding $PROJECT_ID \
--member "serviceAccount:$cm_GSA" \
--role roles/monitoring.metricDescriptors.get

gcloud iam service-accounts add-iam-policy-binding custom-metrics-sa@$PROJECT_ID.iam.gserviceaccount.com \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:$PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]"

kubectl apply -f deployment/metric-adaptor.yaml
kubectl apply -f deployment/pubsub.yaml
kubectl apply -f deployment/pubsub-hpa.yaml

Observe that, I before running the above, I made a preliminary step on the GitHub code to create templates. In other words, I introduced variables like PROJECT_ID, KSA, cm_GSA etc in order to make the code easier to understand and to reuse. More in details, using the sed command, I made the following replacements:

sed -e "s/europe-west2/[REGION]/" \
-e "s/rocketech-de-pgcp-sandbox/[PROJECT_ID]/" \
-e "s/demo\/autopilot-pubsub-consumer/[APP_NAME]\/[Docker_IMG]/" \
-e "s/\:v1.0.0//" \
originals/cloudbuild.yaml >> \
originals/cloudbuild.yaml.tmp && mv originals/cloudbuild.yaml.tmp templates/cloudbuild.yaml.bak

sed -e "s/pubsub-consumer-demo-sa\@rocketech-de-pgcp-sandbox.iam.gserviceaccount.com/[GSA]/" \
-e "s/pubsub-consumer-demo-sa/[KSA]/" \
-e "s/pubsub-consumer-demo/[NAMESPACE]/" \
-e "s/rocketech-de-pgcp-sandbox/[PROJECT_ID]/" \
-e "s/autopilot-test-sub/[SUBSCRIPTION_ID]/" \
-e "s/demo\/autopilot-pubsub-consumer/[APP_NAME]\/[Docker_IMG]/" \
-e "s/autopilot-pubsub-consumer/[APP_NAME]/" \
-e "s/europe-west2/[REGION]/" \
-e "s/v1.0.0/latest/" \
originals/pubsub.yaml >> \
originals/pubsub.yaml.tmp && mv originals/pubsub.yaml.tmp templates/pubsub.yaml.bak

sed -e "s/autopilot-pubsub-consumer-hpa/[hpa_SA]/" \
-e "s/pubsub-consumer-demo/[NAMESPACE]/" \
-e "s/autopilot-test-sub/[SUBSCRIPTION_ID]/" \
-e "s/autopilot-pubsub-consumer/[APP_NAME]/" \
originals/pubsub-hpa.yaml >> \
originals/pubsub-hpa.yaml.tmp && mv originals/pubsub-hpa.yaml.tmp templates/pubsub-hpa.yaml.bak

sed -e "s/custom-metrics-sa@rocketech-de-pgcp-sandbox.iam.gserviceaccount.com/[cm_GSA]/" \
originals/metric-adaptor.yaml >> \
originals/metric-adaptor.yaml.tmp && mv originals/metric-adaptor.yaml.tmp templates/metric-adaptor.yaml.bak

[comaro]

Hi, magenti,

The nodepool’s node where the pod is running should have GKE_METADATA enabled.

Does the nodepool have GKE_METADATA enabled?

If you want to use workload-identity, then you will have to enable GKE_METADATA on your nodepools.
If you want to use the SA assigned to the nodepool to authenticate, then you can add the scope pubsub.googleapis.com to the nodepool.

Please tell me if this answer resolved your issue.

[magenti]

Hello Comaro, 

I have solved the ACCESS_TOKEN_SCOPE_INSUFFICIENT issue. It was actually due to a bad formatting of my YAML files.

Thank you for helping.