Try to pull image from artifact registry in GKE but getting 401 Unauthorized

[namanjain_sf]

I hosted my micro-services-based application in the GKE cluster and tried to pull the images from artifact registry for my pods. We build and push the images into artifact registry repositories, but when GKE Cluster tries to pull the images from artifact registry repositories, it gets the below 401 Unauthorized error. I provide all the necessary permissions to the service account that are needed to pull the image from artifact registry and this service account is added with my gke cluster node-group.

Here is the error which i get from logs:
Failed to pull image "us-west2-docker.pkg.dev/projx/projx-facade/projx-facade:dev-proj-dvdfsf": failed to pull and unpack image "us-west2-docker.pkg.dev/projx/projx-facade/projx-facade:dev-proj-dvdfsf": failed to resolve reference "us-west2-docker.pkg.dev/projx/projx-facade/projx-facade:dev-proj-dvdfsf": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://us-west2-docker.pkg.dev/v2/token?scope=repository%3Aprojx%2Fprojx-facade%2Fprojx-facade%3Apu...: 401 Unauthorized

[Mikfec09]

I have the same error. Have you found solution to that?

[mokit]

Hi, @namanjain_sf.

The 401 Unauthorized error you're encountering while trying to pull images from Google Artifact Registry typically indicates a problem with the authentication mechanism between your Google Kubernetes Engine (GKE) cluster and Artifact Registry.

Could you please double-check that the service account used by the GKE cluster nodes has the necessary roles to access Artifact Registry? Specifically, the service account should have the roles/artifactregistry.reader role on the Artifact Registry repository.

The GKE nodes might not have the correct permissions to pull from Artifact Registry due to missing IAM bindings. To grant the nodes access to Artifact Registry, ensure that the GKE node pool’s service account has the roles/artifactregistry.reader permission on the repository.

Ensure that the repository location specified in the image tag matches the actual location of the Artifact Registry repository. For instance, in your error message, the repository is located at us-west2-docker.pkg.dev. Ensure that the us-west2 region matches your Artifact Registry repository's location.

Regards,
Mokit

[natekarasch]

Hi,

I'm having the same issue. I've confirmed the following:

1. I can run `docker pull <IMAGE>:<TAG>` from the cloud shell successfully (so it's not a typo in the image:tag string)
2. I have confirmed that the Compute Engine default service account has the Artifact Registry Reader role, both from a project-level and from a repository-specific access level (i.e. I ran the command `gcloud artifacts repositories add-iam-policy-binding gcr.io --location=us --member=serviceAccount:<SERVICE_ACCOUNT_NAME> --role=roles/artifactregistry.reader` for each repo, like gcr.io, us.gcr.io, etc.)
3. I have confirmed that the node pools in the cluster are using that service account (using the steps in this SO answer: https://stackoverflow.com/questions/73269263/how-to-find-the-service-account-used-for-creating-a-res...)

Despite all that, I'm still seeing this message:

Failed to pull image "gcr.io/<MY-PATH>/app:release": failed to pull and unpack image "gcr.io/<MY-PATH>/app:release": failed to resolve reference "gcr.io/<MY-PATH>/app:release": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://gcr.io/v2/token?scope=repository%3A<MY-PATH>%2Fapp%3Apull&scope=repository%3A<MY-PATH>%2Fapp%3Apull&service=gcr.io: 401 Unauthorized

Any help is greatly appreciated!