Getting large amounts of "Unauthenticated request." errors when using GCP Batch.

[vedantroy-genmo]

Task Name: rw-id3779-task4000-par4000-stage0-download.

Task UID:  rw-id3779-task4000-637b650e-07c4-429d0.

1942 succeeded, 2056 tasks failed, 2 running. (Total 4K tasks).

I'm getting a lot of errors of the form:
```

docker: Error response from daemon: Head "https://us-central1-docker.pkg.dev/v2/${my_project}/gcpbatch-us-central1/${my_repo}/manifests/latest": denied: Unauthenticated request. Unauthenticated requests do not have permission "artifactregistry.repositories.downloadArtifacts" on resource "projects/${my_project}/locations/us-central1/repositories/gcpbatch-us-central1" (or it may not exist).

```

[vedantroy-genmo]

It certainly looks like many (if not most?) of the task failures are due to this error:

 

[vedantroy-genmo]

Update: Reducing docker image size cut down on failure rate dramatically, so that was likely the issue.

[ket0825]

I guess you are using COS for run the container. And the issue occurs when you try to pull image from artifact registry.

I think your docker config does not have any permission to pull image from artifact registry.

You should modify /root/.docker/config.json to get permissions.

You can use docker-credential-gcr.

Simply, just enter like this.

/usr/bin/docker-credential-gcr configure-docker --registries [REGION]-docker.pkg.dev

But In COS, you cannot modify the /root/.docker/config.json due to the security. You can Read-only at root directory.

So,  I used the cloud-init.yaml file to bypass the root config and make new user.

And give access to myuser about docker.

you can reference my cloud-init.yaml file.

------------------------------------

#cloud-config

users:

- name: myuser

  uid: 2000

write_files:  

- path: /etc/systemd/system/docker-credential-gcr.service

  permissions: 0644

  owner: root

  content: |

    [Unit]

    Description=Configure Docker to use GCR credentials

    Wants=docker.service

    After=docker.service

    [Service]

    User=myuser

    Type=oneshot

    ExecStart=/usr/bin/docker-credential-gcr configure-docker --registries [REGION]-docker.pkg.dev

    StandardOutput=journal+console

    StandardError=journal+console

    Restart=on-failure

    RestartSec=10s

    [Install]

    WantedBy=multi-user.target

- path: /etc/systemd/system/group-myuser-docker.service

  permissions: 0644

  owner: root

  content: |

    [Unit]

    Description=Group myuser with docker

    Wants=docker.service docker-credential-gcr.service

    After=docker.service docker-credential-gcr.service

    [Service]

    User=root

    Type=oneshot

    ExecStart=usermod -aG docker myuser

    [Install]

    WantedBy=multi-user.target

- path: /etc/systemd/system/my-app.service

  permissions: 0644

  owner: root

  content: |

    [Unit]

    Description=Run a application container

    Requires=docker-credential-gcr.service group-myuser-docker.service

    After=docker-credential-gcr.service group-myuser-docker.service

    [Service]

    User=myuser

    Type=simple

    ExecStart=/bin/bash -c 'docker run --rm \

      -u 2000 \

      --name=my-app \

      [IMAGE_URL]'

    ExecStop=/usr/bin/docker stop my-app

    Restart=always

    RestartSec=10

    [Install]

    WantedBy=multi-user.target

runcmd:

  - systemctl daemon-reload

  - systemctl enable docker-credential-gcr.service

  - systemctl start docker-credential-gcr.service

  - systemctl enable group-myuser-docker.service

  - systemctl start group-myuser-docker.service

  - systemctl enable my-app.service

  - systemctl start my-app.service

you can get some more details below links.

ref: https://cloud.google.com/container-optimized-os/docs/concepts/security

https://cloud.google.com/container-optimized-os/docs/how-to/run-container-instance

https://cloudinit.readthedocs.io/en/latest/

https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6