Re: Anthos config management operator

Andrei_M · 04-12-2023 03:29 AM

I pulled and installed config management operator from

gs://config-management-release/released/latest/config-management-operator.yaml

and this is working ok but according to redirecting image requests from k8s.gcr.io to registry.k8s.io I've made changes in the manifest:

from

    spec:
      containers:
      - command:
        - /manager
        - --private-registry=
        name: manager
        image: gcr.io/config-management-release/config-management-operator:20230307231553-op

to

    spec:
      containers:
      - command:
        - /manager
        - --private-registry=
        name: manager
        image: registry.k8s.io/config-management-release/config-management-operator:20230307231553-op

And the Pod creation is failing with ImagePullBackOff error. What is the possible cause of this error?

Willbin

Hello Andrei_M,

Welcome to GCC!

Can you send me the Events output when you run kubectl describe <pod name>? We have to know what's causing the issue and this will show us the error log of the pod.

Possible causes:

Image doesn't exist
Typo in the image name
Image registry requires authentication
Download limit on the registry exceeded

Thanks!

banksie

Hi

We're also experiencing issues pulling from registry.k8s.io. We are running an Anthos cluster on VMWare and having some issues pulling container images for Secrets Store and KubeStateMetrics. We are seeing error messages for e.g.

Failed to pull image "registry.k8s.io/csi-secrets-store/driver-crds:v1.3.3": rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/csi-secrets-store/driver-crds:v1.3.3": failed to resolve reference "registry.k8s.io/csi-secrets-store/driver-crds:v1.3.3": failed to do request: Head "https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/csi-secrets-store/driver-crds/manifests/v1.3.3": x509: certificate signed by unknown authority
Warning Failed 3s (x4 over 88s) kubelet Error: ErrImagePull

We've checked our firewall rules and there is nothing blocking access so I'm thinking its an issue with the trusted certs in the Anthos node images. We're using the default Ubuntu containerd image, which i believe is based on Ubuntu 18.04. Our Anthos version is 1.14.1-gke.39.

If I try and curl `https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/csi-secrets-store/driver-crds/manif...` from the Anthos Admin workstation (also uses a Google provided OS image) I also get an error: `curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it.`

If I do the same from our jumpbox (in same VLAN), which uses a standard Ubuntu 20.04.5 its all ok. I can also pull the image on that box.

The solution appears to be to install the appropriate certs on the Anthos nodes, but this seems in-practical. Do Google need to look at updating the Anthos node images?

yannnnnn

I think this post on stack overflow might be helpful:

https://stackoverflow.com/questions/76162163/cant-pull-images-from-registry-k8s-io-from-anthos-kuber...