Updating secrets in GKE Cluster setup in Autopilot...

ChandraMohit · ‎07-13-2023

Google offers GKE cluster setup in two modes — Standard and Autopilot mode. Updating secrets for any workload running may be a tiresome job and sometimes, it may result in access errors if workload pulls images from a secured repository.

One of such scenario happened where one of the user deleted the key generated for a service account and deployment of the workload within GKE started failing. This key was initially used for deploying the workload. On analysing the root-cause, it was found that key need to be regenerated for the service account and also the secret need to be updated within Google Kubernetes workload too. I have listed the steps taken below to resolve the issue.

When this can be used ?

This approach is only applicable for a Kubernetes cluster which uses the secret of kubernetes.io/dockerconfigjson type to authenticate with a container registry to pull a private image. Storing or downloading service account key is not an ideal approach or advised from security standpoint.

Steps to be followed:

Step 1: Check the existing secret present within the secrets and configmaps. You can use either Google Cloud UI by going to option Kubectl → Get YAML present within a secret or you can run command directly within cloudshell.

gcloud config set project <project-name>

gcloud container clusters get-credentials <cluster-name> --region <region-name> --project <project-name> && kubectl get secret <secret-name> --namespace <namespace-name> -o yaml

Step 2: After executing step1, refer to data section within the secret listed against .dockerconfigjson tag. The value present will be present in base64. Copy the contents and run command in cloudshell—

echo <copied-content>| base64 --decode|jq

Step 3: Above command will result in structure listed below. All the values will remain same expect the password tag present. Steps 4,5 and 6 will be used to create the JSON key in desired format.

{
  "auths": {
    "<RegistryName>": {
      "email": "<Service Account Name>",
      "password": "<JSON Key>",
      "username": "<UserName>"
    }
  }
}

Step 4: Create a new JSON key for the service account from IAM&Admin section present on Google cloud UI. Use three dots present against service account to create or manage the keys. Do check if you have access to create the serviceaccountkey and have download permissions.

Step 5: Once the key is downloaded, copy the contents of the JSON key file and minify the JSON. For this you can use some online tools like JSON Minifier.

Step 6: The minified JSON can be further used to convert into JSON with escape characters. You can use any of the online tools if required.

Step 7: Run command in the cloudshell by passing the values —

kubectl create secret docker-registry $secret_name \
  --docker-server=<DOCKER_REGISTRY_SERVER> \
  --docker-username=<DOCKER_USER> \
  --docker-password=<DOCKER_PASSWORD> \
  --docker-email=<DOCKER_EMAIL> -o yaml > docker-secret.yaml

Replace the values -

$secret_name - Name of the secret which you want for deployment
DOCKER_REGISTRY_SERVER - Use RegistryName which we got from Step3. 
DOCKER_USER - UserName from Step3.
DOCKER_PASSWORD - JSON with escape characters from Step6. 
DOCKER_EMAIL - Service Account Name from Step3.

Step 8: After Step7, a docker-secret.yaml file is created within cloudshell session for the user and below command can be used to upload secret. This will upload the secret in base64 encoded format only.

kubectl apply -f docker-secret.yaml -n <namespace-name>

The secret is now uploaded and you can redeploy the workload after updating the values.yaml file with new secret name.

These steps can be useful even for GKE cluster setup in standard mode.

Updating secrets in GKE Cluster setup in Autopilot mode