Using GCP secrets as part of cloud build

[Lizzardd]

I'm trying to pass in confidential info into a bash script that is part of our cloud build process I've followed the CloudBuild docs to try set it all up.
After many many different tires copied the docker example as a test even that is not working.
Here is what I have at the moment

$ gcloud secrets list
NAME     CREATED              REPLICATION_POLICY  LOCATIONS
SECRET1  2021-08-18T04:37:47  automatic           -
SECRET2  2021-08-18T04:38:11  automatic           -

$ gcloud secrets versions access latest --secret="SECRET1"
Secret1Value
$ gcloud secrets versions access latest --secret="SECRET2"
Secret2Value

$ cat cloudbuild.yaml
steps:
- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args: ['-c', 'docker login --username=$$USERNAME --password=$$PASSWORD']
  secretEnv: ['USERNAME', 'PASSWORD']
availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/SECRET1/versions/1
    env: 'PASSWORD'
  - versionName: projects/$PROJECT_ID/secrets/SECRET2/versions/latest
    env: 'USERNAME'

My understanding was that it would substitute the value of SECRET1 and SECRET2 into the USERNAME and PASSWORD envs but all I'm getting is $USERNAME and $PASSWORD

Arguments
 bash -c docker login --username=$USERNAME --password=$PASSWORD

Feels like I've missed something simple yet fundamental

 == UPDATE 1 ==
Here is my new yaml file

steps:
- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args: ['-c', 'echo Username=$$USERNAME && echo Password=$$PASSWORD']
  secretEnv: ['USERNAME', 'PASSWORD']
- name: 'gcr.io/cloud-builders/docker'
  entrypoint: 'bash'
  args: ['cloudbuilds/script.sh', '$$USERNAME', '$$PASSWORD']
  secretEnv: ['USERNAME', 'PASSWORD']
availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/SECRET1/versions/1
    env: 'PASSWORD'
  - versionName: projects/$PROJECT_ID/secrets/SECRET2/versions/latest
    env: 'USERNAME'

This is the very simple bash script

#/bin/bash
secretVar1="$1"
secretVar2="$2"
printf "\n\nVARIABLES\nSecret1: $secretVar1\nSecret2: $secretVar2\n\n"

And this is the build log

BUILD
Starting Step #0
Step #0: Already have image (with digest): gcr.io/cloud-builders/docker
Step #0: Username=Secret2Value
Step #0: Password=Secret1Value
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: 
Step #1: 
Step #1: VARIABLES
Step #1: Secret1: $USERNAME
Step #1: Secret2: $PASSWORD
Step #1: 
Finished Step #1
PUSH
DONE

So the first command echos the values correctly, yet the bash script is not passed in the secret values.

[alexmoore]

Are you able to share the build log? Also just to double check you granted the cloud build service account the roles/secretmanager.secretAccessor role?

[Lizzardd]

Here is the build log, the docker command will fail even if the process actually pulled the secret values.
As you can see, what is being passed to the scripts is $USERNAME and $PASSWORD

FETCHSOURCE

hint: Using 'master' as the name for the initial branch. This default branch name

hint: is subject to change. To configure the initial branch name to use in all

hint: of your new repositories, which will suppress this warning, call:

hint: 

hint:   git config --global init.defaultBranch <name>

hint: 

hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and

hint: 'development'. The just-created branch can be renamed via this command:

hint: 

hint:   git branch -m <name>

Initialized empty Git repository in /workspace/.git/

From https://source.developers.google.com/p/amiti-plex-can-dev/r/bitbucket_noldortech_amiti-payments

 * branch            a72363459d9ff5bed31411e960cc3e021febc322 -> FETCH_HEAD

HEAD is now at a723634 Secrets test 1

BUILD

Already have image (with digest): gcr.io/cloud-builders/docker

WARNING! Using --password via the CLI is insecure. Use --password-stdin.

Error response from daemon: Get "https://registry-1.docker.io/v2/": unauthorized: incorrect username or password

ERROR

ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

[Lizzardd]

I also did double check that the cloud build service account that is running the build has the "Secret Manager Secret Accessor"  role

[Lizzardd]

So using

args: ['-c', 'echo Username=$$USERNAME && echo Password=$$PASSWORD'], worked as expected

But,   args: ["cloudbuilds/script.sh", "$$USERNAME", "$$PASSWORD"] does not

cloudbuilds/script.sh
#/bin/bash

secretVar1="$1"
secretVar2="$2"
printf "\n\nVARIABLES\nSecret1: $secretVar1\nSecret2: $secretVar2\n\n"

[Lizzardd · Accepted solution]

Finally got the yaml syntax correct to execute the script as I need it

steps:

- name: 'gcr.io/cloud-builders/docker'

  entrypoint: 'bash'

  args: ['-c', ". ./cloudbuilds/script.sh $$SOMEVAR1 $$SOMEVAR2"]

  secretEnv: ['SOMEVAR1', 'SOMEVAR2']

availableSecrets:

  secretManager:

  - versionName: projects/$PROJECT_ID/secrets/SECRET1/versions/1

    env: 'SOMEVAR2'

  - versionName: projects/$PROJECT_ID/secrets/SECRET2/versions/latest

    env: 'SOMEVAR1'

BUILD

Already have image (with digest): gcr.io/cloud-builders/docker

VARIABLES

Secret1: Secret2Value

Secret2: Secret1Value

PUSH

DONE

[JaimeFabian]

Hi @Lizzardd ,

Thanks for this post. I've been looking for this, since I needed to set up a CI/CD pipeline but I'm not getting access to the secrets.

I've added the script to see what values I'm retrieving from the secrets and it shows this:

I've also checked the execution details in cloud build and under secret env. variables are both names of the secrets listed but at the moment of using them as for example $$REACT_APP_AUTH0_DOMAIN it only shows $RACT_APP_AUTH0_DOMAIN.

would you have any idea why? maybe some permissions for the service account being used?

[JaimeFabian]

I've tried with ['-c', 'echo $$REACT_APP_AUTH0_DOMAIN && echo $$REACT_APP_AUTH0_CLIENT_ID] and the secrets are printed as expected but then when I try to use as build arguments for docker, docker receives only $REACT_APP_AUTH0_DOMAIN instead of the actual value.

[JaimeFabian]

So, sorry for the spam but right after replying to this, came the inspiration (as always).

So, I replace the build step's args with this:

running it as bash including the docker command works like a charm.

[sfalk]

Posting a solution is never spam 😄 

I got the same issue but I still got no luck with this here:

The GITHUB_TOKEN is set in my secret manager (no doubt about that). Yet I still get a 401 error when `npm i` gets called from within the Dockerfile.

I did even what I was not supposed to do in order to verify that the token is actually set - and it is:

Something is fishy here 🤔

[sfalk]

Oooff.. it was me. I dynamically update the GITHUB_TOKEN in my CI/CD and made the mistake of echo-ing the secret. The problem: without -n there will be a \n added to the token which is invalid.

The error presented to me was not helpful though. 

What I did learn, however, is, that providing a misformed bearer token to npm it will print the entire token to standard out:

2024-10-05 20:36:17.508 CEST
Step #0: �[0m�[91mnpm error Bearer ghs_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2024-10-05 20:36:17.508 CEST
Step #0: npm error is not a legal HTTP header value

Not sure if this is a good idea for security reasons..

[datascientist11]

What did you do differently? I'm noticing the double quotations. Facing the same issue - thanks