ssh keys injection issue

[vadim-vi]

I am trying to create a service that would run in a container, connect to a remote host by ssh and perform several commands remotely. This service will be built and run in cloud build. Right now I am stuck with trying to inject SSH keys into a container - build fails with the error:

Step #2 - "run replication command": Load key "/root/.ssh/id_rsa": invalid format

My keys were created with `ssh-keygen` so they are probably correct.
There seems to be some sort of issue with SSH keys, but I have no idea what is wrong.
What I have already tried:

1.  As this is the follow-up to this issue, I have already tried
   injecting the keys through the secret manager at build stage. This
   failed because builder refuses to perform substitutions properly and
   passes the variable name instead of the value (that is, I have a
   secret env variable called PRIVATE, and when I reference it as
   $$PRIVATE, builder sets value to $PRIVATE which is not what I need);
   
   
2.  I have tried creating the secrets in different ways, none of them worked:
   * echo $(cat container_private_key.pem) | gcloud secrets create private-key --data-file=-  
   * echo -n <secret value here> | gcloud secrets create private-key --data-file=-  
   * gcloud secrets create private-key --data-file=container_private_key.pem
3. I also tried inserting keys both with and without headers, that had no positive effect.

Right now the keys are injected as follows:

mkdir -p /root/.ssh && chmod 0700 /root/.ssh && echo $CONTAINER_PRIVATE_KEY > /root/.ssh/id_rsa && \
echo $CONTAINER_PUBLIC_KEY > /root/.ssh/id_rsa.pub && chmod 400 /root/.ssh/id_rsa && \
chmod 600 /root/.ssh/id_rsa.pub

As you can see, the values are taken from the environment variables, and these variable actually do contain the keys, I tried echoing them.
I assume that the issue is somehow related to the keys being mangled at some stage between my computer, secret manager and echoing the values from environment variables, but I cannot understand where exactly it might happen.

[vadim-vi · Accepted solution]

I did some digging and found out that my hypothesis about SSH keys getting mangled at some point was correct. The problem illustrated:

1. I submit a secret to Secret Manager in a file (no problem here);
2. My secret gets referenced and exported as an environment variable to Cloud Build and that is where things get bad. Somewhere around this place the key gets stripped of all the special unprintable symbols and SSH will consider it a malformed key in the next step;
3. The value enters the file successfully, but since it is malformed, SSH does not care and will not use the key.

So, to work around this issue I did the following:
I encoded my keys into base64 on my host machine and decoded them at the very last moment. Note the calls to the base64 utility

mkdir -p /root/.ssh && chmod 0700 /root/.ssh && echo "$CONTAINER_PRIVATE_KEY" | base64 --decode > /root/.ssh/id_rsa && \
echo "$CONTAINER_PUBLIC_KEY" | base64 --decode > /root/.ssh/id_rsa.pub && chmod 400 /root/.ssh/id_rsa && \
chmod 600 /root/.ssh/id_rsa.pub