GCP Batch with GCS Signed URL

[andre_john]

Using Google Cloud Batch Instance to generate with GSUTIL a signed URL; The command in BASH is;

  SIGNED=`gsutil signurl -r $REGION $GOOGLE_APPLICATION_CREDENTIALS $url | tail -n 1 | python3 -c "import sys; print(sys.stdin.read().split('\t')[-1])"`
  curl -H "Authorization: Bearer $TB_TOKEN" \
    -X POST "$TB_HOST/v0/datasources?name=$DESTINATION_DATA_SOURCE&mode=append" \
    --data-urlencode "url=$SIGNED"
  echo

This line I always runs into the following error;

2024-11-28 13:10:27.279 EST CommandException: The signurl command 
requires the pyopenssl library (try pip install pyopenssl or easy_install pyopenssl)

I get this despite having done the pip install

Successfully installed cffi-1.17.1 cryptography-44.0.0 
pycparser-2.22 pyopenssl-24.3.0

I understand this occurs because the Python environment for GSUtil is probably different from the one I install... I have tried several different ways to do this without success. My script currently looks like this;

PATH_GSUTIL=mnt/disks/share

sudo apt-get update
sudo apt-get install -y python3-pip
python3 -m pip install pyopenssl
sudo pip install --no-input pyopenssl --target $PATH_GSUTIL
gsutil version -l
# echo $(which pip3)
export PYTHONPATH=$PYTHONPATH:$PATH_GSUTIL
export CLOUDSDK_PYTHON=$(which python3)

What is the correct way to install pyopenssl in a batch instance so that I may use gsutil with signurl?