Unable to connect to a GCP VM through IAP from a Bitbucket pipeline

[mezhaka]

I need to open a tunnel to our Google virtual private cloud to be able to query a service that is not exposed to the internet. Google allows to create a tunnel with:

gcloud compute start-iap-tunnel vm-name target-port

This usually prints something like:

Picking local unused port [38441].
Testing if tunnel connection works.
Listening on port [38441].

The crucial bit is "Listening on port ...". I am able to run this command on my development machine using the service account credentials that I use in Bitbucket. I have also been able to run it from a Docker container in which I followed the steps from the pipeline: installed gcloud and set it up with a service account.

However, in Bitbucket pipelines the command hangs: the "Listening on port [...]" is never printed. I tried to run this command with some additional verbose output both in bitbucket pipelines and locally. The failed bitbucket version outputs the following:

gcloud compute start-iap-tunnel my-vm-name 8080 --zone=europe-west3-b --log-http --verbosity=debug
DEBUG: Running [gcloud.compute.start-iap-tunnel] with arguments: [--log-http: "true", --verbosity: "debug", --zone: "europe-west3-b", INSTANCE_NAME: "my-vm-name", INSTANCE_PORT: "8080"]
=======================
==== request start ====
uri: https://compute.googleapis.com/compute/v1/projects/<SANITIZED>/zones/europe-west3-b/instances/my-vm-name?alt=json
method: GET
== headers start ==
b'accept': b'application/json'
b'accept-encoding': b'gzip, deflate'
b'authorization': --- Token Redacted ---
b'content-length': b'0'
b'user-agent': b'google-cloud-sdk gcloud/436.0.0 command/gcloud.compute.start-iap-tunnel invocation-id/e86258106fc74f53b220deb1a2567db0 environment/None environment-version/None client-os/LINUX client-os-ver/5.15.0 client-pltf-arch/x86_64 interactive/False from-script/False python/3.9.16 term/ (Linux 5.15.0-1037-aws)'
b'x-goog-api-client': b'cred-type/sa'
== headers end ==
== body start ==

== body end ==
==== request end ====
DEBUG: Starting new HTTPS connection (1): compute.googleapis.com:443
DEBUG: https://compute.googleapis.com:443 "GET /compute/v1/projects/<SANITIZED>/zones/europe-west3-b/instances/my-vm-name?alt=json HTTP/1.1" 200 None
---- response start ----
status: 200
-- headers start --
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Cache-Control: private
Content-Encoding: gzip
Content-Type: application/json; charset=UTF-8
Date: Thu, 22 Jun 2023 16:05:56 GMT
ETag: <SANITIZED>
Server: ESF
Transfer-Encoding: chunked
Vary: Origin, X-Origin, Referer
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 0
-- headers end --
-- body start --
{
"kind": "compute#instance",
"id": "<SANITIZED>",
"creationTimestamp": "2023-02-08T08:44:58.596-08:00",
"name": "my-vm-name",
"description": "",
"tags": {
"items": [
"<SANITIZED>",
"<SANITIZED>"
],
"fingerprint": "<SANITIZED>"
},
"machineType": "<SANITIZED>",
"status": "RUNNING",
"zone": "<SANITIZED>",
"canIpForward": false,
"networkInterfaces": [
{
"kind": "compute#networkInterface",
"network": "https://www.googleapis.com/compute/v1/projects/<SANITIZED>/global/networks/default",
"subnetwork": "https://www.googleapis.com/compute/v1/projects/<SANITIZED>/regions/europe-west3/subnetworks/default",
"networkIP": "10.156.0.27",
"name": "nic0",
"accessConfigs": [
{
"kind": "compute#accessConfig",
"type": "ONE_TO_ONE_NAT",
"name": "External NAT",
"natIP": "34.159.245.120",
"networkTier": "PREMIUM"
}
],
"fingerprint": "<SANITIZED>",
"stackType": "IPV4_ONLY"
}
],
"disks": [
<SANITIZED>
],
"metadata": {
"kind": "compute#metadata",
"fingerprint": "<SANITIZED>",
"items": [
{
"key": "windows-keys",
"value": "{<SANITIZED>}"
}
]
},
"serviceAccounts": [
{
"email": "<SANITIZED>@developer.gserviceaccount.com",
"scopes": [
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring.write",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/service.management.readonly",
"https://www.googleapis.com/auth/trace.append"
]
}
],
"selfLink": "https://www.googleapis.com/compute/v1/projects/<SANITIZED>/zones/europe-west3-b/instances/my-vm-name",
"scheduling": {
"onHostMaintenance": "TERMINATE",
"automaticRestart": true,
"preemptible": false,
"provisioningModel": "STANDARD"
},
"cpuPlatform": "Intel Cascade Lake",
"labelFingerprint": "<SANITIZED>",
"startRestricted": false,
"deletionProtection": false,
"reservationAffinity": {
"consumeReservationType": "ANY_RESERVATION"
},
"displayDevice": {
"enableDisplay": false
},
"shieldedInstanceConfig": {
"enableSecureBoot": false,
"enableVtpm": true,
"enableIntegrityMonitoring": true
},
"shieldedInstanceIntegrityPolicy": {
"updateAutoLearnPolicy": true
},
"confidentialInstanceConfig": {
"enableConfidentialCompute": false
},
"fingerprint": "<SANITIZED>",
"lastStartTimestamp": "2023-06-05T06:32:48.873-07:00",
"lastStopTimestamp": "2023-06-05T06:20:24.134-07:00",
"keyRevocationActionType": "NONE"
}

-- body end --
total round trip time (request+response): 0.211 secs
---- response end ----
----------------------
Picking local unused port [36513].
WARNING:

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

Testing if tunnel connection works.
DEBUG: credentials type for _GetAccessTokenCallback is [<google.oauth2.service_account.Credentials object at 0x7fadb593edf0>].
DEBUG: Using new websocket library
INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=<SANITIZED>&port=8080&newWebsocket=True&zone=europe-west3-b&instance=my-vm-name&interface=nic0']
DEBUG: RECV opcode [2] data_len [348] binary_data[:20] [b'\x00\x01\x00\x00\x01VAbvJZZ7uNrVf1j']
DEBUG: CLOSE
INFO: Received WebSocket Close message [None: 'Connection closed while receiving data.'].

The succesful run of this command locally is pretty much similar, but does print the "Listening on port.". Here are the last lines for the case everything works fine:

Testing if tunnel connection works.
DEBUG: credentials type for _GetAccessTokenCallback is [<google.oauth2.service_account.Credentials object at 0x7f58cb680a30>].
DEBUG: Using new websocket library
INFO: Connecting with URL ['wss://tunnel.cloudproxy.app/v4/connect?project=lofty-seer-161814&port=8080&newWebsocket=True&zone=europe-west3-b&instance=my-vm-name&interface=nic0']
DEBUG: RECV opcode [2] data_len [348] binary_data[:20] [b'\x00\x01\x00\x00\x01VAbvJZZ5Kz16cIn']
DEBUG: CLOSE
Listening on port [37863].
DEBUG: CLOSE
INFO: Received WebSocket Close message [None: 'Connection closed while receiving data.'].

I have exhausted all the debugging options I could think of and would be terribly grateful for any suggestion.