Can not pull from pubub (using either unary or streaming pull) in cloud run (job or service)

[jcrs]

Hi,

I'm trying to pull from a regional pubsub subscription (europe-west9) within a python process that runs in cloud run (job or service - observed behavior is exactly the same). The application layer was tested using a container running on my local machine and everything went fine - the issue really seems to be related to cloud run, and how networking is handled for that resource.

As a reference, the code I use is very close to this tutorial.

On the `pull(...)` (or future_streaming.result(...)),  PubSub Python Client tries fetch information from the internal metadata server, which results in:

```
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/google/api_core/bidi.py", line 644, in _thread_main
self._bidi_rpc.open()
File "/usr/local/lib/python3.10/site-packages/google/api_core/bidi.py", line 282, in open
call = self._start_rpc(iter(request_generator), metadata=self._rpc_metadata)
File "/usr/local/lib/python3.10/site-packages/google/pubsub_v1/services/subscriber/client.py", line 1832, in streaming_pull
self._validate_universe_domain()
File "/usr/local/lib/python3.10/site-packages/google/pubsub_v1/services/subscriber/client.py", line 559, in _validate_universe_domain
or SubscriberClient._compare_universes(
File "/usr/local/lib/python3.10/site-packages/google/pubsub_v1/services/subscriber/client.py", line 536, in _compare_universes
credentials_universe = getattr(credentials, "universe_domain", default_universe)
File "/usr/local/lib/python3.10/site-packages/google/auth/compute_engine/credentials.py", line 154, in universe_domain
self._universe_domain = _metadata.get_universe_domain(
File "/usr/local/lib/python3.10/site-packages/google/auth/compute_engine/_metadata.py", line 296, in get_universe_domain
universe_domain = get(
File "/usr/local/lib/python3.10/site-packages/google/auth/compute_engine/_metadata.py", line 229, in get
raise exceptions.TransportError(
google.auth.exceptions.TransportError: Failed to retrieve http://metadata.google.internal/computeMetadata/v1/universe/universe-domain from the Google Compute Engine metadata service. Compute Engine Metadata server unavailable
```

Also - when I try do directly hit `europe-west9-pubusb.googlepais.com` (or `pubusb.googlepais.com` - still from within cloud run service/job), I get a "DNS can not resolve" error.  

I did a lot of research to try and debug the issue, and didn't manage to find any tangible clue. From what I understood, I'm facing a networking issue - but no effort whatsoever I did to make it work change anything (I am quite the beginner in network).

The issue is quite critical for many reasons - so any help would be warmly welcomed.
 
Thanks in advance,
Best,
JC

[-Rhett]

Hi @jcrs,

We’re currently aware of a known issue with Compute Engine that occurs when running jobs that involve retrieving metadata, resulting in the error:

google.auth.exceptions.TransportError: Failed to retrieve
http://metadata.google.internal/computeMetadata/v1/universe/universe_domain
from the Google Compute Engine metadata service. Compute Engine Metadata server unavailable

We’re working to address this. In the meantime, you can try the following:

• Perform troubleshooting for Cloud Run deployment issues
• Confirm that your application follows Cloud Run networking best practices
• Run a Connectivity Test to further diagnose the issue

As a workaround, since the error happens due to a library running on Compute Engine, consider migrating your application to run on Google Kubernetes Engine instead of Compute Engine.

If none of these resolve the issue, you can file a detailed bug report through the Issue Tracker. Include steps to replicate, configuration settings, screenshots, and other details of the issue to help us investigate. While there isn’t a specific timeframe for resolution, once we've fixed an issue in production we'll indicate this and then update and close the bug. You may also contact Google Cloud support for more assistance.

I hope this helps!