Error predicting from a vertex ai endpoint in python (error 503 DNS resolution failed)

[ducc]

Traceback (most recent call last):
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\api_core\grpc_helpers.py", line 76, in error_remapped_callable
    return callable_(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\grpc\_interceptor.py", line 277, in __call__
    response, ignored_call = self._with_call(
                             ^^^^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\grpc\_interceptor.py", line 332, in _with_call
    return call.result(), call
           ^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\grpc\_channel.py", line 440, in result
    raise self
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\grpc\_interceptor.py", line 315, in continuation
    response, call = self._thunk(new_method).with_call(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\grpc\_channel.py", line 1198, in with_call
    return _end_unary_response_blocking(state, call, True, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\grpc\_channel.py", line 1006, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "DNS resolution failed for ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog:443: UNAVAILABLE: getaddrinfo: WSA Error (No such host is known.
 -- 11001)"
        debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-03-31T12:27:51.0519886+00:00", grpc_status:14, grpc_message:"DNS resolution failed for ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog:443: UNAVAILABLE: getaddrinfo: WSA Error (No such host is known.\r\n -- 11001)"}"
>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\api_core\retry\retry_unary.py", line 144, in retry_target
    result = target()
             ^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\api_core\grpc_helpers.py", line 78, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.ServiceUnavailable: 503 DNS resolution failed for ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog:443: UNAVAILABLE: getaddrinfo: WSA Error (No such host is known. -- 11001)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "c:\Users\zacis\Desktop\New folder\FootySphere (1)\Code\site\d.py", line 17, in <module>
    response = endpoint.predict(instances=[{"content": "Hello world"}], use_dedicated_endpoint=True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\cloud\aiplatform\models.py", line 2384, in predict
    self._sync_gca_resource_if_skipped()
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\cloud\aiplatform\models.py", line 742, in _sync_gca_resource_if_skipped
    self._gca_resource = self._get_gca_resource(
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\cloud\aiplatform\base.py", line 692, in _get_gca_resource
    return getattr(self.api_client, self._getter_method)(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\cloud\aiplatform_v1\services\endpoint_service\client.py", line 1133, in get_endpoint
    response = rpc(
               ^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\api_core\gapic_v1\method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\api_core\retry\retry_unary.py", line 293, in retry_wrapped_func
    return retry_target(
           ^^^^^^^^^^^^^
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\api_core\retry\retry_unary.py", line 153, in retry_target
    _retry_error_helper(
  File "c:\Users\zacis\AppData\Local\Programs\Python\Python311\Lib\site-packages\google\api_core\retry\retry_base.py", line 221, in _retry_error_helper
    raise final_exc from source_exc
google.api_core.exceptions.RetryError: Timeout of 120.0s exceeded, last exception: 503 DNS resolution failed for ENDPOINT_ID.us-central1-PROJECT_NUMBER.prediction.vertexai.goog:443: UNAVAILABLE: getaddrinfo: WSA Error (No such host is known.
 -- 11001)

Currently receiving an error when running the code below:

from google.cloud import aiplatform

PROJECT_NUMBER = "..."
REGION = "us-central1"
ENDPOINT_ID = "..."  

# Initialize Vertex AI with the dedicated domain name
aiplatform.init(
    project=PROJECT_NUMBER,
    location=REGION,
    api_endpoint=f"{ENDPOINT_ID}.us-central1-{PROJECT_NUMBER}.prediction.vertexai.goog"
)

endpoint_name = f"projects/{PROJECT_NUMBER}/locations/{REGION}/endpoints/{ENDPOINT_ID}"
endpoint = aiplatform.Endpoint(endpoint_name=endpoint_name)

response = endpoint.predict(instances=[{"content": "Hello world"}], use_dedicated_endpoint=True)
print(response)

I am not sure to as whats causing this error; below are solutions I have tried/things I doubled check.

• Double checking endpoint id and project number
• All python packages up to date
• No errors in google cloud logging logs
• the vertex ai api is ENABLED
• the vertex ai api service account has the correct permissions
• the vertex ai api is NOT over quota or any limits
• the endpoint was created with the dns button being checked.
• Tried with a VPN
• Used two different operating systems and three different internet connections to see if the error lay there

Full error is above but substituted placeholders for my real project number and endpoint ids.

Below are links to webpages I have tried solutions from, yet nothing seems to work:

• https://www.googlecloudcommunity.com/gc/AI-ML/Scheduling-Vertex-AI-Pipeline-Error-503/m-p/553167
• https://www.googlecloudcommunity.com/gc/AI-ML/Error-from-Vertex-AI-Getting-predictions-from-custom-t...

Any help is greatly appreciated 🙂

[nikacalupas]

Hi ducc,

Welcome to the Google Cloud Community!

The error message you encountered indicates a DNS resolution issue with the Vertex AI endpoint. This is usually caused by network or configuration problems. Here are some things you may consider to resolve the issue:

• Check Endpoint Status in Google Cloud Console: Double-check in the Vertex AI console that your endpoint is indeed deployed and healthy. Look for any error messages related to the endpoint itself.
• Your Local Firewall: Ensure your local firewall isn't blocking outbound connections on port 443 (HTTPS) to the Vertex AI prediction API domain. Temporarily disable your firewall to test if this is the culprit.
• Corporate Firewall/Network Restrictions: If you're on a corporate network, it might have stricter firewall rules or proxy settings that are interfering. Contact your network administrator to check for blocks on the specific Vertex AI domain.
• Flush DNS Cache: Clear your local DNS cache. The commands vary depending on your operating system.
• Temporarily Disable Firewall/Antivirus: As a test, temporarily disable your firewall and any antivirus/security software to rule them out. If this resolves the issue, you'll need to configure your firewall/security software to allow connections.

According to this documentation in Vertex AI includes URLs that end in .googleapis.com, while your URL ends in prediction.vertexai.goog. You can also find additional information in this Stack Overflow discussion.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.