GCP logging and Cloud tracing API returning 503 U...

Siva_Tharun · 03-09-2023 06:14 AM

Hi,

i am new to GCP cloud service, i am trying to setup the cloud logging/tracing for one the spring boot repository, so i am using the spring boot cloud gcp libraries, to perform the same.

i was trying to run the service on my local dev environment, i have already set the GOOGLE_APPLICATION_CREDENTIALS file location as environment variable to the application. i have made sure, that the service account has the correct set of roles for writing the logs and traces in to the GCP operation suite (stackdriver).

However i have noticed that i am receiving 503 UNAVAILABLE response status for both the cloudtrace.googleapis.com and logging.googleapis.com REST/grpc calls.

i have made sure, that both the above mentioned API's are enabled for my project, by using the below gcloud CLI command.

gcloud services enable cloudtrace.googleapis.com

gcloud services enable logging.googleapis.com

When i tried to insert the log via the gcloud cli, it was working as expected.

And these are the credential scopes, that are being set to as part of the DefaultCredentialsProvider configuration -

[https://www.googleapis.com/auth/bigquery, https://www.googleapis.com/auth/trace.append, https://www.googleapis.com/auth/logging.admin, https://www.googleapis.com/auth/sqlservice.admin, https://www.googleapis.com/auth/devstorage.read_only, https://www.googleapis.com/auth/monitoring.write, https://www.googleapis.com/auth/pubsub, https://www.googleapis.com/auth/logging.write, https://www.googleapis.com/auth/spanner.admin, https://www.googleapis.com/auth/cloudruntimeconfig, https://www.googleapis.com/auth/devstorage.read_write, https://www.googleapis.com/auth/cloud-platform, https://www.googleapis.com/auth/spanner.data, https://www.googleapis.com/auth/datastore, https://www.googleapis.com/auth/cloud-vision]

Please find below the sample java configuration used to setup the application.

application.properties

spring.cloud.gcp.logging.enabled=true
spring.cloud.gcp.trace.enabled=true
stackdriver.log.name=service.log
spring.cloud.gcp.credentials.scopes=DEFAULT_SCOPES,https://www.googleapis.com/auth/logging.write,https://www.googleapis.com/auth/logging.admin,https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/trace.append

management.tracing.sampling.probability=1.0

Application logs

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at io.grpc.Status.asRuntimeException(Status.java:535)
	at zipkin2.reporter.stackdriver.internal.AwaitableUnaryClientCallListener.onClose(AwaitableUnaryClientCallListener.java:100)
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463)

Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: cloudtrace.googleapis.com/2404:6800:4007:81c:0:0:0:200a:443
Caused by: java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.Net.connect0(Native Method)
	at sun.nio.ch.Net.connect(Net.java:482)
	at sun.nio.ch.Net.connect(Net.java:474)

java.lang.RuntimeException: com.google.cloud.logging.LoggingException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
	at com.google.cloud.logging.LoggingImpl$9.onFailure(LoggingImpl.java:943)
	at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:68)
	at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)

Caused by: io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: logging.googleapis.com/2001:4860:4802:32:0:0:0:174:443
Caused by: java.net.NoRouteToHostException: No route to host
	at sun.nio.ch.Net.connect0(Native Method)

logback-spring.xml

<configuration>
    <include resource="com/google/cloud/spring/logging/logback-appender.xml" />
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml" />
    <include resource="com/google/cloud/spring/logging/logback-json-appender.xml" />
    <root level="INFO">
        <appender-ref ref="CONSOLE"/>
        <appender-ref ref="STACKDRIVER" />
    </root>
</configuration>

pom.xml

<parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.6.8</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>


<properties>
        <java.version>1.8</java.version>
        <spring-cloud-gcp.version>3.2.0</spring-cloud-gcp.version>
        <spring-cloud.version>2021.0.1</spring-cloud.version>
    </properties>

<dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>spring-cloud-gcp-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>spring-cloud-gcp-starter-logging</artifactId>
        </dependency>
        <dependency>
            <groupId>com.google.cloud</groupId>
            <artifactId>spring-cloud-gcp-starter-trace</artifactId>
        </dependency>

<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>com.google.cloud</groupId>
                <artifactId>spring-cloud-gcp-dependencies</artifactId>
                <version>${spring-cloud-gcp.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

dionv

Hello Siva_Tharun,

Here is a documentation about the 503 UNAVAILABLE ERROR with what may cause the error and how to fix it.

GCP logging and Cloud tracing API returning 503 UNAVAILABLE Error