TimeOutException when publish message protobuf

[hankphan]

Configuration: projectA, topicA, subscriptionA

At server application. when publish message received TimeOutException with error message is
Waited 15000 milliseconds (plus 162590 nanoseconds delay) for com.google.api.core.AbstractApiFuture$InternalSettableFuture@79a07647[status=PENDING]
Stack Trace:

io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182), com.linecorp.armeria.internal.server.grpc.AbstractServerCall.invokeOnMessage(AbstractServerCall.java:387), com.linecorp.armeria.internal.server.grpc.AbstractServerCall.onRequestMessage(AbstractServerCall.java:353), com.linecorp.armeria.server.grpc.UnaryServerCall.lambda$startDeframing$0(UnaryServerCall.java:107), java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934), java.base/java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:950), java.base/java.util.concurrent.CompletableFuture.handle(CompletableFuture.java:2372), com.linecorp.armeria.server.grpc.UnaryServerCall.startDeframing(UnaryServerCall.java:100), com.linecorp.armeria.server.grpc.FramedGrpcService.startCall(FramedGrpcService.java:320), com.linecorp.armeria.server.grpc.FramedGrpcService.startCall(FramedGrpcService.java:295), com.linecorp.armeria.server.grpc.FramedGrpcService.doPost(FramedGrpcService.java:263), com.linecorp.armeria.server.AbstractHttpService.serve(AbstractHttpService.java:57), com.linecorp.armeria.server.AbstractHttpService.serve(AbstractHttpService.java:42), com.linecorp.armeria.server.grpc.AbstractUnframedGrpcService.frameAndServe(AbstractUnframedGrpcService.java:166), com.linecorp.armeria.server.grpc.UnframedGrpcService.lambda$serve$0(UnframedGrpcService.java:156), java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934), java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911), java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510), java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179), com.linecorp.armeria.common.stream.StreamMessageCollector.onComplete(StreamMessageCollector.java:67), com.linecorp.armeria.common.stream.CancellableStreamMessage$CloseEvent.notifySubscriber(CancellableStreamMessage.java:258), com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:303), com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:295), com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:432), com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:375), com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:331), com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:317), com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:439), com.linecorp.armeria.server.StreamingDecodedHttpRequest.tryWrite(StreamingDecodedHttpRequest.java:158), com.linecorp.armeria.server.StreamingDecodedHttpRequest.tryWrite(StreamingDecodedHttpRequest.java:33), com.linecorp.armeria.common.stream.StreamWriter.write(StreamWriter.java:70), com.linecorp.armeria.server.Http2RequestDecoder.onDataRead(Http2RequestDecoder.java:354), io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:36), io.netty.handler.codec.http2.Http2FrameListenerDecorator.onDataRead(Http2FrameListenerDecorator.java:36), io.netty.handler.codec.http2.Http2EmptyDataFrameListener.onDataRead(Http2EmptyDataFrameListener.java:49), io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:307), io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48), io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415), io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250), io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159), io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41), io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173), io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63), io.netty.handler.codec.http2.DecoratingHttp2ConnectionDecoder.decodeFrame(DecoratingHttp2ConnectionDecoder.java:63), io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393), io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453), io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529), io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468), io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290), io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444), io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420), io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412), io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280), io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442), io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420), io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412), io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152), io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442), io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420), io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412), io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410), io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440), io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420), io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919), io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800), io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509), io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407), io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997), io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74), io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30), java.base/java.lang.Thread.run(Thread.java:1583)]

At subscriber, I already config correct projectA and subscriptionA.

I test in local with emulator then publish and subscription success.

But when I check on server. Look like subscriber Application cannot receive message so after 15s TimeOutException occur.

What could be the cause?
Many thanks guys~~ 

[ms4446]

TimeoutException you're encountering in your Google Cloud Pub/Sub setup likely stems from several potential issues when transitioning from a local emulator to a server environment.

First, consider the network configuration. Ensure that your server’s firewall, both on the machine and within your Google Cloud project, allows outbound traffic on port 443 (HTTPS) to communicate with the Pub/Sub service. In restricted network environments, enable Private Google Access to allow your server to reach Google APIs without using public IPs. If your server operates behind a proxy, configure the proxy settings in your Pub/Sub client library using environment variables HTTPS_PROXY or HTTP_PROXY.

Next, examine authentication and authorization settings. Verify that the service account used by your application has the necessary Pub/Sub permissions, specifically roles/pubsub.publisher for publishing and roles/pubsub.subscriber for subscribing. Ensure that your application correctly loads these credentials by setting the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to your service account’s JSON key file.

Flow control settings in your subscription may also need adjustment. If your subscriber is overwhelmed by the message volume, it might not acknowledge messages in time, leading to timeouts. Adjust the setMaxOutstandingMessages or setMaxOutstandingBytes parameters in your subscriber configuration to control the number of unacknowledged messages your subscriber can hold.

Additionally, consider the client library configuration. Increasing timeout values in your publisher configuration, such as setPublishTimeout, can be beneficial if your messages are large or your network is slow.

Begin by enabling detailed logging in your Pub/Sub client library and on your server to gather more information about connection attempts and potential errors. Use the Google Cloud Console to monitor subscription metrics, specifically looking for high latency, errors, or unacknowledged messages.

It’s important to remember that the emulator might not perfectly replicate all production behaviors. Network latency, authentication mechanisms, and flow control can differ significantly in a cloud environment compared to a local emulator setup. Here is a Java example to adjust flow control settings:

Subscriber subscriber = Subscriber.newBuilder(subscriptionName, receiver)
    .setFlowControlSettings(
        FlowControlSettings.newBuilder()
            .setMaxOutstandingElementCount(1000) // Adjust as needed
            .setMaxOutstandingRequestBytes(10 * 1024 * 1024) // Adjust as needed
            .build())
    .build();
subscriber.startAsync().awaitRunning();

The error message mentions Armeria, indicating the use of a framework built on top of gRPC. Ensure you follow the framework’s specific guidance for Pub/Sub integration.

By systematically addressing these potential issues and following the outlined troubleshooting steps, you should be able to identify and resolve the root cause of the TimeoutException in your Pub/Sub setup.