GCP PubSub : Message Ack Latency has Increased with Java17

[rahulgoti1]

I'm running service on GAE to consume the message from GCP Pubsub with below configuration and it takes only ~40ms to ack the messages.
Java8 : 
Spring boot : 2.1.5.RELEASE
GCP Pubsub : 1.82.0

- Total time taken by processing 3 out of 3 messages by 38ms

With Java 17 and the configuration below, we have observed an increase in message acknowledgment latency in GCP PubSub : 
Spring boot : 2.7.9
GCP Pubsub : 1.82.0 or 1.129.0

- Total time taken by ack processing 3 out of 3 messages by 234ms 

The main culprit method is : 

public Boolean deleteMessages(final List<String> ackIds, final PubSubConfig pubSubConfig) throws Exception {

final SubscriberStubSettings subscriberStubSettings =
SubscriberStubSettings.newBuilder()
.setTransportChannelProvider(
SubscriberStubSettings
.defaultGrpcTransportProviderBuilder()
.setMaxInboundMessageSize(20 << 20) // 20MB
.build()
).build();

try (SubscriberStub subscriber = GrpcSubscriberStub.create(subscriberStubSettings)) {
final String subscriptionName = ProjectSubscriptionName.format(pubSubConfig.getProjectId(), pubSubConfig.getSubscriptionId());

if (!ackIds.isEmpty()) {

// acknowledge received messages
final AcknowledgeRequest acknowledgeRequest =
AcknowledgeRequest.newBuilder()
.setSubscription(subscriptionName)
.addAllAckIds(ackIds)
.build();

subscriber.acknowledgeCallable().call(acknowledgeRequest);

}
return true;
}
}

[ms4446]

The increase in message acknowledgment latency when migrating from Java 8 to Java 17 in your Google Cloud Pub/Sub application can be attributed to several factors related to changes in the Java runtime, library versions, and configuration settings. Here's a revised approach to diagnose and address this issue:

1. Prioritize Library and Configuration Checks:

• Update google-cloud-pubsub Library: Begin by upgrading to the latest stable version of the google-cloud-pubsub library that supports Java 17. Incompatibilities between older library versions and Java 17 could be a major cause of the latency increase.
• gRPC Settings: Review and adjust gRPC transport settings in SubscriberStubSettings. Ensure they are optimized for Java 17 and your specific workload. Pay close attention to keepAlive settings, message size limits, and thread pool configurations.

2. Analyze Java Version Differences:

• Profile the Application: Use profiling tools like JVisualVM or Java Flight Recorder (JFR) to identify bottlenecks in the acknowledgment process. Look for areas where Java 17's performance characteristics might differ from Java 8.
• Update Dependencies: Ensure that all dependencies, including gax, protobuf, and grpc, are up-to-date and compatible with Java 17. Outdated dependencies can lead to unexpected behavior and performance issues.
• JVM Options: Compare JVM options between Java 8 and Java 17 environments. Pay particular attention to garbage collection settings. For example, Java 17 defaults to ZGC, which might require different tuning than the G1GC commonly used in Java 8. Consider experimenting with different garbage collectors or adjusting their parameters if necessary.

3. Optimize Pub/Sub Acknowledgment Code:

• Batching Acknowledgments: Ensure that your deleteMessages method efficiently batches acknowledgments to minimize the number of RPC calls to the Pub/Sub service.
• Network Latency: Check for increased network latency between your GAE service and the Pub/Sub endpoint, especially if they reside in different regions. High network latency can significantly impact acknowledgment times.

4. Spring Boot Considerations:

• Spring Boot Compatibility: Verify that the Spring Boot version (2.7.9) used with Java 17 is fully compatible with both the Java version and the updated google-cloud-pubsub library.
• Spring Boot Configuration: Review and update any Spring Boot configurations related to Pub/Sub or asynchronous processing to ensure they are optimized for Java 17.

5. Error Handling and Retries:

• Retry Settings: Review and optimize retry settings in SubscriberStubSettings. Ensure that retries are configured appropriately to handle transient errors without introducing excessive delays.

Additional Recommendations:

• Benchmarking: Conduct thorough benchmarks with different configurations and Java versions to identify the optimal setup for your specific workload.
• Monitoring and Logging: Implement detailed logging and monitoring to capture acknowledgment latency metrics, error rates, and other relevant data. This will help you pinpoint the root cause of the issue and track the effectiveness of your optimizations.

Example Updated Code (with batching):

public Boolean deleteMessages(final List<String> ackIds, final PubSubConfig pubSubConfig) throws Exception {
    SubscriberStubSettings subscriberStubSettings =
        SubscriberStubSettings.newBuilder()
            .setTransportChannelProvider(
                SubscriberStubSettings.defaultGrpcTransportProviderBuilder()
                    .setMaxInboundMessageSize(20 << 20) // 20MB
                    .build())
            .build();

    try (SubscriberStub subscriber = GrpcSubscriberStub.create(subscriberStubSettings)) {
        String subscriptionName = ProjectSubscriptionName.format(pubSubConfig.getProjectId(), pubSubConfig.getSubscriptionId());

        if (!ackIds.isEmpty()) {
            AcknowledgeRequest acknowledgeRequest = AcknowledgeRequest.newBuilder()
                .setSubscription(subscriptionName)
                .addAllAckIds(ackIds)
                .build();

            subscriber.acknowledgeCallable().call(acknowledgeRequest);
        }
        return true;
    }
}

[rahulgoti1]

Thanks @ms4446  I found the issue.