Cloud PubSub Subscription Exactly once delivery not wokring

[heisenbergChen]

hi all

I am using Pub/Sub and my Subscription settings are as follows

Acknowledgement deadline : 60 seconds

Exactly once delivery : Enabled
Delivery type : Pull

I have multiple servers subscribing to this Subscription.
When I send a message, in my understanding,
the configuration should ensure that a message is only pulled by a certain server,
and other servers should not be able to pull a message with the same MessageId until it has been nack within 60 seconds.
However, in practice, my servers seem to be pulling the same message after around 1x seconds.

Is my understanding incorrect

[ms4446]

Your understanding is mostly correct. With exactly-once delivery enabled, Pub/Sub guarantees that a message will be delivered to a subscriber exactly once. If a message is pulled by a subscriber, it will not be delivered to any other subscriber until the acknowledgment deadline has passed without an acknowledgment from the first subscriber.

The issue you are experiencing could be due to:

• Acknowledgment Deadline Exceeded: If a subscriber does not acknowledge or nack a message within the acknowledgment deadline, the message is considered unacknowledged and may be redelivered to the same or another subscriber.
• Subscriber Crashes or Restarts: If a subscriber crashes or is restarted before acknowledging a message, the message is considered unacknowledged and may be redelivered.

To address the issue:

• Increase the Acknowledgment Deadline: This provides subscribers more time to process and acknowledge messages.
• Ensure Proper Acknowledgment: Ensure that subscribers are acknowledging messages promptly after successful processing.
• Retry Mechanism: Implement a mechanism in your subscribers to retry acknowledging any unacknowledged messages in case of crashes or restarts.

[heisenbergChen]

The following is a test I conducted on my local system. I started two services, subscribed to the same subscription, and used the same service account.

When I sent a message to the topic, Service A received the message. To process this message, Service A would take approximately forty seconds. However, after about thirteen seconds, Service A received the same message again. Since the previous processing had not yet completed, Service A triggered two events.

At the same time, Service B also received the same message and exhibited the same behavior.

[ms4446]

The behavior you are observing is likely due to the acknowledgment deadline being exceeded. The acknowledgment deadline is the amount of time that Cloud Pub/Sub will wait for a subscriber to acknowledge a message before considering it unacknowledged. If a message is unacknowledged, it may be redelivered.

In your case, the acknowledgment deadline is set to 60 seconds, but Service A takes approximately 40 seconds to process a message. If Service A does not acknowledge the message within the 60-second deadline, it will be considered unacknowledged and may be redelivered to either Service A or Service B.

To address this:

• Increase the Acknowledgment Deadline: If processing typically takes around 40 seconds,consider setting the acknowledgment deadline to a higher value, giving a buffer for any unexpected delays.
• Retry Mechanism: Implement a mechanism to retry acknowledging any messages in case of processing failures.

Here's a revised example in Python using the google-cloud-pubsub library:

from google.cloud import pubsub_v1
import logging

def process_message(message):
  try:
    # Process the message.
    pass
  except Exception:
    # Log the error.
    logging.exception('Error processing message: %s', message.data)
    # Do not acknowledge the message, so it can be retried.
    return False
  return True

def pull_messages(subscription_name):
  subscriber = pubsub_v1.SubscriberClient()
  subscription_path = subscriber.subscription_path('YOUR_PROJECT_ID', subscription_name)

  def callback(message):
    if process_message(message):
      message.ack()
    else:
      message.nack()

  subscriber.subscribe(subscription_path, callback=callback)

if __name__ == '__main__':
  pull_messages('my-subscription')

This code uses the callback mechanism to process and acknowledge messages. If processing fails, the message is not acknowledged, allowing it to be retried.

[heisenbergChen]

In my case, there is only an approximately 13-second gap between receiving the first and second messages from my service, and around 13 seconds later, I receive the third message. This leaves a considerable amount of time before the ACK Deadline of 60 seconds

[ms4446]

The behavior you are observing could be due to a combination of factors:

1. Multiple Subscribers: If a message is delivered to a subscriber (e.g., Service A) and isn't acknowledged within the set deadline, it can be redelivered to another subscriber (e.g., Service B). However, given that you've set a 60-second acknowledgment deadline, it's unusual for the message to be redelivered after only 13 seconds.

2. Service Interruptions: If Service A or Service B crashes or is restarted after starting to process a message but before acknowledging it, the message will be considered unacknowledged and can be redelivered.

3. Network Errors: Network issues between your subscriber services and Cloud Pub/Sub might cause delays in message acknowledgment, leading to premature redelivery. However, this would typically result in redelivery after the acknowledgment deadline, not before.

To address the issue:

• Retry Mechanism: Implement a mechanism in your subscribers to retry acknowledging any messages in case of processing failures or service interruptions.

• Increase the Acknowledgment Deadline: While you've set it to 60 seconds, you might consider increasing it further if processing times are variable. This provides a buffer against unexpected delays.

• Monitor and Logging: Add detailed logging to capture the exact times of message receipt, processing, and acknowledgment. Monitor for any network issues or service interruptions.

• Contact Google Cloud Support: If the issue persists, reaching out to Google Cloud Support can provide deeper insights based on internal logs and metrics.

Additionally, providing more details about your environment can help in troubleshooting:

• Programming language used for your subscribers.
• Version of the Cloud Pub/Sub client library in use.
• Network configuration details.