pubsub subscription

[Tzachi_Israel]

Hi
I have pubsub subscription with the the following settings:

Delivery type = Pull

Message retention duration = 10 minutes

Expiration period= 14 days
Acknowledgment deadline – 60 seconds

I have also ticked  Enable exactly once delivery and Message ordering.
Retry policy is Retry immediately.

I am using Airflow (Composer) to read from the subscription (and write to BigQuery table).
I want to read only the latest messages every time.
Obviously I can make sure by using the Airflow script not to write twice the same message, but is there anything I can do to configure the subscription so I won't read twice the message?

[caryna]

Hi @Tzachi_Israel,

Welcome to Google Cloud Community!

The Pub/Sub is configured with default at least once message delivery but no bounds on duplicate rates. In pull subscription, you can enable exactly-once delivery (EOD) feature and currently this is not supported in push subscription. 

This feature offers the following:

• Works when subscribers are connected to the service within a cloud region. 
• If a message is outstanding, no duplicates will be delivered and no redelivery will occur once the message has been acknowledged. Sometimes, the problem is subscriber is too slow to process and acknowledge messages within the deadline. The solution you can consider is to reduce the maximum number of outstanding messages to have less messages to process within the deadline. You may refer to this documentation.

For further information on EOD offers, review this documentation for more insights.

Also, it is important to have knowledge about the difference between redelivery and duplicate messages. This is for you to understand the type of messages you are receiving and how you will troubleshoot it in the future. 

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.