Pub/Sub message stuck for specific ordering_key (user_id)

[rayhanmustofa]

Hi, my workplace has been using pub/sub as messenger in our system.
We use user_id as the ordering key, and this week our system faced problem where messages from one specific user (which also is the ordering key), are always successfully published but subscriber/listener does not receive it at all. Other messages from different user does not get this problem.

This happens for all message of the one specific user/ordering_key for several days until we finally restarted our listener/subscriber, only then it finally works normally again.

I'm having assumption that there's something wrong with the subscriber ordering mechanism on the specific ordering key at one point, causing the newer message could not be received.

Does anyone ever face this kind of problem?
And how do I prevent this from happening in the future?

[jangemmar]

Hi @rayhanmustofa,

Welcome to Google Cloud Community!

Unacknowledged messages for a given ordering key can potentially delay delivery of messages for other ordering keys, especially during server restarts or traffic changes. The Re-deliveries of a message trigger redelivery of all subsequent messages for that key, even acknowledged ones. 

To maintain order of messages, here are the workarounds that you can do:

• Ensure timely acknowledgement of all messages. If timely acknowledgement is not possible, consider using a dead-letter topic to prevent indefinite message holding.
• Your publisher client can republish all the failed messages in order or publish a new set of messages
• Use Pub/Sub metrics and logs to identify any anomalies related to message delivery, processing, or subscriber behavior.

Here are the best practices for ordered messaging in subscribing: 

• Choose either StreamingPull or Pull subscriptions
• Enable message ordering in the subscription
• Acknowledge message in order

Note: Ordered delivery does come with some tradeoffs. In the ordered delivery case, failover requires coordination to ensure the messages are written to and read in the correct order.

I hope the above information is helpful.