Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

Best Practices for Dead Letter Topic originating from a One Topic - Many Subs?

Hi Google Cloud Community,

I've been working with Pub/Sub for a few months now and enjoying how straightforward it is compared to RabbitMQ.  I can't seem to find the answer to how to handle/re-play messages in a Dead Letter Topic (DLT) when the messages originated from a subscription whose topic has more than one subscription.

Things are straightforward when a DLT comes from a subscription whose topic has only one subscription.  You can handle those messages with a cloud function, and dataflow and there are many options to move DLT to a DLT subscription and back to the original topic.

However, things are not very clear when you have one topic with two subscriptions. If one subscription has an issue with retries that end up in the DLT, the only supported options seem to route messages back to the original topic. This is fine for the subscription that missed the message, but the other subscription will get a duplicate message.

Pubsub_Seek_Replay_DeadLetterExchange__Lucidchart.png
I've come up with some unideal workarounds, but I rather avoid them if possible.

  1. Every Subscriber needs to subscribe to the main subscription AND a dead-letter subscription OR
  2. Every Subscription must be set to filter out attributes that are added when it hits a Dead Letter Topic. You can remove those filters when resolving issues, and put them back OR
  3. In situations where the subscriber is a push type, you could make a dead letter subscription that is a pull. If there are messages there, you can change it to push and target the URL that actually needs the replies.

Ideal Solutions/Feature Requests

  1. Pub/Sub has a way to easily transfer messages in dead letter subscriptions to a target subscription (not a topic). This is unlikely possible since MessageID is unique to the topic.
  2. Pub/Sub has a rule that prevents a subscription from receiving messages from two different topics. Add an exception so that if it's a subscription of a DLT, then allow it.

Other Notes:
Message Subscribers should be set to ignore duplicates by using MessageID and Publishing Time. However, messages coming from a dead letter topic have new unique MessageID/Publishing Time making the subscriber unable to determine whether not the message is a duplicate.

Thank you in advance for any assistance or thoughts.

0 0 55
0 REPLIES 0