I'm developing an application which would subscribe to PubSub and pump the data to downstream services after clean up, I'm looking out ways to recover from a specific message if there is any failure in my pipeline.
Does the message Id of messages are in lexicographical order, If so, can I use that to filter out the older messages before the last successfully processed message?
I'm guessing you have tried this but just to be sure, can you not just use publish_time to order the messages and check that way? The precision of that timestamp is pretty solid.
Another way would be to send the messages that have some processing error to another Pub/Sub topic so you can re-process them in some other way at your leisure.
Check out these videos:
Pub/Sub ordering: https://youtu.be/f8nQk62IkHM
Understanding Pub/Sub ordering: https://youtu.be/S2evHtbl4F8
Post-processing / creating Pub/Sub microservices: https://youtu.be/tsKZ_u_uIAs