We are discussing the design of an application that would need a fanout queue to work.
But as far as I can tell Pub/Sub would not be suitable for this usage case as you need to create a Subscription for each running copy of the software to get a "fanout" effect.
While this can be done in code. creating a subscription takes 4-7 seconds and removing it on shutdown takes about the same. What if the app crashes? now we have an orphaned subscription.
If you connect multiple copies of the software to the same sub, only 1 of the N copies gets the message but we want to deliver a message to everyone subscribing to a topic.
The application in question is a Node.js app that runs in Cloud Run. It daily scales up and down between 2-30 copies
I know Kafka, RabbitMQ etc etc would all be able to do solve it, but we are trying to build it with as minimal maintenance and moving parts as possible while keeping it all on GCP without involving third party stuff like Confluent Cloud etc
Would greatly appreciate any feedback. Maybe I just missed some G product that can solve this for us 🙂
Update:
Subscription expiration might be an option, but minimum is 24 hours and on a busy day this could create hundreds of subs just laying around. Will these be autodeleted after 24 or just put in some inactive state where we still have to remove them manually?
LinkedIn
Twitter
