At work we plan to replace a totally overkill Kafka instance with a combination of SNS and SQS. I don’t want to get into a discussion on the feature difference between these two, for our particular use-case SNS and SQS are a good fit.
Let me lay it out real quick:
- a single application publishes events to a single SNS topic
- multiple other apps are interested in these events
- for each interested app a SQS queue is created which subscribes to the SNS topic
This way we effectively implemented a fan-out pattern using SNS and SQS. With a queue for each subscriber.
The thing I’m wondering now - and wasn’t able to find any meaningful content on - is how to other people are running this particular setup. We will setup most of this up using Terraform, and while this is perfectly fine for the SNS topic I’m not so 100% certain about the SQS queues.
As we have a bunch of apps interested in these events - with a single SQS queue for each one - setting the SQS queues up in Terraform would mean we would have to remember creating a new queue for each new app which wants these events.
From this perspective it seems easier to have some shared logic which provisions a queue for each new app as soon as this app wants to start consuming events. As we’re running in Kubernetes this would most likely be an initContainer which ensures the SQS queue exists for this particular service.
Now here is the thing: while this sounds good on paper, I have no idea if this actually is a good idea (or maybe even an anti-pattern?). Do you have any experience with an approach like this, or input on how you’re running a similar setup at work?