The Pub/Sub pattern is not something new but with the growing complexity of event systems combined with advances in distributed computing Pub/Sub is growing in popularity. In this article, I will explain the high-level Pub/Sub pattern and try to give you some reasons to include in your projects of varying sizes.
When building complex event-driven systems that require two or more different components to communicate with each other the traditional and simplest approach is to wire those components directly together. In some cases, this is done using web service APIs, flat-file exchange, or through shared data stores like databases. These approaches work but come with a set of challenges:
- The modules become coupled together and a change to one component might require updates to any other components that it interacts with
- New integrations become time-consuming to build and test
- There is a general lack of scalability as the volume of events and number of integration points grows
Event for smaller hobby projects that involve just 2 or 3 components and very little data I find that this typical direct integration approach is extremely detrimental and discourages me from making iterative enhancements. I find it daunting that to add a new module I need to revisit and reconfigure several components.
The Pub/Sub Pattern
Enter the Pub/Sub pattern. Instead of wiring parts of your infrastructure directly together the communication is done through a set of channels. A module in your infrastructure can either be a publisher to a channel to send events or a subscriber to read events. This presents several benefits and helps address some of the disadvantages outlined above.
- The modules become completely decoupled from each other and their job when it comes to integration is to simply properly format and publish data, or receive payloads as subscribers and know how to process them
- Development of new components is simplified as developers don’t need to worry about various integration points and the network layer
- Testing can be isolated and simplified by subscribing unit test scripts to topics or publishing simulated inputs
- If the correct Pub/Sub solution is used your system should become infinitely scalable without touching any other components of your infrastructure
The Pub/Sub pattern is becoming the norm for building scalable and maintainable solutions. If you are not using this pattern as part of your infrastructure today I highly recommend at least evaluating it as an option as the benefits of reduced maintenance and future development costs would likely outweigh the effort needed to migrate your solution to this scalable pattern. As I mentioned about I highly recommend this pattern for even smaller hobby projects to improve the solution's future flexibility and to get experience with the Pub/Sub pattern. I use Google's Pub/Sub for most of my personal projects as it comes with 10GB of free data transfer per month which more than covers all my needs.