What do such useful patterns for implementing reliable high-load software systems like CQRS, Event Sourcing, EDA have in common? All of these patterns are based on working with event streams. But for all the simplicity of this concept, there are surprisingly few ready-made tools for its implementation, and in fact only one is popular – Kafka.
I must say that often people do not make a distinction between the concepts of event streams and message queues, which leads to misunderstanding and underestimation of the prospects of using event streams. But enough abstract talk, let’s get to the point.
What are event streams and what makes them different from queues? Event streams imply long-term storage of messages in a continuous sequence. Each message in a stream has its own sequence number. These features of event streams have obvious significant advantages over using message brokers:
- You always have the opportunity to process not only current events, but also the history of events, which is available to you automatically, without any additional effort from your.
- Adding new event receivers does not affect the amount of data that the server processes, all receivers read events from the same streams.
- There is an opportunity, not available in other ways, to easily implement guaranteed delivery of an event in the correct sequence and without loss. And this is worth a lot, unless of course the reliability of the software you develop is important to you.
- Traceability – you can always see the history of events and how your microservices reacted to them and in many cases even reprocess these events if necessary.
- Extensibility of the system – it is always possible to add a new microservice or report and feed it the entire event history.
How does a miracle work? How is it guaranteed that delivery is in the correct sequence and that each event is handled exactly once? It’s pretty easy to do this if you can get any event by number. When event is received, the microservice processes it and, when making changes to state in the database, adds there the number of this event. If something went wrong and the microservice was stopped at that moment, after restarting it requests the number of the last processed event from the database and starts processing other events starting from the next sequence number. This ensures that the event does not affect the state of the microservice twice and is not lost.
Of course, there are some issues that will need to be addressed in such systems, for example, what to do when changing the structure of events in case of releasing new version of microservices that these events generate or how long to store the history and to what depth. Or what to do if incorrect events were generated and stored. But these questions are not unique to event streams, similar problems arise when working with any data stores, and I think here you can come up with something or peep from others. Perhaps I will discuss this topic in future posts.
And summing up, I want to please you, if you don’t really like Kafka for its heaviness, there is also NATS with its JetStream and EventStoreDb, undeservedly overlooked. Yes, such systems are the future, and I suggest that you touch this future right now, moreover, many have been using it for a long time.