In an event-driven microservice architecture, services communicate with each other through event messages. Producers generate business logic and they emit events that will be handled later down in the stream. Consumers receive business logic and they listen to and consume events that producers have emitted. In this article, we will go into detail about this type of architecture, highlight the benefits and downsides, as well as talk about tools that implement this architecture.

What is event streaming?

By definition, events are simply messages that happen within a software system and are communicated back and forth amongst microservices. For instance, an event could include a customer logging into a service, a distribution center updating its inventory, or a payment transaction completing.

Event streaming allows microservices to analyze data that pertains to an event and respond to that event in real time. This allows events to be passed from one microservice to another asynchronously, which progresses the business logic forward.

Benefits of event-driven architecture

A big challenge with microservices is the coupling that can occur between the services because they have to rely on each other’s state. That is, they rely on each other to be available at all times. But what happens when the dependent microservice is moved offline? That means that services can’t respond to requests and the requestor microservice will lose data. We could retry requests when this happens, but this adds an extra layer of complexity and cost. Not to mention, what happens if the microservice we are calling is offline for an extended period of time? Event-driven architectures allow us to store streams of events durably and reliably for as long as we want. This is beneficial because it allows us to read those events when the microservice is back online so that we can process those business logic retrospectively so that we don’t lose data.

With an event-driven architecture, we publish and subscribe to streams of events. You can even continuously import/export data from other systems in real-time, which allows you to analyze data in real-time! This is a benefit to us because we can analyze business data so that leaders can make well-informed business decisions that can impact the business.

Kafka vocabularies

The most prevalent and popular tool for event streaming is Apache Kafka, which allows microservices to send, store, and request data when and where they need it. In Kakfa, producers and consumers are decoupled and agnostic of each other, which allows it to be highly scalable.

Events in Kafka have names assigned to them, which Kafka refers to as “topics”. A topic is similar to a folder and the events are the files in that folder.

Setting up Kafka using Docker and Docker Compose

Let’s try our hands at setting up a Kafka server, publishing an event, and then subscribing to it. This will help us understand how Kafka works from a programmatic perspective and also allow us to be better understand how Kafka operates under-the-hood.

In order to install Kafka on any machine, you can download it from the official website and then start the server. However, we will be using Docker and Docker-Compose because it allows us to virtualize it:

1. First, install Docker and Docker Compose on your OS. Docker Compose is included if you choose to install the bundled version of Docker on Mac and Windows.

2. Check that you are able to access Docker and Docker Compose:

docker --version

docker-compose --version

3. We will need to create Kafka’s Zookeeper service in order to keep track of Kafka cluster nodes, topics, partitions, etc. Zookeeper gives us a way to coordinate tasks, state management, configurations across a distributed system, so it is a must for our use case.

4. Creating a docker instance from scratch every time we need to is very tedious. Docker Compose provides us a way to specify the commands that Docker should run in order to stand up your Kafka server. It does this via docker-compose.yml.

5. Use the following in your docker-compose.yml file:

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    container_name: zookeeper
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    container_name: kakfa
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

6. From the docker-compose.yml file, we see that there are two Docker services defined: zookeeper and kafka. We also have defined the docker-compose.yml file so that the kafka service relies on the zookeeper service (depends_on). We use bitnami/zookeeper:latest, which is a Docker image that you can find on Dockerhub. We also define the default Kafka port of 9092 and a default Zookeeper port of 2181. Additionally, we define several environment variables.

7. Let’s run this Docker-Compose file:

docker-compose up -d

This will, by default, run the docker-compose.yml file, create the zookeeper and kafka images, and run them in detached mode, which basically means that they will run in the background.

8. Once that finishes, the docker images should appear in the list if you run the following:

docker images

9. Next, execute the following:

docker ps

This will allow you to check if your docker images are running, and on which port they are running.

10. In order to connect to the docker container, we’ll have to use a docker command to directly call the Kafka Docker container.

docker exec -it <docker-container-id> /bin/sh

This will open an interactive shell session for us in the Kafka container.

On a side note, -it allows us to have an “interactive mode” with the Docker container. The command basically tells Docker to keep STDIN open even if not attached and will allows us to continuously interact with the Docker container.

If Docker gives you the following error:

OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: "/bin/sh" is not a directory...

then run the following command and look for the CMD option. This will tell you what type of command line you have in your container:

docker inspect <docker-image-name>

11. Once you are inside the container, cd into:

cd /opt/bitnami/kafka/bin

12. From this directory, let’s create a Kafka topic:

kafka-topics.sh --create --zookeeper zookeeper:2181 --replication-factor 1 --partitions 1 --topic <TopicName>

In this command, you are able to create a Kafka topic and specify the Zookeeper and its port, as well as the topic name. 13. To verify the list of topics that you have created, you can execute the following:

kafka-topics.sh --list --zookeeper zookeeper:2181

14. And that’s it! You can now consume or produce any messages with the name of <TopicName>!

In this article, we delved and understood more about event-driven architecture when working with microservices, defined some vocabulary terms, saw the benefits of using this architecture, as well as set up a small Kafka Docker container! I hope that this article was as useful for you as it was for me in understanding more about event-driven programming!