Introduction to event-based programming

Event-driven programming is a great approach for building complex systems. It embodies the divide-and-conquer principle while allowing you to continue using other approaches like synchronous calls.

When discussing event-based systems, several different terms often refer to the same concept. For simplicity, we’ll primarily use the terms listed below in bold:

• Event, message, and notification
• Producer, publisher, sender, and event source
• Consumer, receiver, subscriber, handler, and event sink
• Message queue and event queue

What is event-driven programming?

Event-driven programming, or event-oriented programming, is a paradigm where entities (objects, services, and so on) communicate indirectly by sending messages to one another through an intermediary. The messages are typically stored in a queue before being handled by the consumers.

Unlike in using direct calls, event-driven programming completely decouples the producer from the consumer, leading to some noteworthy benefits. For example, multiple producers and multiple consumers can all collaborate to process incoming requests. Retrying failed operations and maintaining an event history are also simplified. Event-driven programming also makes it easier to scale large systems, adding capacity simply by adding consumers.

Let's look at the actors of event-based systems: events, producers, consumers, and message queues.

What are events?

Events are pieces of data that are sent by producers and eventually consumed by consumers.
For example, a mouse down event typically contains the following information:

• Mouse pointer coordinates
• Which mouse buttons are pressed

Events are usually structured but can sometimes just be blobs of data (such as a chunk of JSON) that consumers should know how to parse and handle.

What are producers and consumers?

Producers are entities that generate events and send them to a message queue. Consumers either subscribe to receive new events or poll periodically from the queue. Key to the event-driven paradigm is this main point: Producers and consumers are blissfully unaware of one another and interact through the message queue only.

What are message queues?

Message queues are repositories of messages (events). The messages may be stored in memory and/or durable storage. Message queues are typically partitioned into topics. Producers and consumers send and receive messages for particular topics separately.

A message broker is responsible for making sure messages that are sent to the queue are delivered to all subscribed consumers. Some message brokers in popular use today include RabbitMQ, Redis, and Apache Kafka.

Consumers use pull or push mode to consume messages. In pull mode, the message queue keeps all the messages for all the consumers. When a message has been received by all consumers, that message may be removed from the queue (depending on retention configuration). Some message queues allow consumers to pull historical messages.

In push mode, the message queue pushes any new message to all current subscribers.

What makes event-driven programming special?

Let's explore some of the properties of the event-driven programming paradigm and see what makes it cool—and what makes it tricky. Our focus will be on push-based systems.

Tightly-coupled versus loosely-coupled workflows

Traditional call-based workflows are tightly coupled. If object A has some information X that is relevant to object B, then object A needs to satisfy the following requirements to deliver this information successfully to B:

1. It needs to know that B is interested in X
2. It needs a reference to B
3. It needs to know about B's interface (for example, a setX() method)
4. It needs to call B to deliver X
5. If something goes wrong, then it needs to know what to do (such as retry or raise error)

That’s a lot of requirements.

To further complicate matters, consider a system in which B is not the only one that is interested in X. Maybe C and D are also interested in X, and they have different interfaces for receiving X. Now, A needs to get references to C and D, along with their respective interfaces. A also needs to consider what to do in case of partial failure (for example, if B and C received X successfully, but D failed).

Now, let's look at the requirements on object A for a loosely-coupled, event-based system:

1. A sends an event X to the event queue

That’s it. Now B, C, and D can all subscribe to receive event X—if they are interested. The event framework managing the queue can have policies about error handling and retries, and these can often be specified by consumers.

The downside of loosely-coupled workflows, however, is that these interested parties are difficult to see in the code or in a call graph and workflow.

Request-response versus publish-subscribe

The request-response approach to communication is typical of traditional web development. For example, a user navigates to a new URL, the browser requests a web page, and the server responds with information. This is a typical pattern of remote procedure call (RPC) systems.

The publish-subscribe approach (PubSub for short) tackles a different use case: fire and forget. The publisher simply publishes an event to the queue without needing any confirmation or response.

Both approaches have their place. While it is possible to implement asynchronous request-response using events—as we will see later—doing so is cumbersome.

Synchronous versus asynchronous calls

Synchronous calls are your garden variety function calls. You call a function with some arguments, and you get the result on the same thread. The thread will be blocked until the function call returns.

Asynchronous calls return immediately and don't wait for the computation to complete. The result will arrive later on a separate thread. Event-based systems that rely on PubSub often use asynchronous calls to receive the events. The consumer subscribes to specific topics or events, and the events are handled in a separate thread.

Retries and replays

When you make a call to another object or service, there is always the possibility of an error. In general, the caller can take any of several actions:

1. Fail and return an error
2. Log the failure but keep operating
3. Fallback to some alternative
4. Retry

Often, the event framework can be configured to retry sending the events to consumers, making the retry action easy to implement.

However, if a consumer receives an event but fails while processing it, there is no direct way to communicate that failure to the producer. Within event-driven architectures and programming, this is a non-issue because the producer is unaware of the consumers and is not responsible for handling their failures.

The event framework managing the queue can retry several times in case the failure is intermittent. If the failure persists, the event can be stored for later processing and proper error messages can be logged.

The almighty queue depth

In event-based programming, determining the state and performance of the system is very straightforward. If the number of events in your queue—also known as queue depth—grows, then you need more consumers. If your consumers are sitting idle, waiting to process events, then your producers are not sending events.

You can find issues by checking which queues grow uncontrollably and which queues starve. Sometimes, it is more useful to track the age of messages in the queue rather than the number of messages.

Event delivery options

In PubSub systems, there are primarily three different models for delivery: exactly once, at most once, and at least once.

Exactly once

In this delivery model, every consumer is guaranteed to receive exactly one copy of every event. Actually, this model is very difficult to accomplish. When a consumer fails or disconnects from the event queue, it’s often unknown if the consumer received the event (but failed to acknowledge) or if it didn't even receive the event at all.

At most once

Here, some consumers may not get all events, but they will never get duplicates. This model is the easiest to implement. Events are just shotgunned out one time to all of the consumers. Either the consumers get the events or they don’t. No retries.

At least once

In this most common model, every consumer is guaranteed to receive every event, but some consumers might also get duplicates. The event framework will retry several times until it gets a confirmation from the consumer. Since duplicate events are possible, the consumer should be idempotent, meaning it can safely process duplicate events.

In pull-based systems, the consumers are responsible for pulling messages from the queue. If the queue retains historical messages, then consumers can pull messages as many times as they want.

How does event-based programming work?

Let's examine an event queue with multiple producers and multiple consumers.

Generating events

The producers generate events and send them to a specific topic. In the diagram, you can see that we have two producers. Producer 1 sends Event A to Topic 1. Producer 2 sends Events B and C to Topics 2 and 3.

Subscribing to events

Each topic may have a group of subscribers (consumers). This is relevant mostly for PubSub systems. Consumers 1 and 2 subscribe to Topic 1, which creates Group 1 and Group 2 to represent the subscriber list. Consumer 2 also subscribes to Topic 2, which creates Group 3.

When Producer 1 sends Event A to Topic 1, Consumers 1 and 2 will receive it through their subscriber groups. When Producer 2 sends Event B to topic 2, Consumer 2 will receive it through Subscriber Group 3.

When Producer 2 sends Event C, no consumer will receive it since no consumer has subscribed to it. The event may be discarded, or it may be stored in Topic 3, waiting for consumers to subscribe.

Configurable retention

Some queues immediately dispose of events once they are delivered to all subscribers. Other queues might keep events around. For those queues, it is useful to configure how long to keep events. This can be done based on event age, the number of events in the queue, or the total size of queued events.

Event-based patterns

Event-based programming opens the door to several interesting and useful patterns.

Single producer / single consumer

The simplest configuration for events is a single producer and single consumer. In this pattern, the producer and consumer are isolated from one another, free to operate independently.

Single producer / multiple consumers

A single producer with multiple consumers is useful when handling an event takes longer than generating an event. In this case, multiple consumers can share the load to handle the events together. A load balancer is a good example of this pattern.

The dead letter queue (DLQ)

When an event fails to be processed even after several retries, it is common to remove it from the main queue and post it to a dedicated DLQ. Engineers can analyze these events later to determine the root cause.

Time to live (TTL)

Systems that generate many events may want to keep a history, but not forever. Generally speaking, the value of data degrades with time. For example, when debugging a production issue, you may want to see the stream of events since the last deployment, but probably not events from five years ago. A common pattern is to assign a TTL to each stream of events or each topic, and those events will be discarded when their TTL expires.

Asynchronous request-response with events

Event handlers typically don't return results. In some cases, it is useful to perform a query or an action asynchronously and then get the result later. Polling is common in these situations, but can introduce a lot of overhead.

An event-based alternative is to establish request topics and response topics. The caller sends a request to the request topic and subscribes to the response topic. When the response is ready, the handler of the original request sends the response to the response topic, and the original caller can process the response.

Event splitting

Event splitting is useful when the same event needs to be handled by multiple entities. This is different from the single producer/multiple consumers pattern, in which one consumer handles each event. For example, consider an event that needs to be processed for storage as well as aggregated for later analytics. One consumer can store the event in the proper data store and another consumer can do the aggregation. This division of responsibilities is the hallmark of loosely-coupled systems. The storage code is unaware of the aggregation code even though both operate on the same data.

Examples

The following examples of event-driven programming are from a blocks puzzle game I developed with my sons. You can find the source code for the project at https://github.com/the-gigi/blocktser.

Blocktser is a simple game that runs in a browser. It is implemented in TypeScript, the Canvas API, and the phaser.io framework. In Blocktser, you drag shapes from the staging area at the bottom and move them around the screen to eventually drop them in the main area. You get points when you complete rows or columns. If you can't place a shape, the game is over.

Here is a screenshot:

Subscribing to GameObject events

The shapes are Phaser Game objects. Conveniently, Phaser already translates raw Canvas API mouse events to higher-level events.

To receive and handle events, we need to take several actions. First, we make our shapes draggable by setting the container object to be interactive (setting the mouse cursor) and draggable.

this._container.setInteractive({ cursor: 'pointer' })
scene.input.setDraggable(this._container);
Copy to clipboard

Then, we call the scene input object's on method to subscribe to the drag, dragstart, and dragend events. Triggering (producing) the relevant event calls the provided event handling function (the consumer).

scene.input.on('drag', function(pointer, gameObject, dragX, dragY) {
  self.onDragging(self, pointer, gameObject, dragX, dragY)
})

scene.input.on('dragstart', function (pointer, gameObject) {
  self.onDragStart(self, pointer, gameObject)
})

scene.input.on('dragend', function (pointer, gameObject) {
  self.onDragEnd(self, pointer, gameObject)
})
Copy to clipboard

Handling events

The subscription event handlers delegate the actual logic to the onDragging, onDragStart, and onDragEnd methods of the Shape class.

The Shape's onStartDrag method ignores drag start events on other objects. Then, it performs some logic to scale the shape and update it. (Dragging the shape changes its size.) Finally, it calls onDragStart on a list of dragHandlers. This accomplishes the event splitting pattern we discussed earlier.

onDragStart(shape, pointer, gameObject) {
  if (shape._container !== gameObject) {
    return
  }

  shape._unit *= shape.dragScale
  shape.updateShape(true)
  shape.dragHandlers.forEach((h) => h.onDragStart(shape))
}
Copy to clipboard

The Shape's onDragging method is called whenever the mouse is moving during a drag operation. The event handler here updates the shape's position according to the delta since the last position. Then, it invokes the onDragging method of each of the dragHandlers members.

onDragging(shape, pointer, gameObject, dragX, dragY) {
  if (shape._container !== gameObject) {
    return
  }
  const dx = dragX - gameObject.x
  const dy = dragY - gameObject.y
  shape._container.x += dx
  shape._container.y += dy
  shape.dragHandlers.forEach((h) => h.onDragging(shape))
}
Copy to clipboard

Last but not least is the Shape's onDragEnd method. Here, we see the familiar structure of ignoring the event if it's coming from a different object, then resizing the shape and invoking the drag handlers.

if (shape._container !== gameObject) {
  return
}

shape._unit /= shape.dragScale
shape.updateShape(false)
shape.dragHandlers.forEach((h) => h.onDragEnd(shape))
Copy to clipboard

Transforming events to high-level application events

The event handlers we've seen in the previous section were defined by the Phaser framework, and they were in terms of Phaser game objects and their containers. In Blocktser, there is another layer of abstraction that operates in terms of Blocktser shapes. The interfaces.ts file defines the following interfaces:

export default interface ShapeDragHandler {
  onDragStart: (shape: Shape) => void
  onDragEnd: (shape: Shape) => void
  onDragging: (shape: Shape) => void
}

export default interface MainEventHandler {
  onDrop: (shape: Shape, ok: boolean) => void
}
Copy to clipboard

Let's focus on the onDrop event. It is triggered when a shape ends its dragging over the main area. First, the onDragEnd method is called:

onDragEnd(shape: Shape) {
  this.settleShape(shape)
  this.destroyPhantom()
}
Copy to clipboard

The onDragEnd method calls the settleShape method, which determines whether or not the shape was dropped properly. Then, it calls the onDrop method with either true or false.

settleShape(shape: Shape) {
  // bail out if not on grid
  if (!this.isOnGrid(shape)) {
    this.shapeEventHandler.onDrop(shape, false)
    return
  }

  // bail out if shape intersects with any occupied cell
  const [row, col] = this.findGridLocation(shape)
  if (!this.canShapeSettle(shape, row, col)) {
    this.shapeEventHandler.onDrop(shape, false)
    return
  }

  // Populate cells with the shape's image
  ...

  this.shapeEventHandler.onDrop(shape, true)
}
Copy to clipboard

The onDrop handler operates at a higher abstraction level of the scene, which includes both the main area and the staging area of the game. It understands the meaning of a shape being dropped. If the shape was dropped correctly in the main area, then it takes several actions:

• play sounds
• update the staging area if needed
• clear completed rows and/or columns
• update the score
• check if the game is over

If the shape was dropped improperly, then it simply returns to the staging area.

Let's follow the chain of events from the OS level all the way to Blocktser's shape drop event:

• The OS generates a stream of mouse down, mouse up, and mouse move events.
• The browser intercepts these events and reflects them to Canvas as mouse events.
• The Phaser framework handles these events and translates them to GameObject drag events.
• Blocktser handles these events, using the event splitting pattern to send through its own event interfaces to multiple consumers.
• Finally, Blocktser translates the dragEnd event to a higher-level shape drop event.

Conclusion

Event-based programming is an incredibly useful paradigm, with use cases ranging from UI in a single-process system to communication between services in large-scale distributed systems. The loosely coupled nature of event-based programming provides many benefits, though it may require a shift in how you think about inter-object communication.

With the trend of software development moving away from monolith applications toward distributed systems and decoupled microservices, an event-driven paradigm for architectures and programming is certainly here to stay. The event queue solution you choose to use—whether that be Kafka or any other message broker—is one that requires wise consideration. At the very least, it’s time to consider whether your fleet of microservices will get a boost by shifting toward the event driven development.

About Gigi Sayfan

Gigi Sayfan is the DevOps team manager at Helix — a bioinformatics and genomics start-up and author of several books and hundreds of technical articles.

Gigi has been developing software professionally for more than 20 years in domains as diverse as instant messaging, morphing, multimedia applications for game consoles, brain-inspired machine learning, custom browser development, web services for 3D distributed game platforms, IoT sensors and virtual reality.

Find out more on Amazon.com!

--

Not using Aiven services yet? Sign up now for your free trial at https://console.aiven.io/signup!

In the meantime, make sure you follow our changelog and blog RSS feeds or our LinkedIn and Twitter accounts to stay up-to-date with product and feature-related news.