What is Karapace?

Karapace is an open source HTTP API interface and schema registry for Apache Kafka®, the open source distributed event streaming platform.

It's an open source project, so you can find Karapace on GitHub, under the Aiven organization. Adding Karapace to your Apache Kafka project gives additional functionality that can be very useful in event-driven or other data-centric applications.

Apache Kafka^® and data schemas

One of the key features Karapace offers is the schema registry. So what is a schema registry? Let's recap on what a schema is and how we use one with Kafka.

Kafka is a very flexible and powerful tool, allowing users to add data of any format or structure to any topic. However, with great power comes great responsibility, and for many applications, flexibility in data structures is definitely not a feature. A schema is a way to describe the data format of a message payload. It will describe the fields, their data types, and default values if applicable.

Because Kafka doesn't have an opinion about the structure of your data, an additional tool called a schema registry is needed. The schema registry is a separate tool, so consumers and producers should be configured with the location of this additional endpoint. As the name implies, the desired schema can be registered with the schema registry.

Why use Karapace

Once the schema is registered, producers can validate their payload against the expected schema before producing the record on to the topic. For applications that use compression formats such as Apache Avro, the schema registry is essential. Producers register the schema they used to compress the message payload, and include the schema version number with the message. Consumers can pick up the message, check the version of the schema used, and consult the schema registry to understand the field ordering and other details.

Karapace includes schema registry functionality with support for JSON Schema, Avro and Protobuf data formats. Using a schema registry allows all components of an application to share a common data structure definition. It is also to register multiple versions of a schema with the schema registry if the data format needs to evolve.

RESTful API endpoints for Apache Kafka

Another big feature available from Karapace is the addition of a RESTful HTTP API to an existing Apache Kafka installation. This extra integration adds the ability to work with Kafka over HTTP. This lets you

• Work directly with data, producing and consuming messages.
• Work with metadata, managing the topics, messages, and producer/consumer information over the API.

The additional interface makes it easier to add some extra tooling to interact with Kafka in a tech-agnostic way. You can see this in action on your Aiven for Apache Kafka service by browsing the data you find on the "Topics" tab in the Aiven Console. This functionality is provided to the web console by use of the REST API. If the Karapace HTTP API features aren't enabled for your Aiven for Apache Kafka service when you try to access these features, you see an error message.

Karapace is open source

Karapace is a Python-based project, released and maintained by Aiven under the Apache 2.0 license. We are proud to be maintainers of Karapace, and we welcome issues, contributions and feature requests from others; some excellent features here were not built by us and we think that is a sign of a healthy open source project.

Whether you use Karapace as part of your Aiven for Apache Kafka offering, or with another Apache Kafka solution, adding schema registry and HTTP API features to the basic Apache Kafka brings the tool to a whole new level. We would love to hear what you build with yours!

Further reading

• Learn more about Karapace at karapace.io.
• Check out the project repository to find out how to install and use the tool locally, and connect to an existing Apache Kafka service.

Wrapping up

We've looked at the basics of using a schema registry with Apache Kafka, and what using Karapace can do to elevate your app.

Next you could sign up for an Aiven account, start an Aiven for Apache Kafka service, and enable the REST and schema registry features there as a quick way to try out Karapace yourself.

In the meantime, make sure you follow our changelog and blog RSS feeds or our LinkedIn and Twitter accounts to stay up-to-date with product and feature-related news.