Drive Apache Kafka^® like a boss with Aiven and Conduktor

Evaluating Apache Kafka^®?

Apache Kafka is incredibly popular and its power is proven. According to the Apache Software Foundation, more than 80% of all Fortune 100 companies trust Kafka to operate their production workloads.

But the truth is that Apache Kafka® needs a major investment of time and energy to operationalize, especially at scale.

The good news is your investment in Apache Kafka can be significantly de-risked. In this post, we introduce you to Aiven and the Aiven for Apache Kafka cloud service, and demonstrate how it works with Conduktor’s all-in-one Kafka management platform.

With Aiven you can deploy Apache Kafka on a secure, fully managed platform, while Conduktor makes you 10x more productive with Apache Kafka. It’s a seriously easy win-win solution.

Aiven for Apache Kafka works for you

Aiven takes 100% open source database and streaming projects and provides them as fully-managed cloud services for customers all over the world, across all major cloud providers. The Apache Kafka code you’re running on your localhost or your cloud provider of choice is the same code Aiven installs when provisioning your Kafka clusters.

Not only do we use 100% open source Kafka, but we also contribute back to the open source community. For example, when Confluent changed their open source license for the Schema Registry, Aiven developed a drop-in replacement for the Schema Registry called Karapace and shared it with the open source community.

Aiven has your back with 99.99% uptime

At a minimum, Aiven provisions Apache Kafka in a 3 node cluster, distributing the nodes evenly across the availability zones of the cloud region you select. But you can scale to 30+ nodes, and Aiven’s solutions architects can build custom plans to meet specific requirements.

Aiven takes care of the VM provisioning, setup, security, and the day-to-day operations of keeping your Kafka cluster running at a 99.99% SLA. And over the past rolling 36 months, we’ve observed closer to 99.997% uptime.

We rebalance clusters transparently for our customers, in a self-healing manner.

Four years of Apache Kafka

Aiven has been running Apache Kafka as a fully-managed cloud service for 4 years. That’s longer than any other provider of Apache Kafka as a service.

In those four years, the Apache Kafka ecosystem has also grown significantly, and is one of the main reasons why so many developers choose Apache Kafka today.

While many of the tools in Apache Kafka’s orbit are either sources of data or sinks for data, there are also tools built to make working with Apache Kafka way, way easier – like Conduktor.

Meet Conduktor

A new member of the Kafka ecosystem, Conduktor is leading the charge when it comes to simplifying the Kafka journeys of companies around the world, while also educating the broader Apache Kafka community. Check out their Conduktor Kafkademy to start leveling up!
With more than 60,000 users across 9,000 organizations, Conduktor provides a powerful GUI working with Apache Kafka. From cluster monitoring to data awareness and exploration, Conduktor makes it easier than ever to harness the power of Apache Kafka.
While Conduktor may look and feel shiny and new, their all-in-one friendly interface to work with the Apache Kafka ecosystem has actually been in development as a free-to-use product since 2016. In about that same timeframe, Aiven honed our craft delivering fully-managed, secure, and performant Apache Kafka, Kafka Connect, and MirrorMaker as cloud agnostic services that are ridiculously easy to get started with.

Conduktor gets you 10x more productive with your data

Conduktor was built by Apache Kafka experts, and is an elegant solution for a more civilized use of Apache Kafka. If you interact with Apache Kafka, Conduktor will make your life better.

• Developers - with Conduktor you can configure topics, find data, manage consumers, Kafka Streams applications, and manage Connectors.
• Operators - Conduktor makes it easier to configure and monitor your brokers, reassign topic partitions, and ensure the metrics are good.
• QA - using Conduktor you can run scenarios by sending custom messages to run functional tests or test resiliency by sending random data.

For enterprise scale use of Apache Kafka, Conduktor standardizes the use of Apache Kafka to make everyone more informed and productive across the organization.

Storyblocks, the innovative stock footage platform, uses Conduktor’s desktop GUI to simplify and streamline Kafka operations. “Conduktor is amazing,” says Ilya Berdichevsky, Lead Data Engineer at Storyblocks. “It’s one of those rare 10x tools. It cleverly hides Kafka powers under a thoughtful UI.”

Connecting Conduktor to Aiven for Apache Kafka

Let’s take a look at how these two tools work together. We’ll use Aiven for Apache Kafka to take care of management, and Conduktor to make working with Apache Kafka and our data 10x easier.

Getting started with Conduktor is pretty simple and there is built-in support for setting up the connection. All you need is a CA certificate for each Aiven project in Conduktor.

Add CA certificates for Aiven projects in Conduktor

Go to the Service Overview page for your Aiven for Apache Kafka® service and click the service name (“kafka-conduktor” in the image below).

Download the Access Key, Access Certificate, and CA Certificate to your computer.

Note down the Host and Port information. You’ll need them soon.

Add your Aiven cluster in Conduktor

1. In the Conduktor desktop application, click New Kafka Cluster in the main pane.

1. Click the Aiven icon.

1. In the Add new Cluster screen, click the Aiven integration button.

1. Add here the information from the Aiven Console that you noted down earlier, add the two files you downloaded, and specify a folder for Conduktor to create Keystore and Truststore files in.
2. Click Create.
   Conduktor creates a configuration to connect to your Aiven for Apache Kafka service.

1. Click Save.

Congrats, you have Conduktor connected to your Aiven for Apache Kafka cluster!

Now you can visualize all of your data within Apache Kafka. Take a look:

Next up

For this introductory blog post, we did not introduce demo data to further explore Conduktor.

In our next blog post of this series, we’ll explore all of the cool things you can do with Conduktor on Aiven for Apache Kafka using demo data.

Remember, Aiven for Apache Kafka is the same 100% open source code you’re running yourself, but we make it our entire business (literally) to do that for you, so you can focus on building things that matter and delight your customers, rather than the care and feeding of Kafka!

--

To get the latest news about Aiven and our services, plus a bit of extra around all things open source, subscribe to our monthly newsletter! Daily news about Aiven are available on our LinkedIn and Twitter feeds.