What is 'Current' with data streaming?

The Aiven team was at the Current 2022 event last week joining many folks, speakers and the event streaming community in Austin. The event took place on October 4-5, 2022 and had the ambitious goal of expanding the grounds of the well-known and much-loved Kafka Summit. Current 2022 broadened the scope of the former Kafka Summit event to include a plethora of event streaming technologies and frameworks, including but not limited to real-time data processing, streaming storage, streaming databases and many other technological domains.

Attendees who worried that Apache Kafka® would not receive enough attention in the conference agenda during Current 2022 were awestruck to see that the program committee managed to include 50% more Apache Kafka content this year! Nonetheless, the audience at Current 2022 was offered a bigger and wider range of sessions on a variety of topics such as real-time analytics, data governance and streaming. The session quality was so high this year that the only challenge was how to be in multiple locations at the same time. On our behalf, we would like to congratulate and thank the conference organizers and the entire program committee for giving us two days of amazing content!

Those who prioritized networking could fully enjoy that as well. The event was packed with leaders and professionals from different industries across the whole spectrum of data technologies. Special attention was given to diversity and inclusion, which helped Current 2022 bring different people together to learn, share their experiences and expertise, and network in a supportive and encouraging environment.

In the following paragraphs, we'll do our best to transfer the atmosphere of the event and share some of our key observations from both the keynote and breakout sessions across both days of the conference.

Keynotes: Apache Kafka^® and Streaming Era

Current 2022 featured inspiring keynotes, focusing on real-time data streaming, its history, evolution, and the role that Apache Kafka® has played over the years, as well as giving us a glimpse of what’s in store for Kafka in the future.

Jay Kreps, CEO of Confluent, opened the first keynote of the conference talking about the key turning point in handling data and witnessing the rise of a major data platform. This data platform, like a central nervous system across any organization, will help to connect different outposts into a coherent environment that stores, connects, processes, governs and shares the data. The emergence of this streaming data platform unites operational databases, SaaS applications and analytical systems into a single ecosystem that drives innovation and unlocks an even broader set of use cases.

To illustrate this Glan Merlino, co-founder and CTO of Imply, and Anush Kumar, VP of Technology at Expedia, took the stage and shared their stories of how data streaming influences their products and business. Streaming data is all about constant change, which reflects life in a more authentic way.

The second keynote of the day was presented by Jun Rao, co-founder of Confluent, and covered the journey of Apache Kafka® through time; its history and the motivation behind its creation, how it evolved and what is the path for Apache Kafka® in the future.

In particular, Jun focused on the improvements that were made to Apache Kafka® and its wider ecosystem over the past year. Many of those focused on performance, security and reliability. More connectors, client libraries and a wealth of stream processing engines were introduced.

One of the notable changes that was implemented recently in Kafka was the replacement of Apache Zookeeper® as Kafka’s control panel with KRaft. Ismael Juma shared the story of KRaft, a new control panel that helps Kafka scale better and run with millions of partitions. The journey to build KRaft started 3 years ago, and finally with Kafka 3.3 it is production ready.

Jun was later joined on the stage by Pritha Mehra, Chief Information Officer at United States Postal Service and shared how Apache Kafka® enabled the USPS, using an event streaming architecture, to administer millions of COVID-19 test kits throughout the United States when the pandemic hit back in 2020.

Common Topics emerging from Current 2022

While the sessions at Current 2022 covered multiple themes this year, we were able to identify some common topics among sessions, panel discussions and community conversations. Some of them were:

Data Governance is seen as paramount for event streaming

As the adoption of event streaming grows across organizations (and also within an organization), so does the complexity of managing and operating Kafka in a secure and compliant way. With more Kafka clusters and topics created by multiple teams and systems producing and consuming data from these topics, having a unified and centralized way to ensure that the right teams have access to the right streaming data oftentimes becomes a bottleneck for operations teams.

At Current 2022 we saw the theme popping up again and again in discussions and sessions. Specifically, Vanessa Burckard from the Social Security Administration discussed their approach of Data Governance as a Service for their internal teams, and how this helps streamline Kafka adoption for new use cases for the organization. Other discussions focused around how you can leverage a cloud service’s integrated security and governance controls to set up appropriate governance tooling for your Kafka environments.

From Aiven’s side, we were happy to see an increased demand for data governance tooling around Kafka, as this is directly in line with our recent acquisition of Klaw (formerly Kafkawise) and our intention of building an open source community around it to ensure the appropriate governance between teams managing and building Kafka workloads. You can find more information about Klaw here.

Batch vs. Streaming: Is Batch going away? And what does the future hold for both?

Throughout the conference, the theme of Batch vs. Streaming was apparent. Discussions covered how they can be unified, how batch processing’s performance can / must be improved for real-time applications, and more. There was even a dedicated panel discussion with Adi Polak, Amy Chen, Eric Sammer and Tyler Akidau discussing the state of streaming adoption today, and debating if streaming will ever fully replace batch. You can view some interesting points from the panel discussion in the Twitter thread from Robin Moffatt.

Getting performance right is not easy

Performance is always a hot topic in any tech conference, and Current 2022 was no exception. Several talks followed this theme, like Gunnar Morling’s "Keep the Cache always fresh" talking about CDC and caching mechanism; or Olena Kutsenko’s "Beginners Guide to Balance Your Data Across Apache Kafka Partitions" addressing the correct usage of partitioning.

The topic of performance was also discussed in Ben Gamble’s "Going Multiplayer with Apache Kafka" detailing how Kafka and stream processing can be used to power real-time multiplayer and multiuser interactions. The set of examples included a live chat, concurrent document editing, and even Minecraft. Ben was not only showing cool demos, but also reviewing and discussing some of the streaming data topologies required to support them.

On a similar topic, Francesco Tisiot's “Breathe in and breathe out: get Kafka Connect configs right” defined the struggles of starting with Kafka Connect and shared some magic tips to start a connector and keep it running properly. From preparing the field by creating the needed resources, passing on through validating the requirements and the network path, and finishing up with version control and automatic deployment, Francesco shared a lot of information that can make the Kafka Connect experience smoother and more reliable.

Diversity and inclusion

Special place at Current 2022 was given to the support of members from underrepresented groups with a goal to build a diverse and inclusive community within data streaming.

In particular, at the panel #IamRemarkable, which was organized by Google, the panelists talked about how cultural and gender norms may prevent us from talking about our accomplishments and sharing our achievements. The leaders shared advice on how to overcome such challenges to improve self-promotion.

In addition, the 'Intersectional Happy Hour' that took place on the same day gave attendees from underrepresented groups and allies the opportunity to meet each other, network and chat in an informal setting. Those who didn’t have a chance to come to Intersectional Happy Hour joined the ‘Intersectional Speed Networking Hour’ on the second day of the conference.

Conclusion

At Current 2022 one thing became clear: Apache Kafka® is growing at an unprecedented pace as the adoption of streaming data becomes the de facto paradigm in building and maintaining modern data systems across organizations of any size and industry. And none of this would be possible without the Apache Kafka® community at its core. All those who share their time and energy to both contributing code to Kafka, and improving its community experience by sharing their expertise, are ensuring that Kafka continues to thrive as an ecosystem.

We at Aiven are beyond excited to be part of such a vibrant and inclusive community and cannot wait to meet them at a future conference about event streaming soon!

To get the latest news about Aiven and our services, plus a bit of extra around all things open source, subscribe to our monthly newsletter! Daily news about Aiven is available on our LinkedIn and Twitter feeds.