Maintaining Innovation and Access to Skills for OSS Projects

At the start of the Open-Source Software (OSS) revolution, there were, by definition, fewer OSS technologies. With cloud still in its infancy, those projects could only be self-hosted by enterprises. This necessitated the need to hire engineers with skills and experience in these OSS technologies, which ensured a career of some certainty for those willing to commit to an OSS project of worth. A win-win for all. Innovation with rapid development and greater flexibility and control over how you structure your data for the business and a reliable career path for developers who could specialize and gain deep expertise in a technology, both in a cycle of perpetual improvement.

How times have changed.

Hunting for OSS skills

The advent of the cloud altered the playing field considerably. It reduced the opportunity for people to make careers out of being committers (to OSS projects) because the prevalence of activity now happens increasingly where the technology is hosted. With most of the deep technical configuration and troubleshooting pooling within the cloud providers, there is less technically interesting or challenging work for developers in the companies choosing to use these OSS projects. This poses a challenge to OSS commitment with the expertise now concentrated in cloud vendors, often referred to as the elephant factor.

No surprises then that, according to Statista, in 2022 the most in-demand open source skill was previous experience with cloud and containers, sought by 69% of hiring managers. Though the distortion in the market created by the cloud has not dampened the desire for OSS skills. An Open UK report in late 2023 found that 77% of companies are hunting for programming skills in the UK.

The winning bet?

The demand is considerable because the ecosystem is maturing. New projects are emerging daily covering everything from infrastructure to more specialized technologies, like Kubernetes and databases for AI. Indeed, every time there's a new technology there are multiple OSS versions of it. The footprint by volume of projects that a developer can choose to commit to is drastically bigger than it was 10 years ago. Yet there’s no way of knowing which one is going to be the winning bet, leading to developer uncertainty.

That breadth of options combined with the lack of guarantee has created another dilemma. Because of the increase in project volume, OSS projects are not garnering the same acceleration in innovation and development as would have historically been the case when projects were fewer. This makes purchasing and investment decisions for businesses harder and increases the risk of backing a technology or OSS project that becomes stagnant after they’ve made their bet.

Instead we are increasingly seeing employers outside the cloud vendors, looking to recruit developers with cloud development skills and experience.

Consequences of commercialization

To complicate matters further, new technologies are adopted at a more rapid pace, and boards of companies are increasingly looking to maximise the value of their investment in a company championing a specific OSS managed service, earlier in their lifecycle. Previously this was to protect themselves from cloud vendors hosting their technology and profiting from it, giving little or nothing back to the community. This practice has been most evident recently in the Graph database ecosystem with Neo4j and Tigergraph, but I expect this trend to become more prevalent in new and novel technologies as they emerge. This not only impacts community contributions but can deter engineers from long-term commitment to a project (because doing so will wed them to a technology that no other company is running, reducing their opportunities, and the impact their code can have in the world). The shift in licensing jeopardizes both innovation and a project's long-term viability while leaving a less vibrant OSS community than what was there before.

As a result, an increasing number of OSS projects such as Qdrant, Pinecone and LangChain, are adopting a permissive license to keep the movement going, to attract talent that cares deeply about OSS and to maintain levels of flexibility and choice that appeals to the companies they hope will adopt it. In all likelihood, as in the past, older technology is perfectly capable of keeping up with changing needs but because of the distortion in the skills market, there is a lack of expertise and skills to do the work of keeping the project or technology relevant, and so customers are less committed to long term use of any one technology.

It was a point made by Jim Zemlin, executive director, The Linux Foundation, speaking recently at Open Source Summit Europe 2023, who said, “Our biggest challenge is hiring smart people to come in and help facilitate all of the collective development that happens across the now 1000s of open source projects that we steward.” Indeed, according to a June 2022 report by Linux Foundation and edX, 93% of people responsible for filling OSS positions said they’re bumping into difficulty trying to find qualified people with the right OSS skills.

When faced with this combination of a pressurized skills market, OSS projects increasingly being commercialized and reticence about backing a project and losing investment as a result, how can companies protect themselves?

Open core is harmful

It’s a situation that has given rise to OSS managed services. These services offer the value associated with hosting open-source software, for instance, upgrading open source tools to address bug fixes and unlock new features. They are also a step-up from the open core model often adopted by well funded startups (where only selected and limited functionality is available under an open source license). Open core projects are not true open source, with no vibrant community, valuable features are blocked in the open core product, while the key features are paywalled in the licensed version of the offering, as we see with Redis. The primary goal for OSS managed service providers is to make configuration, monitoring, and management of complex open-source IaaS and SaaS simpler, faster, more predictable, and more reliable. These vendors naturally gain from the pooling of technology experts where the services are hosted and by employing committers to the various OSS projects, so that they can continue to develop and represent the communities needs, which their customers are obviously part of, in the upstream projects. This way we can ensure a bright future for these open source projects.

Going with an OSS managed service provider means that companies are protected from the license change because the OSS managed service companies are often stewards, participants and contributors to the OSS technology packages that they support. They are often founding members of organizations that sponsor and guide the technical direction of complex open-source projects.

Similarly, they also offer protection against the challenges with skills access and the innovation associated with it. It’s precisely why we have an open source program office (OSPO) which employs committers to projects that we host, with a clear mission, making these open source projects sustainable, so ours and our customers' business can thrive and grow without worrying whether their infrastructure open source projects will exist in the future. Doing so ensures that customers' workloads are managed by companies with the right skills and means we give our customers the best support with the most vibrant community possible.

Evidence of the OSS managed service providers acknowledging this need, is clear in the https://opensourceindex.io/. Hyperscalers are increasingly aware of the need to support and contribute more directly to the OSS projects that drive their customer success. A question remains whether smaller providers have the resources to do the same, and what the ROI of committing resources to contribute to the upstream is for them.

A fruitful future for OSS

Companies want to focus on their IP and critical business functions, leaving OSS projects and the issues associated with them to the OSS managed providers. But realizing success here means businesses must ensure that the OSSMS providers that they use have the skill sets in house to meet their objectives, otherwise failure is inevitable.

This is key to a fruitful future for OSS. One that mitigates the challenges of skills access and commercialization while providing value and innovation to enterprises and, in doing so, builds the talent pool and OSS community to keep innovation and value high for the future.

For more information about Aiven and our work in the open source community, please check out our website and our Open Source Program Office (OSPO) page.

Further reading

• Aiven statement on license changes for the Confluent Platform