Open at Intel host Katherine Druckman spoke with Matt Butcher, CEO of Fermyon, and a maintainer of the Helm project, about Helm, a nearly ubiquitous package manager for Kubernetes. Butcher discussed the philosophical and technical changes he anticipated moving from Helm 3 to Helm 4 and how you can get involved and take your seat at the big Helm table. Enjoy this transcript of their conversation.
“This is one of those moments where we know there's going to be a big change in health. And we also know there's an opening for opportunity, where if you bring in some novel ideas and are dedicated to getting a new feature in there, this is the opportunity to do it.”
— Matt Butcher, Fermyon CEO
Katherine Druckman: Hey, Matt. It’s great to have you back. The first time we talked was in Chicago, so about exactly a year ago.
Matt Butcher: Hello. Happy to be on again.
Katherine Druckman: So in addition to all of the many things you do, you are a maintainer of the Helm Project, which is, if not ubiquitous at this point, it very nearly is.
Matt Butcher: You can call it popular.
Helm's Evolution and Future
Katherine Druckman: Before we get into it, why don't you tell us a little bit more about who you are and what you do and what you're doing at KubeCon for anybody who does not yet know you?
Matt Butcher: I'm Matt Butcher. I'm the CEO of Fermyon where we're building the next wave of cloud computing with WebAssembly. Our main thing is that we believe that virtual machines have solved a whole bunch of problems, containers have solved a whole bunch of problems, but there's a small class of problems left that we need a different kind of runtime for. And WebAssembly has been that. We've really kind of leaned into serverless functions and edge functions as a way to sort of build applications that have nearly instant startup time, like half a millisecond of a cold start time, which is orders and orders of magnitude faster than a container or a virtual machine can start up. And so we're here showing off Spin 3, our newly released open source project. Spin 3 has been contributed to CNCF and I think in January it goes up for the vote. And if it makes it past the vote, it'll be a sandbox project.
Katherine Druckman: So let's talk about Helm now. There's some exciting stuff going on. So Helm, as we just said, turned five, is that right?
Matt Butcher: That was five since the last major release. I think it's coming close to 10 in terms of its whole lifetime. In the early days, everybody was kind of finding their way with Kubernetes. We knew we needed some way to manage bundles of manifests and Helm was born out of that need to do something. At the time, we called it a Debian package manager for Kubernetes. And it has been so wild to watch it evolve over the years. We've had to make some tough choices. The architectural changes from Helm 2 to Helm 3 have been disruptive to say the least. We had to make some architectural changes that impacted every single Helm user, and required a lot of migrations. That was five years ago now. Helm 3 got a lot of those things and now it’s a good utility piece of the Kubernetes ecosystem.
10 years after I started doing containers, I'm doing WebAssembly and I'm installing WebAssembly into the Kubernetes cluster with Helm charts. And that was not a thing I ever imagined doing back then. But even as powerful as Helm 3 has been, the ecosystem evolves, the tooling evolves, the code evolves, and we get to these points where we say, "All right, time to do another major release of this. We need to spend some time refactoring. We need to spend some time writing those bigger features that people have asked for that we've had to delay for so long.” So it's been really exciting here at KubeCon to kick off the Helm 4 development cycle.
Katherine Druckman: That is very exciting. So given again that Helm 3 is five years old, what are the biggest things you see changing from 3 to 4?
Philosophy Behind Helm 4
Matt Butcher: My background is in philosophy, so I'm going to start with philosophy here and then explain how we're thinking about the changes. I am by no means an East Asian philosophy expert, but I took several courses in grad school from some fantastic people who worked in that field. And I remember one of the professors said at one point articulating the perspective in Taoism and some really ancient East Asian philosophies. He says, "A core thing you have to understand about life from this worldview is we always desire stability in our lives, a home of relationships, things like that. And yet at the same time, we also always desire change. And the key is to understand that you don't have to resolve that." It’s just a fact that we have competing desires and there's no way to say that change is bad and stability is good all the time, or that stability is the right thing to pursue and change is the wrong thing.
Any time we hit a major version release like this, that kind of perspective comes to mind, because we have 70% to 80% of Kubernetes users touching Helm at one point or another. That means we can't introduce change that's going to rock the boat so much that countless thousands and thousands of developer hours are spent rewriting charts, rebuilding the CICD pipelines, or rewriting the plugins to all the platforms we use that integrate with Helm. So there's a responsibility there to maintain stability in a lot of key ways. For example, we really need to make sure that today's charts can be installed with Helm for when it's released a year from now. But at the same time, this is one of those rare opportunities where we get to make some changes and some of those changes can be very forward-looking.
Matt Farina talked earlier this week about how there's some features that have been sidelined for a long time. We’re rethinking how we're going to do plugins and lifecycle hooks inside of Helm so that developers can say, "Hey, after this template is rendered, but before the chart is installed, duck out of the main process, run this thing over here, and then provide me the updated manifest." Things like that will help developers and platform engineers express more at installation time. There's also a lot of refactoring. Like you said, software changes. And some of that software is core level Go changes in the Go programming language in which Helm is written. We have been holding off doing any updates that would break internal APIs.
Overall, there's some changes and some tidying up, some streamlining that needs to be done there. The CRD pattern was just gaining momentum when we released Helm 3. Now, we've got about five more years of experience around that. I'm interested to see people bringing some new and novel ideas into that space on how a Helm chart might better handle that. I'm looking forward to seeing if we can take another shot at cracking some of the harder problems that we've had in that space.
Katherine Druckman: I like to always define our acronyms. For anybody who doesn't know, can you tell us what CRD pattern is?
Matt Butcher: Custom resource definitions. The cool thing about Kubernetes, and what has made Kubernetes not a flash in the pan technology, but one that's suitable for long-term infrastructure management, is that the whole thing is flexible. There are all these flex points and CRDs are one of them, where Kubernetes comes with a built-in set of stuff. You have pods, you have replica sets, you have config maps and secrets, and all of those things. But the CRD toolkit is the one that allows you to create custom ones, custom types and add them to your toolkit. One of the most popular is the extensions for handling SSL certificates within your cluster. You have a type in there that represents an SSL certificate, which represents an SSL certificate authority.
These CRDs pop up all over the place as people have used them as a way to describe to Kubernetes what to do when it sees a manifest that references something. I was talking earlier about installing WebAssembly with a Helm chart. Well, how did we do that? Well, we built a new operator that sits in Kubernetes and looks for a particular kind of thing. And there's a CRD that says, "Hey, when you see a Spin app manifest come in," tells the Kubernetes control plane, "When you see a Spin application come in, parse that YAML, pass it over to this particular handler, and this handler knows what to do with it." This means the Kubernetes control plane doesn't need to be edited every single time you want to support a new feature.
Instead, it's extensible when you just define the CRD custom resource definition that says, "When you see this, this is what you do with it." Five years ago we were just seeing the early, early design patterns emerge there. Now, I think it has become a very established design pattern inside of Kubernetes. And so one of the things I think Helm can revisit is this: "Hey, Helm team, can we do this a little bit differently and make it a little more robust so that we can help people build better charts?"
Community Involvement and Contribution
Katherine Druckman: So is the development and release cycle for Helm 4 already pretty well-defined? Can we predict vaguely when at least the beta will come out?
Matt Butcher: The goal is to do the development in one year. That's the way we've done it in the past. One year gives us a good amount of time to do some of the heavy lifting, the new feature additions, but also plenty of time to say, "Are we rocking the boat too much? We're not breaking the charts, right? We're not adding in features that are going to be frustrating for people or that are going to require them to rethink things that they used to be comfortable with.” So one year should do it. The other cool thing is we have a release manager this time around, a release engineer. Matt Farina has stepped into the release engineer role. This is really important for open source projects.
His role will be to bring the wisdom and the accrued knowledge that he's garnered over the years and be able to look at these inbound features and patches and say, "From our experience, and being probably at this point the foremost expert on Helm, I have this concern about this patch or this feature." That release engineering role is going to play a very central role in keeping us on that one year timeline, but now is the time to get involved. A year may seem like a long time, but you want to get those features in there early so that people do have the opportunity to upvote it, express concerns, and whatnot.
Katherine Druckman: I appreciate that you pivoted there because I did want to talk about this. Okay, I feel like the best tools are the ones that you don't necessarily notice, that go unseen. They're seamless. You don't really think about them. We don't really spend a lot of time every day talking about why the sky is blue. It just is. We don't have to do a lot of work to keep it blue. We get up every day and if it's not too overcast, we see a blue sky. And the same kind of goes for software. The more successful and ubiquitous and seamless a tool becomes, the less people think about it and the less feedback they give. They create fewer issues, and pull requests, and feature requests, and all of those things. So I wondered, if you could, just tell us a little bit about the community, firstly. Secondly, where are those opportunities to really get involved and help keep a project going that so many people rely on?
Matt Butcher: I love the way you framed that out because I think it's very true in life in general, that some things become so commonplace that we stop thinking about where they came from and what the process was involved. This is really true of Open-source software in particular. It’s created and maintained often by a group of people who are doing it in their spare time or doing that as their job allows. On one hand, people bring different perspectives when they come to an environment like that, when they're not coming out of a monoculture (I don't mean this in a pejorative way). When you have a company, they tend to have their goals. They have their profitability in sight. They have their product line in sight. When you have an open-source community that draws from lots and lots of different organizations and from individuals in the community, you get this breadth of perspectives.
For an architectural component like Helm or Kubernetes, where you don't want to end up in one sort of vendor ecosystem, the openness of the platform and the general purposiveness of the platform is what you desire. That means that we need to bring in a lot of perspectives. Bringing in lots of people from lots of different environments, often as volunteers, means that you have more turnover than you might expect. People's jobs change, or their life circumstances change, or their interests grow and develop. In some cases they burn out, or even retire, or things like that. That means that you see transitions sometimes on a rolling basis, sometimes in a punctuated way, where a number of people drop off at one time. And the only way an open source project can continue forward at that point is to grow the community base, to develop the contributor base.
Helm has gone through a couple of rollovers. Many of the people who worked on it very early on have since done all kinds of different things. Many of the people who were enthusiastic fans for a long time are now maintainers and core maintainers, but this is one of those moments where we know there's going to be a big change in Helm. And we also know there's an opening for opportunity. If you bring in some novel ideas and are dedicated to getting a new feature in there, this is the opportunity to do it. This particular moment in time, when we’ve announced things, is the ideal time for you to jump in and get involved in this project. CNCF has done a fantastic job and Helm has participated in this effort of making it easier for newcomers to figure out the mechanics of a project and begin to get involved.
We have probably thousands at this point, maybe tens of thousands of contributors to Helm over the years. But in this moment, you can also play a bigger role in being able to influence the architecture. And if our development cycle is a major release every five years, there's an amount of responsibility and an opportunity that comes right now, at this moment. It won't come again for five more years.
Katherine Druckman: That's a good point.
Matt Butcher: This is the time. We're really excited about the way things have shaped up. Again, having Matt Farina driving this process just instills a ton of confidence in me that the project will remain stable. And, again, that stable and novel trade-off here. I think he's the right person to do that, but it would be fantastic to see some new engineers jump in here and say, "Yeah, I would love to be part of this next five-year chunk of Helm's history, or even this next one-year chunk of Helm's history."
Encouraging New Contributors
Katherine Druckman: I always tell people: if you rely on a thing, whatever that software project or whatever it is, you want a seat at that table. Because you want the influence, but take the seat. It's a cool table to be a part of, right?
Matt Butcher: Yeah, and with these, as Kubernetes itself ages, we now know that it's the de facto way that people are building large-scale infrastructure and medium-scale infrastructure. It's not going away anytime soon. But at the same time, people are changing. Brian Grant retired last year. Back when I started, Brian was leading the Kubernetes project very actively and he did such a great job of transitioning over time and bringing in other people. I think that's an illustration of how important it is to have a core architectural piece, to grow the community the right way, and ensure people understand that they can play a big role in contributing. In many cases they ask their employers, "Hey, can I spend a chunk of my work time on this?". Those opportunities translate into a long-term rolling impact that keeps Kubernetes the best platform to run your containers on. That keeps Helm the easiest way to install things into your Kubernetes cluster.
In software, we tend to glorify the novelty piece of things, the stability versus novelty, and say, "Well, there's a new greatest thing." But again, like your point earlier, we forget about the foundational pieces upon which we're building those. And if we don't do our diligence to keep those things going, eventually we'll hit a wall. We'll hit stagnation. And that is actually kind of the thing we want to avoid within the CNCF ecosystem. We want to avoid running up against stagnation because we can't keep the momentum high. So I think I'm doing carrot and stick here. The carrot is as follows: this is the time to get involved because you can bring your ideas into Helm. You have this opportunity to influence over five years. The stick part is that if we don't keep the momentum up, then a very valuable piece of the ecosystem could go stagnant.
And then we end up struggling to deploy things the way we want, struggling to make do without the features that would be really beneficial, struggling to keep up with the changes in the Go language and the broader Kubernetes ecosystem. So there's a carrot and a stick here.
Katherine Druckman: So all these people that are now going to be inspired to pitch in, how do they do that? What's the best way to get started as a new contributor to Helm?
Matt Butcher: We've got some great guides out there. Probably the easiest way is to just drop into the Slack channel in the CNCF Slack and chat a little bit with people. For example, "Hey, I've got a real interest in being involved in this." Another good way is to open up issues. There's a whole process documented in the Helm/governance repository. Maybe it's the Helm repository about suggesting new features to go into Helm 4. There's also a weekly developer call on Thursdays in US time. That is a great opportunity to show up, hear what’s happening, and express your level of interest, whatever that may be. And that Thursday call is really devoted to talking about the development issue.
That is a great way to get involved there. But really, I just feel like Slack always ends up being the right first step because you start to observe the community interacting sort of asynchronously. And you can ee what's going on there and weigh out how you might choose to become involved.
There’s something else that I think is absolutely important, and I think we in CNCF do it better than the vast majority of communities. We talk about feature development. We talk about making code changes, but documentation, tutorials, content that helps lead people in the right way. All of those skills are invaluable for a project that has so much utility to the environment. If we can't help people understand how to use Helm 3, probably the most important piece of content one year from now is going to be the Helm 3 to 4 transition guide.
Katherine Druckman: Yes, it will.
Matt Butcher: There's a need for all kinds of skill sets there, from product management and documentation to low-level engineering and Go programming.
Katherine Druckman: So did we leave any stones unturned? What else do people need to know about Helm and the project and its community?
Matt Butcher: It’s a really friendly community, especially as you get close to the core maintainers. It's a group that is very much welcoming because we've been around long enough that we understand that change and transition is going to be part of the process and that we desire stability. We always bring those two things back together. To me, it's personally very meaningful that in addition to building a piece of software that is useful, we've made a concerted effort over eight, nine years to also build a developer community around this that is just a friendly and welcoming place. So I know these things can be scary and we've all had our experiences with communities that are somewhat standoffish. Helm's not one of those.
Katherine Druckman: Fabulous. Well, thank you so much. I'm a Helm fan. I was a Helm user. I've benefited from the project many, many times, so I would love to see more and more people get involved and help keep it as successful as it currently is.
Katherine Druckman: You've been listening to Open at Intel. Be sure to check out more about Intel’s work in the open source community at Open.Intel, on X, or on LinkedIn. We hope you join us again next time to geek out about open source.
About the Guest
Matt Butcher, Co-Founder and CEO, Fermyon
Matt Butcher is co-founder and CEO of Fermyon, the serverless WebAssembly in the cloud company. He is one of the original creators of Helm, Brigade, CNAB, OAM, Glide, and Krustlet. He has written or co-written many books, including Learning Helm and Go in Practice. He is a co-creator of the Illustrated Children’s Guide to Kubernetes series. These days, he works mostly on WebAssembly projects such as Spin, Fermyon Cloud, and Bartholomew. He holds a Ph.D. in Philosophy. He lives in Colorado, where he drinks lots of coffee.
About the Host
Katherine Druckman, Open Source Security Evangelist, Intel
Katherine Druckman, an Intel open source security evangelist, hosts the podcasts Open at Intel, Reality 2.0, and FLOSS Weekly. A security and privacy advocate, software engineer, and former digital director of Linux Journal, she's a long-time champion of open source and open standards. She is a software engineer and content creator with over a decade of experience in engineering, content strategy, product management, user experience, and technology evangelism. Find her on LinkedIn.