In this episode of the Open at Intel podcast hosted by Intel Open Source Security Evangelist Katherine Druckman, Katherine chats with Madelyn Olson, a maintainer of the Valkey project and an AWS engineer, about the life of an open source maintainer and the experiences surrounding the launch of the Valkey project. They cover the pivotal moments that led to the creation of Valkey, a Redis fork, following the Redis license change. Madelyn also shares insights on the challenges and pressures of being a maintainer, strategies to manage burnout, and the significance of creating a community-driven, open source project. The episode highlights the technical advancements and future directions for Valkey, working to leverage modern hardware, manage large clusters, and expand the extension ecosystem.
“We have a lot of different features that are coming from a lot of different individuals, and it's really helped us have a community-driven project with a community-driven roadmap where I want it to feel like anyone can contribute. I want it to feel like anyone coming into the project feels like their opinions are heard, they can get features in, and they're not just shut down because one group of people doesn't agree with them. I think it's part of the reason a lot of people have resonated with the Valkey message. They see Valkey as a real open source community general response to the licensing change.”— Madelyn Olson
Listen to the interview here: https://openatintel.podbean.com/e/when-you-have-to-fork-a-project-all-about-valkey/
Katherine Druckman: Madelyn Olson, who represents the project Valkey, and works for AWS, is joining me today. I'm really glad to have you, Madelyn, thank you.
Madelyn Olson: Yeah, thank you, Katherine, so much for inviting me on the show.
Katherine Druckman: Just as a little background, we met briefly at Open Source Summit in Seattle, and back then I was having this conversation with quite a few people about how the life of an open source maintainer, especially one on a very widely used product, is challenging, and I wanted to talk about that. But at the same time, we have another big topic, which is frankly, Valkey, the project itself.
A lot of people listening will remember that in March the Redis license changed and suddenly people had to pivot, as you do when licenses change. And pretty quickly Valkey was launched as a fork. I wondered if you could just tell us about your experiences with that.
Redis License Change and the Birth of Valkey
Madelyn Olson: Yeah, for sure. It happened during the middle of KubeCon. I know a bunch of people mentioned how it ruined their KubeCon. I was not there, but I was having a lot of conversations with people at midnight their time trying to figure everything out. I was previously maintainer of the open source Redis project and had been a maintainer for multiple years. I was a long-time part of the Redis open source community. I joined back in 2018. I was the one that originally built TLS, built the original proof of concept, built a lot of the work for Redis, and helped merge the final version into Redis 6.
Some people might know there was previously a single person running the project. He stepped down in 2020 and I was part of the five-person team that took over maintaining the open source Redis project in 2020. I had been a maintainer of Redis for about four years, and that went all the way up until March when the license changed and the Redis folks, the ones who owned the Redis trademark, decided that they were also going to dissolve the governing board of the open source project.
That was my incentive to be like, "Hey, what's next?" I was no longer a part of the Redis project, and so I got together with five other folks from the Redis community. I mentioned before there were five maintainers of Redis, three of them worked for Redis. There was me, who worked for AWS, and then another guy named Zhao, who worked for Alibaba.
Zhao and I got together, and we got four other really active community members from Redis, and we went and started the Valkey project under the Linux Foundation. The Linux Foundation was really the folks that helped us launch the project. They're also the ones that own the trademark and IP to make sure the project stays open source for the long term. The Linux Foundation really helps us solve that problem to make sure that something like this fork doesn't happen again.
Valkey cannot do the same licensing change that happened with Redis. That's really what happened with the whole Valkey project at the time. That was way back in April. So that's when we were meeting. It had just happened, we had just launched, we had just launched 7.2, and so just to give an update on the project, we've been hard at work the last couple of months. We're almost ready to launch a new version, Valkey 8. I was really hoping we'd have it done last week, but it's still not done. Right before we were recording this, I was reviewing a bunch of code that needs to get in, so we're really close. Probably by the time this podcast is out, Valkey 8 will have its first release standard, so you should go download and try it.
Katherine Druckman: Awesome. I've seen some commentary about how successful the project has been in terms of contribution. I wondered if you could speak to that a little bit.
Madelyn Olson: Yeah. The project has been more successful than I could ever hope. You see a lot of these forking situations happen and the forks, they often do fairly well. But I think Valkey has really done better than anyone expected. As you said, there's been a lot of reporting about how there's a lot of contributions, there's a lot of major contributions. I don't know all the numbers off the top of my head because I don't pay too much attention to all that stuff.
Katherine Druckman: Sure, sure, and...
Madelyn Olson: And really just focusing on...
Katherine Druckman: ... ultimately project health, period, right?
Madelyn Olson: ... project health. But yeah, the project's been super helpful. There have been so many people who have come in. We've heard so many great anecdotes from folks online talking about how easy it was to migrate, and not just the community, but also all the corporate entities around the project coming up have been really exciting as well. We have a lot of corporate sponsors like Aiven who launched a version, a managed version of Valkey not too long ago. There's NetApp, DigitalOcean, there's some non-managed providers, like Ericsson and Verizon, have been big contributors.
It's been really exciting to see all of the community show up. Percona has been another big supporter of us. Canonical, AlmaLinux is now a supporter. It was easy because when we first launched it back then at the Open Source Summit, I had the whole list of corporate sponsors I could just rattle off, but now the number is up to 25 different companies. It's hard to remember everyone.
Katherine Druckman: Wow. That's just a great problem to have.
Madelyn Olson: But yeah, just everyone coalescing on this project. Yeah, it's a great problem to have. I agree.
Maintainer Life and Burnout
Katherine Druckman: I do want to talk more about the experience of forking and when you might want to do that, but I also wanted to talk to you about your experience as a maintainer. The maintainer life, as I mentioned earlier, it's a tough one, especially when you have the responsibility of the role of a maintainer on such a successful project. The more users you have, the more pressure… for me anyway.
I mean, I've been a maintainer on smaller projects, but to me that's a lot of pressure put on a human being. Our life is complicated and we all have our pressures, but I wondered if you could tell us a little bit more about your experience, and especially in a moment like that when there is, I think it's safe to call it, a bit of a crisis within a project. Something changes, whether it's good or bad or whatever, but it becomes all hands on deck. You have to solve some sort of problem or address it very quickly. What is that like for you?
Madelyn Olson: It took me a long time to really figure that out. As I said, I was involved in Redis for almost six years, and for the first couple, I wasn't a maintainer. I just worked on it when it was convenient, and it was really easy for me to, say, work on a feature, hand it off, be like, "Hey, here's the code?" and then disappear for a while and come back.
But once I got upgraded to being a maintainer, there was always something more to do on the project and it was very hard for me for a long time to limit it to being like, "Hey, I'm spending 20 hours a week on the project and no more." Because, as I said, there's always something more to do, there was always more pull requests to review, there's always more issues to look into, there's always more bugs to deep dive. And one of the things about Redis is Redis is wildly successful. If you go look at the GitHub today, there's still hundreds of outstanding issues because there's just so much engagement, and there was just a never-ending list of stuff to do.
I got to the point where I was working on weekends, I was working all the time on this project, and it was definitely very bad for me. I really felt a lot of burnout. Also, I got added to the project in July of 2020, which was basically the start of the pandemic. It was at the time where I had lots of free time and an endless number of things to do. I pretty quickly burnt out right away when I worked on the project. And that's when I got to the point where I was like, "Hey, I can only spend so much time on this project. I can't spend all my time." For a long time it was kind of like 10 to 20 hours a week. This is how much time I spent on the project and I had to disengage when I'd worked that amount of time.
That worked more or less okay until Valkey came along. And now, all of a sudden, not only is there an infinite amount of stuff to do, there's also some amount of time pressure because now we're like, "Hey, we got to make sure this project works." Every bug is a critical bug, if it's blocking adoption, if it's blocking... if someone wants to move, but they can't, getting those features fixed ASAP is important.
And then, as well, when we switched in Redis, there was a bunch of other super experienced maintainers, and a lot of them stayed with Redis. When we moved to Valkey, me and Zhao were the two people who had all of the institutional knowledge. And so a lot of times stuff was getting blocked on me because he and I were the only ones that knew how to do it. Zhao works for Alibaba…he lives in Mainland China...there's a big time lag. If someone's blocked on me in the U.S. time zone, I felt like I had to be available 24/7 to help unblock a lot of that stuff.
I think I worked...you can even go check GitHub…I probably committed to the project every single day for almost two and a half months. And that, again, is obviously very unsustainable. I got very burnt out. There were definitely days that just sat and did absolutely nothing, and it's not sustainable, because this is one of those projects that's at the core of a lot of applications. And so, you can, but you really don't want, to be making mistakes within code and writing code. This stuff is really, really essential. It's really important to take a step back, make sure you're reviewing stuff thoroughly.
I wasn't able to resume that 10 to 20 hour limit, but instead it's just like if I don't think I'm doing a good job, I have to take a step back and just say this can wait.
I know we talked a little bit before this conversation. I'm currently reviewing code for Valkey 8, the new release, and I was hoping that Valkey 8 would be out last week. But part of the reason it's not out is like we weren't able to do the reviews at the quality that we all want them to be at. It's better to wait a little bit longer. We picked a date for Valkey 8 and we're not quite hitting it, but it's more important to make sure we're all doing a good job and maintaining our mental health than hit some arbitrary dates.
I'm hopeful that once this version goes out, I'll feel a little bit better. I'll make sure the project feels a little bit more established. But it's really hard. I don't have a great answer for this, the crisis moment. The crisis moment is just always going to burn people out.
Katherine Druckman: Yeah, yeah. I wondered, having been through this, do you have advice for people? And the two things that come to mind are, one, recognizing when you need to do something about the burnout. Recognizing it, I feel like that's challenging in and of itself. And then, two, the hardest one I guess is then, what do you do about it?
Madelyn Olson: You're definitely right that identifying that you need to take a break and disengage is really important. And for me, the biggest thing has always just been making sure if I'm not doing the quality of work that I want, that's the time for me to go and take a step back. And there may be something similar for what other people see. For me, it's really easy. I live in the Pacific Northwest. I live in Seattle. I just go hiking. I just go and disappear from the world.
Katherine Druckman: That must be lovely.
Madelyn Olson: It's lovely.
Katherine Druckman: I'm a little jealous.
Madelyn Olson: Yeah, just getting away from technology, getting away from the stress of work, getting back in nature is something that I value a lot. Other people spend time with friends, spend time with family, just really things that aren't stressful for the most part, things that don't... not necessarily productive things.
Katherine Druckman: Spend time with your dog, that's what I would do.
Madelyn Olson: Oh, yeah, sure.
Katherine Druckman: Pet a floof, it's very soothing.
Madelyn Olson: And there's something else that I think is important. Burnout often comes not just from doing too much work, but not feeling momentum on the work. It's a little bit easier to keep pushing really hard to get that thing done. I think it's hard to recognize burnout when things are going really well because you're making a lot of progress, things are happening, but there's still, you know, mental fatigue and it becomes harder to keep doing stuff. I found that it was harder, at least with Valkey to make sure I was taking the needed breaks, because there was a lot of code getting written and we were making a lot of progress, but still it was taking a significant toll on me, and I just wasn't really acknowledging it because we were making so much progress.
Katherine Druckman: Yeah, I can see that. When you're in the moment, especially, it's got to be very energizing when you see that kind of success and momentum and whatnot. I would think it would be easier to double down on the things that are leading to that kind of burnout situation.
Madelyn Olson: Right. Especially with Twitter. And I've not been a super active social media person, but I got onto Mastodon, Twitter, and I follow LinkedIn a lot, and I just see all these people being excited about the project and that is really energizing and that's helped push a lot of the burnout out. It's really just been fatigue and tiredness of just working too much that's been getting at me, not the lack of motivation and momentum that normally...especially with…now that we have a community-driven project, whereas back in Redis it was still predominantly controlled by Redis limited employees. If they didn't want something, they could veto stuff and that was pretty demoralizing, and that caused quite a bit of burnout for me. But now that we're completely vendor neutral—we have six different maintainers from six different companies—we're able to make a lot more progress. At least it feels like it's much easier to make progress, because there's no one person that can say no. That also ends up being very energizing.
Forking a Repository: When and Why
Katherine Druckman: Pivoting a little bit back to the topic of just the experience of forking a repo, right? This scenario... I mean, this seems like an obvious one…there's a license change, there's a very obvious situation where somebody's made the decision for you, you're backed into a corner, you have to fork, right? But I feel like there are a lot of other scenarios, and I wondered if you have any thoughts on that.
For example, when you’re consuming a project and you maybe have slightly different needs; you have different security considerations maybe; you have features that you're not able to get merged or upstreamed; or the diplomacy that goes into these kind of things isn't working. How do you know when you have to have that conversation about forking?
Madelyn Olson: Yeah. I think there's definitely a lot of the spectrum in how much you fork it. I'll talk a little bit from my experience from AWS about this. For a long time, we also had an internal fork of Redis because we had slightly different requirements, and there was always this push and pull between how much code do we want to keep forked just for our private stuff, because you don't necessarily have to fork something and make it open source. You can fork it and change the license. You can fork it and keep it proprietary. And that really just comes down to how much code do you need to change, and how generalizable are your requirements.
The thing that I got really engaged with at AWS, was saying, "Hey, a lot of this code that we're writing we have to maintain, and it's not really doing anything for us." That pushed us more towards contributing to upstream. And one concern that we had a lot was, well, what if we can't get to upstream? Well, maybe we'll create a fork and create a set of patches. There are some projects that are literally just a set of patches on top of another project.
But what we really found most beneficial was instead of trying to fork the whole project, we just forked part of the project. One innovation that the Redis project made was introducing the concept of extensions called “modules.” Modules allowed you to add features into the core project without having to fully fork. I know that's another layer and not a complete fork, but it allows you to do what you want to do a little bit differently without having to maintain a lot of extra code.
I think all of those are the levels of, maybe you can still get some coding in upstream, but you can't get everything you want. But at some point, it might become necessary. If they change the license on the code, it is a huge consideration of you should probably fork, or if they're trying to take the project in a completely different direction.
I would also always be a little bit skeptical. That happens the most when there's a single vendor controlling the project. There's a difference between an open source project, one with a permissive license, and one that's community-driven, something that's backed by the Apache foundation, for example, that makes sure the entire community's input is considered.
I don't think I've seen a lot of forks where the main project isn't going the direction you want. If the main project is open source, you don't see a bunch of high-profile forks. Most of the high-profile forks have been around licensing, but you see a lot of much smaller ones where someone wants to take something a little bit of a different direction and they keep merging from upstream. I think that's mostly what I've seen.
Katherine Druckman: Yeah. I feel like 100 years ago I saw some in web CMS type projects. There was a lot of forking that happened a long time ago, but recently, like you say, it's generally licensing. Now I think there is a security conversation to be had, especially when you're, again, maintaining an internal fork or something like that, that's interesting.
Community-Driven Open Source Projects
Madelyn Olson: There's an interesting story about security, which is that there was a pretty high-profile fork of Redis that had an SSL at one point because Redis was very anti adding SSL for a long time because it had a lot of complexity. I think one of the cool things about that fork is not that it went anywhere, but that it really pushed the main project to care about security. That helped me get to the point of being like, "Hey, other people are starting to fork because this isn't happening.” It's easier to say, “Hey, now's the time. Let's go actually build it."
Katherine Druckman: Yeah, that community peer pressure is something, right? Not a bad thing in this case. Speaking of community, we talked a little bit before and you hinted at it, but we talked about having open source code and having an open community are distinct things, and it's a different kind of set of values. I wondered if you could elaborate on your feelings there.
Madelyn Olson: I talked about how back in the Redis days it was an open source project, but at no point in time was there really ever a community-driven effort to push features. There was this core team and it was controlled predominantly by a single vendor. They controlled what went in and went out of the project. I think that made it a bit technologically conservative in what they would accept because if it wasn't aligned with their vision, it didn't get in, and there was no one really challenging the status quo.
We saw the things where it really took forks to convince the project to be like, "Hey, these things need to get done." And that was one of the big things I wanted to change with Valkey. When we moved over to Valkey, it was a very important to me that the initial group of maintainers, which we call the technical steering committee, or the TSC, were all members of different companies so that we had a very rich set of different opinions that went into the decision-making process.
And I think that's made it very successful, at least this upcoming version we have. We have a lot of different features that are coming from a lot of different individuals, and it's really helped us have a community-driven project with a community-driven roadmap where I want it to feel like anyone can contribute. I want it to feel like anyone coming into the project feels like their opinions are heard, they can get features in, and they're not just shut down because one group of people doesn't agree with them. I think it's part of the reason a lot of people have resonated with the Valkey message. They see Valkey as a real open source community general response to the licensing change.
The Future of Valkey
Katherine Druckman: Yeah. And getting back to Valkey, what are you most excited about? If we talk again in a year, what do you hope to be able to tell me?
Madelyn Olson: Valkey has great fundamentals for being a good caching project. There are three big areas that I want to see us get better in. The first is I feel like a lot of the technology that we use internally is built for processing and hardware from a decade ago. I really want us to take better advantage of the hardware we have. That's really about performance.
We have a bunch of ideas. Some of them basically saves a bunch of memory, makes it more performant We are having a bunch of a multi-threading improvements so that we can better use more cores. I'm really interested in that.
The second is, when Redis was originally created, we were in a pre-Kubernetes world. There wasn't much self-managing going on that wasn't literally installing processes on VMs. I want us to get better at running large clusters of Valkey processes.
There's this big project we have called Cluster V2, which we're slowly making progress on and should hopefully be around for Valkey 9, which will come out next year, to make it easier to manage big clusters, because that's the better way to scale the cache.
And then the third thing is I want to make the extension ecosystem way more vibrant. I think there's a lot of cool ways to add new data types and add new access patterns. I really look to Postgres as a role model here. The Postgres ecosystem is really robust. It has a large number of different extensions that people have built over time that really extends what Postgres really means. And I'd like to see Valkey do the same thing with its extensions to make it more ubiquitous in-memory data platform.
Those are the three things that I'm really looking forward to, that I hope we can make a lot of progress towards in the next year. We have some individual ideas for a lot of these, but I'm hopeful that it'll be a lot more fleshed out in the next year.
Katherine Druckman: Awesome. That sounds cool. Thank you again. I'm glad we met. I'm grateful that there are people like you who are willing to put in the work to do these kinds of things. Thank your maintainers, people.
And congratulations on a massively successful project. I can't wait to see where this goes. I'm excited that you've gotten this far, and everything has gone so smoothly after seeing you on stage at Open Source Summit. It was a big moment, right?
Madelyn Olson: It's true.
Katherine Druckman: You could tell, it was a big moment for you, it was a big moment for the community, and it seems to have gone smashingly well, and I'm very happy that that happened. And again, thank you for being here and we hope to get a chance to do it again.
About the Guest
Madelyn Olson, Principal Software Engineer, AWS
Madelyn Olson is a co-creator and maintainer of Valkey, a high-performance key-value datastore, and principal engineer at Amazon Web Services (AWS). She focuses on building secure and highly reliable features, with a passion in working with open source communities.
About the Host
Katherine Druckman, Open Source Security Evangelist, Intel
Katherine Druckman, an Intel open source security evangelist, hosts the podcasts Open at Intel, Reality 2.0, and FLOSS Weekly. A security and privacy advocate, software engineer, and former digital director of Linux Journal, she's a long-time champion of open source and open standards. She is a software engineer and content creator with over a decade of experience in engineering, content strategy, product management, user experience, and technology evangelism.