In this episode of Open at Intel, host Katherine Druckman chats with TestifySec’s John Kjell, who maintains several projects, including Witness and Archivista, under In-toto at the Cloud Native Computing Foundation (CNCF), and contributes to SLSA in the OpenSSF community. They talk about the significant burden maintainers face, particularly regarding security, and how tools like OpenSSF Scorecard can be both beneficial and overwhelming. John also sheds light on the complexities of maintaining multiple projects, balancing professional and personal life, and the evolving nature of secure developer identities in open source. Additionally, he discusses the importance of inclusivity in the community, his journey into the open source world, and the collaborative projects aimed at enhancing supply chain security.
“Maintainers have many burdens placed on them, and we’re adding to their load by asking them to do more from a security perspective.”— John Kjell
Katherine Druckman: Hi, John. Thank you for joining me. Tell us a little bit about who you are, what you do in your daily life, and what you do in the open source community.
John Kjell: Yeah, well, thanks for having me. I'm John Kjell. I'm director of open source at TestifySec. We're a startup focused on supply chain security, compliance, governance, things like that. In the open source world, I’m a maintainer of a couple of different projects, Witness and Archivista. They're sub-projects under in-toto in CNCF. I also contribute to SLSA in the OpenSSF community. Previously, I co-led the Security Tool Belt special interest group (SIG) within OpenSSF, trying to help people connect a lot of the different projects OpenSSF has. But that's gone away and now there's a new security baseline effort coming up. We're trying to help the projects within OpenSSF adopt those practices as a starting point for that learning journey. And then, additionally, I just became a technical lead for the CNCF Security Technical Advisory Group. So that's another exciting and fun thing to be a part of.
Maintainer Burnout and Security Challenges
Katherine Druckman: I have to say, that's an overwhelming number of things to be involved in and I would like to pull that thread in a little bit. When I ran into you at Open Source Summit in Seattle, I was talking to a lot of people about the idea of maintainer burnout. At that point, it was very much in our collective consciousness because the XZ vulnerability had just happened, and people were hyper-aware of the role of open source maintainers in bolstering all of our security. There's a large burden placed on people maintaining open source projects. Maybe we just start there. Tell me about how your work in the community is informed by your feelings on what it's like to be a maintainer and the weight placed on your shoulders from a security angle.
John Kjell: Yeah. I think I have a unique experience in this; working on security tools gives us a unique perspective because we have both challenges. One is the time and effort we put into maintaining the projects that we're a part of, contributing to the external requests that come in from potential users or answering questions from people who want to contribute code. And at the same time, we want other projects and maintainers to use the tools that we're building to improve the security of their projects.
Maintainers have many burdens placed on them, and we’re adding to their load by asking them to do more from a security perspective. And not to blame any specific piece of tooling, but, for example, the security Scorecard from OpenSSF gives you a whole bunch of security results that now you have to go and do something about it. People absolutely should use the tool, but if you don't do anything, you never get the feedback that you need to improve, which is an “ignorance is bliss” sort of thing. At the same time, maybe you don't have time to address all those concerns and feedback, and until you decide to use that tooling, no one else is going to know either unless they're really looking deeply at your project.
Katherine Druckman: I think that's a good point. Scorecard is something that I like to talk about because I think it's incredibly valuable. In my previous life as a developer, and for a little while as a maintainer, I was always hyper-aware and wanting to learn more about security and get that kind of feedback. I would've found something like Scorecard incredibly valuable. Who doesn't love a good checklist, especially when there's some automation there?
But it is a bit of a burden because there's always this sense of, for me, a little bit of insecurity about my security expertise, or at least there was at that time. So yeah, that also adds to the emotional burden that comes with maintaining a project. You have the heavy responsibility that comes with security, and you also have a lot of people with opinions and requests and stuff always coming at you, and that's very difficult.
Balancing Multiple Projects and Personal Life
Katherine Druckman: You mentioned you are involved in so many different projects now. They obviously go together and they dovetail nicely. But how do you manage your time and how do you balance your contributions in all these different areas? How do you prioritize? How do you triage your daily life? Deep breaths.
John Kjell: It's hard. Yeah. And outside of all of that, I've got three kids at home, a couple of dogs, a couple of cats, my wife. There's a lot going on in life all the time. I think that the first part is to have a long-term view of balance. There are weeks that are mostly all work and there are weeks where I don't get hardly any work done because I'm taking a break from everything else. When it comes to the projects themselves, my employer pays for me to contribute to Witness and Archivista so that's kind of my primary focus day to day. And then the other projects are somewhat my free time and much more limited in the amount of effort that I can put towards them. Attending the weekly community calls and trying to stay up to date on the pull requests and issues can be really challenging.
But there are really great people taking notes at different community meetings where you can catch up afterward if you have to miss a meeting or miss a call. I think in some of the specification work, especially in the SLSA community, there's a lot of progress made during those weekly calls, and the conversations that happen there push things forward. And it's a great place to come and offer your perspective and opinions. At the same time, other projects are much more asynchronous. The work that we do with Witness and Archivista is usually driven through GitHub as people file issues or pull requests. We have one community meeting each month, which allows for some balance and being able to address concerns over time.
Security Risks in Smaller Projects
Katherine Druckman: Let's talk a little bit about these various projects. The projects you mentioned are obviously quite a bit smaller than a big-name project. Let's say Kubernetes, for example. Kubernetes is a massive machine with a lot of support and a lot of people working toward getting releases out and steering the technology in general. But seen through a security lens, what are the risks when a smaller project, particularly a single maintainer project or a project with few maintainers, gets to a place where maintainers are overwhelmed and overburdened, starting to burn out, and don’t have the support they need? What is the security risk there?
John Kjell: At the starting point with XZ, we didn't even really know how many people were involved. It's so easy to create a new GitHub account and generate activity over time. We've seen the new Siren mailing lists from OpenSSF. There was a blast about reputation farming, people going to GitHub, commenting and improving old pull requests, and being able to get credit for those things in a sense. Then they show up in the community as an active member with some contributions and it's more likely they’ll be accepted. And the way all these projects get consumed is fairly organic, especially the smaller ones. It starts with one project here or there and grows over time in a way that is unexpected. It's when something happens that people realize the full extent of all the places where the code is used.
A concrete example of this unexpected wide distribution is the Sigstore project, which is also part of OpenSSF. They use the in-toto specification, which they consume from the in-toto-golang library. They use a very small piece of the library and you wouldn't really know. The only way I found this out was by looking at the consumers of in-toto-golang. There are like 6,000 different projects using it, and the only reason for that is because it's indirectly consumed through Sigstore. So, people have this transitive dependency to another project, which if something happened to one, it can flow all the way down through these other projects as well.
Katherine Druckman: Yeah, the number of hidden dependencies that we may at some level be aware of, I don't know about you, but I always tended to forget they exist until something came up. That's an interesting conversation.
Developer Identity and Reputation
Katherine Druckman: Something you said earlier about reputation farming is really interesting. This whole conversation around developer identity and crediting systems and the way that we craft a reputation online today is complicated. It raises some interesting things because any system, to an extent, can be gamed.
You can game green check boxes on a GitHub profile, you can game credit. I come from the Drupal world and there's a whole credit system there. The conversations in that community around crediting systems tended to be around companies trying to game it in terms of marketing, but when it starts to become a security issue, it's a completely different conversation. The ways in which we identify ourselves as humans/developers on a platform like GitHub have become far more critical than ever intended. I don't think it was built that way. It was not built to be such an important and integral piece of the software supply chain puzzle, and yet here it is and here we are. I wonder if you have any thoughts on what a more secure developer identity would look like in the future?
John Kjell: That is a good question. I have more thoughts about what is probably not the future than what is.
Katherine Druckman: That's how you get to the answer, right? The process of elimination is a valid methodology.
John Kjell: Yeah. Requests for information about the true human identity of a contributor are not something we should ever need. There could be malicious actors from any country in the world. It doesn't matter if you're in the United States where we are or somewhere halfway around the world. Other than malicious intent, there are valid reasons that people should not have to expose their identity. We've seen an unfortunate history of people, usually minorities, underrepresented people, and women, being harassed in the community because people know their identity. And it extends well beyond GitHub issues and comments there. It's so easy to find people in a community; we have to do everything we can to protect those who need protection. And after that, the other important part is inclusivity, right? I can show up for community calls between 8:00 a.m. and 5:00 p.m. during my workday because my employer is paying for that.
Other people can't show up for those communities during the day because their employer won't let them contribute. Or we have a ton of community meetings that are in the morning my time, which works out well for people in Western Europe. I don't have too many in the afternoon because it excludes people from Europe. But at the same time, we're excluding people from Asia, Eastern Europe, and other parts of the world, which is unfortunate because the diversity of perspectives and opinions can really be valuable. And tying that to contributor identity, when these people show up, they turn their cameras on, and you know they're human beings, not an AI bot. You can build these relationships over time if you're fortunate enough to have the financial ability to travel to an event like a KubeCon or Open Source Summit, where we met, and now we know each other personally. We've seen each other in person.
Katherine Druckman: I know you’re an actual human. I have verifiable proof.
John Kjell: But not everybody has that privilege or that opportunity. Whatever solution we come up with should take all those things into consideration.
Katherine Druckman: I appreciate you saying that, especially having been around for a while. I have seen women, and others, disappear from communities because of the things that you mentioned, and that is something I think we need to really be careful to avoid going forward. I wondered if we could just pivot a little bit to the projects. We talked about how many things you're involved in, which honestly, kudos, because I'm incredibly impressed by that. Could you talk a little bit more about those projects, especially the ones that help us with this whole security conversation?
John Kjell: A lot of the work I do is in some way related to in-toto, which is a specification for capturing information about supply chains in the process of building software. Everybody today is probably using it whether they know it or not, especially developers. And we've seen a lot more recent adoption as well. The Supply-chain Levels for Software Artifacts (SLSA) specification creates an in-toto attestation. Homebrew just recently started creating SLSA attestations for everything and publishing attestations as well. So, any time you do a Homebrew install, you're using in-toto. And GitHub just announced support for storing attestations as a part of GitHub projects. You can go to a few projects that are producing these and if you go to the project URL/attestations, you can see all this metadata.
Once you have that metadata, you can create policies. The in-toto community calls these policies layouts, and they specify the set of expectations for the first step of the supply chain, which is a git-clone operation. And then here's where we should be building and testing the software. As we discussed earlier today when we were talking about CrowdStrike, software testing is critically important. And you may have other compliance concerns, such as scanning for vulnerabilities, or checking open source licenses. You can connect all these things together and verify the policy before you deploy or release that software.
We talk about a supply chain because all these things are linked together. Within in-toto and these policies, you can require that the inputs and outputs of different jobs match up with the subsequent ones. So not only did things happen, but they happened in the right order. People didn't modify things in between one step or another. That was one of the things we saw with SolarWinds. In between the source code being checked out, the source code repository was never compromised. The build process was never compromised, the results were signed, but because someone was tampering with all those files inside of the build process, they weren't the same things that were being checked out from the source repository. And so, being able to understand and detect those things is really important to the software’s overall security.
Katherine Druckman: You mentioned SLSA, and I want to mention that we had an episode, quite a while ago about SLSA. The podcast was called Security at Every Step: Why Software Supply Chains Are Critical, with Marcela Melara of Intel, and she went over it a bit. But I wonder if you could tell us at a higher level what SLSA is, for anybody who's not familiar.
John Kjell: Yeah. I think when most people try to remember the words of the acronym SLSA, they stumble a little bit. It’s Supply-chain Levels for Software Artifacts.
John Kjell: They have separated the idea into different tracks. The main track that is released as 1.0 is the build track, and the specific thing it's concerned with is build provenance. So, really understanding that where you got an artifact from is where it should have come from. The best example of this is the GitHub repository, which uses GitHub actions to build software and then creates a GitHub release out of that. When you download the release, it becomes a container image or something else, and it's removed from that GitHub repository. Maybe it goes into a different packaging ecosystem's repository. You do an NPM install of something that's not coming from GitHub anymore, but you can use this provenance attestation that's also build provenance to verify this is the artifact. It actually came from this GitHub repository and the expectations we have about it are still accurate.
Open Source Origin Story and Community Involvement
Katherine Druckman: Thank you, that was great. I'd love to hear more about your open source origin story and how that relates to the work that you do with organizations like the CNCF and the OpenSSF. I know you were lucky enough to have a company sponsor your open source contributions, which is fabulous. I have the same, and I realize that we are in a privileged position, but I wondered how you got here and why it is important to you personally as well as to your company to contribute to things like this.
John Kjell: My first exposure to open source came in my first job out of college. I was working for Veritas, which was bought out by Symantec. I was in the storage side when it split out and I was ready to be done with tech. I worked there for seven years. We had five different CEOs, we had layoffs pretty much every quarter. There was a lot of toxicity, and I wasn't quite sure that I could keep doing that. And then I was very fortunate to join Pivotal. Pivotal Software was eventually acquired by VMware and now Broadcom. But the community there, the people pretty much throughout the entire company, were amazing human beings. The things I learned about empathy and inclusivity and the community nature at Pivotal was incredibly impactful to the way that I continue to engage with the community and other people in my life day to day.
I originally worked with Cloud Foundry. And then as I moved on to different groups inside of Pivotal, I started working on software distribution. We had a website where customers could download software. After the VMware acquisition, we learned that VMware had their own website for downloading software and that ours wasn’t really needed. We were asked to start thinking about how we could turn the idea of what we did with software distribution at Pivotal into a product, and how we could help ensure that it was really secure. I started showing up to a new CNCF working group called the supply chain security working group. They were working on an original version of a paper, the Supply Chain Security Best Practices Guide. And that was where I met a huge number of people in the community.
One of the people in that group at the time was Emily Fox. I showed up on the first call basically saying, "I have no idea what I'm doing. I don't know why I'm here. Thank you for letting me show up." And she asked me what I thought on the first day. And that was the reason I kept showing up. Not only was I there, but I was also invited in, I was welcomed, and people expressed a value for the things I had to say. It was one of the next most important things in this kind of progression. I showed up and I haven't left that community since. And I keep discovering these new communities from experiences I have along the way, and it's so hard to leave any of them. Great people involved and it's a really good experience to have.
Optimism for the Future of Open Source Security
Katherine Druckman: That's fantastic. I appreciate that you mentioned how important it is to make people feel valued and make sure that everyone is heard, because I think sometimes, especially newcomers to certain communities, don't always have that experience. And it's important to remember that for the community to be successful you do need to ensure that is their experience. Well, I appreciate all of this. I thank you so much. I think our conversation has, frankly, made me pretty optimistic about the open source community's ability to address all of these complex problems that we're facing. I wondered if you had any parting thoughts. You seem pretty optimistic, so I feel like it's safe for me to assume that's going to be your take, but is it?
John Kjell: It is. Yeah. And I think we've seen really positive progress specifically with the projects I work on. With supply chain security, we started out with nobody signing anything. And then the Sigstore Project came along and it was so much easier for people to sign things. We got way more signed container images and artifacts, and then SLSA came along and that was the next stage. Now we have signatures, we have provenance, we have a better understanding, and we keep getting more and more of these attestations. We can put it all together and get a comprehensive picture in transparency around how software is built and where it comes from. And I think all of that, combined with the great communities we have out there, is going to make a big dent in solving a lot of these problems.
Katherine Druckman: Fantastic. Well, thank you so much for joining me and for getting on a call and talking about these pretty tough problems. I look forward to seeing you at the next thing.
John Kjell: Yeah, thanks for having me. I enjoyed the conversation.
About the Guest
John Kjell is responsible for open source at TestifySec, a software supply chain security startup. He is a maintainer for the Witness and Archivista sub-projects under in-toto. Additionally, John is an active contributor to CNCF's TAG Security and multiple projects within the OpenSSF. Before TestifySec, John was an engineering leader at VMware, helping to bring supply chain security features to the Tanzu Application Platform.
About the Host
Katherine Druckman, an Intel open source security evangelist, hosts the podcasts Open at Intel, Reality 2.0, and FLOSS Weekly. A security and privacy advocate, software engineer, and former digital director of Linux Journal, she's a long-time champion of open source and open standards. She is a software engineer and content creator with over a decade of experience in engineering, content strategy, product management, user experience, and technology evangelism.