Jimmy Zelinskie, Co-Founder of AuthZed, discusses the importance of authorization systems and shares an overview of the Kubernetes authorization landscape. He believes the best way to solve authorization problems is to store relationships between entities in systems and find a path in the graph of those relationships to determine access. Zelinskie also introduces SpiceDB, a tool that takes principles from Google's authorization solutions and makes them open source. He mentions the adoption of SpiceDB by small businesses and Fortune 500 companies, and the ability to model custom schemas for permissions. Zelinskie highlights the importance of open source in the ecosystem and the need for new abstractions in Kubernetes to support heterogeneous workloads.
" We believe that the best way to actually solve authorization problems is to store relationships, store data that represents the relationships between entities in your systems." – Jimmy Zelinskie
Katherine Druckman: Hey Jimmy, thank you so much for devoting a little time to me here at KubeCon. It's an intense environment and everybody's very busy, and I really appreciate it.
Jimmy Zelinskie: Thanks, Katherine. Yeah, it's definitely the biggest KubeCon. The floor is just packed.
Katherine Druckman: It's been packed the last three days that I've been here on the expo floor. It's way bigger than the last one for sure. I don't even know what to make of it. It's kind of overwhelming.
Jimmy Zelinskie: Honestly, what's most interesting about it is they've built in 30-minute breaks in between talks, and in that time, it just pulsates in the expo center and you get a rush of people. I've seen many, many booths completely overwhelmed. We've been overwhelmed.
Katherine Druckman: Yeah? That's great. I mean, it's a great problem to have.
Jimmy Zelinskie: It's a great problem to have, but I do wonder if there's some other way to spread out the traffic.
Katherine Druckman: Right, you spread the load.
Jimmy Zelinskie: Yeah, exactly.
From BitTorrent to Kubernetes Pioneer
Katherine Druckman: I wonder if you could just tell us a little bit about who you are and why are you here at KubeCon?
Jimmy Zelinskie: Sure. I actually have a super long history with KubeCon. I actually started going to KubeCon EU Copenhagen, but I have been in the cloud native ecosystem since before the CNCF was even created. I basically started my career working in open source. Basically, in college, I wrote a bunch of BitTorrent software that became a part of the container orchestration layer at Facebook. And when I graduated school I decided, hey, this kind of distributed systems, ecosystem, containers, these are a real thing that's happening right now. And at the same time, my future co-founders Jake and Joey had just gotten acquired by CoreOS, a brand new operating system that was revolutionizing the way people were doing operations. I knew CoreOS from the Go ecosystem.
Katherine Druckman: Okay, yeah.
Jimmy Zelinskie: And decided, hey, I want to jump into this ecosystem. And now they are in New York, and I'm on the East Coast. I didn't want to move to the Bay Area right out of school, with all my family around the East Coast.
Katherine Druckman: Sure.
Jimmy Zelinskie: I jumped to that startup, and basically, it's been history ever since. We worked on Quay, which is the first private Docker registry. We worked on Tectonic, which is CoreOS's Kubernetes distribution, and then ultimately led to creating operators as a pattern in Kubernetes. Then CoreOS got acquired by Red Hat in 2018, and around that time, we actually became the largest SaaS product running at Red Hat, where we created a whole SRE organization there to teach Red Hatters how they should run software for their customers and not just support software that's running in their on-prem environments. And then the pandemic hit, and in 2020, we decided, hey, I think it's time to jump right back on the horse and do a new startup.
Jimmy Zelinskie: We ended up leaving to create AuthZed, which is our current company. What we do there is build SpiceDB, which is taking similar principles to what CoreOS was doing back in the day and taking internal Google magic and making it open source and available for everyone else. And so what we're doing now is taking the authorization solutions, the solutions that Google is using to secure their whole product suite, the things that power experiences where Gmail can pop up warnings because they know you haven't shared a Google Doc with somebody before you send your email. These types of integrations are not sharing code across these products. What they're actually doing is querying one central system, and that central system is called Zanzibar. And what we do is we implement Zanzibar completely open source for everyone else called SpiceDB.
Katherine Druckman: Beautiful, you've been busy.
Jimmy Zelinskie: Oh, you bet. You bet.
SpiceDB: From Concept to Adoption
Katherine Druckman: I don't even know where to start with that; you've been so busy. I love it. Tell me really quickly again, what has the trajectory of SpiceDB been so far?
Jimmy Zelinskie: Yeah, SpiceDB, we've seen it get a surprising amount of adoption. Basically, we had this premise coming out of the gate that SpiceDB would solve problems for everyone. We started the company, we talked to a lot of startups. What we quickly learned was that startups were actually maybe not the best fit. If you work in something like healthcare, you might have really sophisticated authorization problems. For example, let's think of a typical hospital environment where you have doctors and they have access to patient records, but probably only if they're working at a particular clinic, at a certain day they're assigned particular clinics. You can see how the access that they're allowed to have depends on all these different variables and all this data. Even when you're a small healthcare company, you have to deal with all these types of rules. The only kind of folks we saw in the startup environment who really cared about authorization were folks in more sensitive areas, like government compliance or healthcare.
Katherine Druckman: Okay, sure. Makes sense.
Jimmy Zelinskie: But what we did realize was that basically everyone from small businesses past their startup and MVP point all the way up to the large tech giants that hadn't invented this themselves, they were all potential customers for us.
Katherine Druckman: Sure. Everybody needs authorization, yeah.
Jimmy Zelinskie: Exactly. Everyone needs sophisticated rules around authorization and making sure that they're not introducing new security flaws when they make code changes. Because when a customer, for example, for an enterprise app says, "Hey, actually, well, we have this other layer of indirection where we actually have business units at our organization, so we have teams under business units, and then those teams can have other teams inside them. And then we have users." You start talking about these more complex hierarchies, but they're not just hierarchies. They can be completely directed graphs, so they can be even more complicated shapes. The second you get requirements like this from an enterprise customer, you're basically in for a world of hurt where now you have to go and rewrite a whole bunch of code, you probably have assumptions all over your code base about how that was supposed to work.
It's all security critical. Once you get someone who is an expert in that, to actually change the code, now you probably have to go through a security audit process with another vendor that's going to guarantee that you're not introducing any flaws. You just have this terrible, terrible loop and you don't know how long until you get the next customer that's going to make the next requirement change that's going to cause just as big of a refactor. It could happen before you even finish the refactor that you're already currently working on. We experienced this a couple of times, and we're like, "We're going to set out and solve this problem," and that's what SpiceDB solves. We've seen adoption from small businesses where it's just co-founders working at the company all the way up to the Fortune 500.
Navigating the Authorization Ecosystem
Katherine Druckman: Wow, okay. Tell me again where this fits in the ecosystem. I've talked to some other people about other authorization or role-based access control type things. I'm trying to think of the project. I talked to OpenFGA; I just talked to somebody from that. Where do you fit in with this world?
Jimmy Zelinskie: Yeah, OpenFGA and SpiceDB are actually quite similar projects. SpiceDB came out first, but OpenFGA is a CNCF sandbox project. There's been a lot of innovation and cross-pollination in that space.
Katherine Druckman: That's great.
Jimmy Zelinskie: Yep. Because it's an emerging space, we welcome a lot of different vendors and different strategies to help advance and move forward with authorization tooling, basically the history of authorization. There's also Open Policy Agent, which is a common tool used in the CNCF ecosystem as well. But there are a lot of different solutions in this space that are newer. What we really want people to do as an industry is move away from building these things custom themselves by non-experts.
Tools like SpiceDB, OpenFGA, and Open Policy Agent all have slightly different use cases, but ultimately, at the end of the day, what it's doing is solving a really important problem, which is having experts build the software that is basically securing your software so that you're not introducing any mistakes in your logic.
Katherine Druckman: This is one of those areas, again, we all talk about shift left. We love to say that, and I start to wonder if everything shifts left, then what's left to the right? But yeah, it's such a critical, critical thing here because it's one of those areas in software where it's easy to leave some doors open. It's just so easy and it's so hard to get it right. I could certainly see that if you have people that are obsessed with it, their entire life is figuring out how to not leave those doors open, you want to take advantage of that.
Jimmy Zelinskie: Absolutely.
Katherine Druckman: Solve it as a common goal.
Jimmy Zelinskie: Yeah, it's really interesting because the history of research into authorization systems and securing systems goes just as far back as something like database research. Every application engineer these days understands that they shouldn't build their own database.
Katherine Druckman: Yes, sure. Exactly.
Jimmy Zelinskie: They should be going off something like Postgres or MySQL. They know that it would be a fool's errand for them to reinvent the whole data layer.
Katherine Druckman: You should not rewrite Kubernetes for your own custom purpose, right?
Jimmy Zelinskie: Absolutely.
Katherine Druckman: Probably not.
Jimmy Zelinskie: This is just another example of that case. In fact, with SpiceDB specifically, we model it as a database. We believe that the best way to actually solve authorization problems is to store relationships, store data that represents the relationships between entities in your systems. And then what we do is find a path in the graph of those relationships to say whether or not someone has access. A really good example of this is back in the day when you were sharing photos on Facebook, when you were deciding who can see your photos, you would say, "Okay, only I can see my photos. Only my friends can see my photos, friends of friends can see my photos." And if you think about it, you're actually talking about relationships for who relative to you can see this data. We think that that's a natural way to actually express permissions.
Katherine Druckman: It fits into org structures.
Jimmy Zelinskie: Exactly, org structures. The really powerful thing that SpiceDB lets you do is you get to model a custom schema so you get to define that structure. A lot of other people are taking pre-existing systems that might be fairly rigid in their shape, and then they're forcing their organization to fit in some kind of software limitation. But what we let you do is actually model organically how people are organizing and how software and things are actually organized together, and then represent that in a way that's safe for you to reason about and test and understand to make sure your system's secure.
Katherine Druckman: That's very interesting. And actually, I do see the similarity after talking about OpenFGA. That's interesting, and I love that you said that there is cross-pollination there. I think that is so important because, again, people are all solving similar problems, and you will need to differentiate yourself for some reason. There is some special use case where you'll be better at one thing and somebody else might be better at another. But it is encouraging to know that there's some commonality and common ground that you're finding.
Jimmy Zelinskie: Yeah, absolutely. Yeah, in general, I think that as time... I think there's a lot of talk about today, what does a standard interface look like in this ecosystem? And right now I would say, "Let's wait a second."
Let's see how all this is going. You don't want to stifle any innovation right now.
Katherine Druckman: Yeah, everybody's being very creative.
Jimmy Zelinskie: Yeah, exactly. And eventually, we're going to get to a place where we will understand what these general shapes look like and what is the most common, or least common denominator, is how I would put it. But today, this is emerging, right?
We want to get the latest and greatest, and not stifle anyone, and get all the cool stuff. I think it's a super exciting time to be looking into the space as well.
Katherine Druckman: That is cool.
Jimmy Zelinskie: Seeing the innovations and playing with these things and understanding how they can impact your workflows.
The Future of Authorization: Innovations and Roadmap
Katherine Druckman: And security being near and dear to my heart, I love that there's so much interesting work happening in this area. I wondered if you could fantasize about if we talked again a year from now, what does your roadmap look like? What do you hope to share in a year?
Jimmy Zelinskie: Yeah, what's super interesting is we've actually just entered early access for one of our products called Materialize. And what Materialize is, it’s a materialized view of the permissions in your database. Folks who are already familiar with some deep database concepts might be already familiar with the concept of a materialized view. But basically what it is in a SQL database, for example, is it takes a query that you're going to run against the SQL database, and it keeps that in memory, basically as a fake table. It basically caches the result of a query so that when someone else makes the next query, it's already loaded in memory and it can just serve directly from that. Instead of actually reading the tables on disk, now you already have the answer precomputed for you.
And then you can do things to update that when changes come in to make sure that it's always up-to-date, and you always have a fast response. We have basically done that for people's permissions. That's super powerful because you can build these crazy abstractions, these crazy graphs that are really complex in nature and have deep hierarchies and recursive groups. You have groups, and groups that have users, and we can flatten all that and make sure that it all is instantaneous for you to query. Not only does that make things faster, but then we can also stream those changes to another system. So we can say, okay, a write just came in and removed admin from this user, and that removed them from these 10 repositories.
And because we can stream that, we can actually send it to other systems. So if you have something like Elasticsearch, we can actually update Elasticsearch in Elasticsearch. So when you go to query Elasticsearch, you can actually just do native filtering on Elasticsearch based on folks' access. That means you can filter infinitely large lists. You can build a search engine that works for the entire internet but has private data being applied and private contexts. You can build all kinds of search engines for your whole company, for all the things that your product suite has, or even use that to update things in your primary relational database like Postgres. You can have all your permissions written back into Postgres, and then you can just do a join in Postgres if you want to filter things based on access in Postgres and never even talk to SpiceDB. You don't even need to know SpiceDB exists.
Community-Based Development
Katherine Druckman: That's very cool. I look forward to seeing what happens in the next year. On a more personal level, again, you have a great story, you've been very busy and have a very interesting background. What connected you to the open source community and what made you really want to jump in and do this kind of community-based development?
Jimmy Zelinskie: Yeah, I think honestly, being a college student and trying to be super scrappy and just work on the things I was passionate about got me involved pretty early in the Go ecosystem. I started programming in Go probably at some time when they were only doing weekly releases of the language. There was no 1.0 or anything like that. This was probably 2010, and the Go ecosystem was just prominently open source. It was a thriving ecosystem where folks were all contributing to the open source compiler and writing libraries. I think the experience of being in a programming language community at the beginning was a huge jumpstart to understanding open source in general, but also how the world can work with open source. Because when you start from the programming language, you have nothing, and you slowly build everything, and you get equivalents to things in other languages, and you get exposed to things that are being built in the other languages being brought into your ecosystem and the cross-pollination there. And then, basically, using that experience, I just got super used to interacting with open source. It was the only thing I knew. I hadn't worked in a big company yet.
Katherine Druckman: I have the same feeling, yeah. That's so funny. I hadn't actually worked in a big company until recently, and open source is the only thing that I know. I almost hate to admit it, but I don't know that much about proprietary systems. I have such an open bias throughout my entire background. It's almost like I might be a fish out of water in any other environment.
Jimmy Zelinskie: Yeah. I think there's a time and place for proprietary products as well.
Katherine Druckman: Sure, secret sauce maybe.
Jimmy Zelinskie: Yeah. I've been a product manager. I'm currently chief product manager. I also still program, I still do everything. I carry the pager for our services. I’m deeply technical, but I also care. I found basically that my motivation as an engineer was actually in providing value to other folks. And when that is your primary concern, product management becomes actually the driving thing. I find that you can be a larger force multiplier by not writing individual code, but by actually empowering a team and making sure you're building the right thing.
Katherine Druckman: Solving the problems, yeah.
Jimmy Zelinskie: Exactly. I became a product manager, and in that time, I have developed opinions on when things need to be proprietary because you do still have to have a business that can drive the development and democratization of software in the open source ecosystem. I think that a lot of people struggle to understand where the boundaries are between something proprietary and something open source. I've always worked at open source companies. I worked at CoreOS, and we had proprietary products, but open source was at our core.
I worked at Red Hat, that was the exact same thing. AuthZed, it's the exact same thing. So for us, I feel like we have a pretty strong understanding of basically making developers successful, making people successful with the open source, and then honestly adding business value in the proprietary aspects of the product. Because businesses want to interact with other businesses. They want certain guarantees. They want things like, I know if I'm going to run this in production, I need a number that I can call and get support from someone when something goes bad.
Katherine Druckman: That's certainly fair. I agree. I feel like sometimes we under-emphasize the human aspects of technology and software and whatnot, and that's what we're talking about at the end of the day, right? Having human features like accountability and helpfulness and all of those things and reliability. Those aren't just machines that we're talking about, and it's nice to have that kind of human connection that is even a business relationship.
Jimmy Zelinskie: Honestly, soft skills are the hardest skills.
Katherine Druckman: Absolutely.
Jimmy Zelinskie: You can always find someone who can write the code for a particular thing, but if no one wants to work with them or they're a terrible person in the community, you can't have them poison the whole well for everyone else.
Katherine Druckman: Yeah, absolutely. Everything that we do, all the problems that we're solving are human problems, so we have to solve them with human solutions, I think.
Tell me what else you're really excited about. What have you seen at the event that's gotten you really excited to get back to work or excited to have a new idea or anything like that?
Jimmy Zelinskie: Yeah. Honestly, I've been spending so much time at my own booth.
Katherine Druckman: Fair.
Jimmy Zelinskie: But I was actually mentioned in a keynote today that I didn't even realize.
Katherine Druckman: Congratulations.
Kubernetes API Proxy
Jimmy Zelinskie: I just got a text message that was like, "Hey, did you know you're on this slide this morning?" It was actually the TOC talking about the things they're excited about coming up in future versions of Kubernetes in the emerging space. I feel like I'm obligated to use the quote they chose from me to answer this question, which is honestly heterogeneous workloads on Kubernetes. Running SpiceDB and running a couple other types of things on top of Kubernetes, I've realized that you end up always rediscovering what parts of Kubernetes are absolutely super critical for your unique workload.
And for us, running a database, a particularly critical latency database, because every request going through your software has to do a permissions check before it can do anything else. We need to be able to answer these things lightning-fast. But to do that, we have to do all kinds of interesting things on Kubernetes like making sure our software runs on our particular core on the machine and that nothing else can interrupt it. Making sure of things like we're getting direct connections to our pods, because we don't want any network proxies in between us that can add extra hops on top or cause any other hiccups in the overall system, because even if we close a connection like a TCP connection, reestablishing one requires a cryptography handshake to do TLS.
That alone is large enough to blow an SLA for an authorization check. So we can't even have any disconnects while we're doing this workload, or we need to be able to basically have our pools primed to replace connections as they die. Super unique requirements and you just discover all these tiny little features in Kubernetes. I feel like there are a lot of different workloads now that are grabbing at all these different little bits. To me, in my mind, being someone who helps pioneer operators, I'm like, "We need new abstractions in Kubernetes." We need new types of workloads that we can talk about these different things and reuse patterns for running these styles of workloads. I am most excited about heterogeneous workloads running on top of Kubernetes in the future.
Katherine Druckman: Your enthusiasm is infectious, I will say that. I'm inspired here. Is there anything that you wanted to talk about that I didn't ask?
Jimmy Zelinskie: We did make one announcement for KubeCon this year which is super cool. This year we announced the Kubernetes SpiceDB API proxy. What this tool does is it actually sits between the Kubernetes API and Kubectl. When you interact with the Kubernetes API, it actually takes the request and then transforms it into a SpiceDB permissions check, and then only if the permissions check passes do you actually get to perform whatever operation on Kubernetes. What this lets you do is actually sidestep the built-in Kubernetes RBAC. There are a lot of reasons why you want to use RBAC in Kubernetes. You want to secure Kubernetes. But what we have realized is actually there are a lot of limitations in the built-in Kubernetes RBAC, and what you can do is actually use SpiceDB to model completely custom new systems. Say you don't care about RBAC, but you care about something like ABAC. You can model these different styles of permission systems and then enforce that on Kubernetes completely on the fly.
What's really, really interesting is you can have a central SpiceDB, and then share that across a bunch of different Kubernetes clusters. So you could say, "I have all my organization over here and it's my SRE team, and here's all the structure and all the groups and all that thing," and then basically have that stored in one place. But then that gets enforced on all of your clusters across all the regions where you're deploying, and you don't have to copy rules or make individual connections to each cluster. They all just get hit at this API proxy interface.
The even cooler thing is because we've been so deep in Kubernetes, we know exactly where the limitations to Kubernetes RBAC are. We've decided to make it an API proxy because we can do really cool things like filter the results. In Kubernetes right now, you can give folks the list capability to list all the resources, like tell me all the namespaces in the cluster, but you can't actually filter down the individual items based on what people can see. But using our API proxy, now you can. Now, you can actually filter the resources to your particular view of the cluster, so you won't even know that there are other resources on there that you can't access.
Katherine Druckman: Well, that's very cool. I appreciate you sharing that. This has been really fantastic. I love that you're so enthusiastic about the work you do. It makes it all fun.
Jimmy Zelinskie: I wouldn't be here if I wasn't.
Katherine Druckman: I know, yeah. I feel the same way. Keeps you going, keeps you energized. Thank you so much; this has been delightful. And yeah, keep doing the thing.
Thanks for helping the rest of us get our security together.
Jimmy Zelinskie: Thank you for having me.
About the Author
Katherine Druckman, Open Source Evangelist, Intel
Katherine Druckman, an Intel open source evangelist, hosts the podcasts Open at Intel, Reality 2.0, and FLOSS Weekly. A security and privacy advocate, software engineer, and former digital director of Linux Journal, she’s a longtime champion of open source and open standards.