Join Open at Intel host Katherine Druckman as she chats with Matt Ray, senior community manager for the CNCF sandbox project, OpenCost, which tracks the cost of resources in Kubernetes clusters and provides insight into cloud bills. The two sat down together at KubeCon Europe in March to talk about the project, OpenCost’s growing community, and the problems OpenCost is trying to solve. Matt talks about OpenCost’s new support for measuring the carbon footprint of Kubernetes workloads and shares more about how the project is gaining traction. In the future, OpenCost plans to support the FinOps Foundation's FOCUS standard for normalizing cloud bills and is also working on a plugin architecture to integrate with external cost providers. Matt also reflects on the increasing interest in cost monitoring and his journey in the open source community: “This is science,” he said. “I want to move civilization forward and you don't do that by writing closed-door software and keeping it to yourself.”
“... for most people saving money and helping the environment, in this case, go hand in hand. We're looking for ways to optimize usage to use fewer resources, whether they're financial or environmental.” — Matt Ray
Katherine Druckman: Hey, Matt, thank you for taking time out of KubeCon to talk to me about cost optimization technology.
Matt Ray: My pleasure.
Katherine Druckman: Tell us a little bit about who you are and why you are here.
Matt Ray: My name is Matt Ray. I'm the community manager for the CNCF project, OpenCost. OpenCost is cloud and Kubernetes cost monitoring. We keep track of how much everything in your Kubernetes cluster costs, and what's going on in your cloud bills. This week we announced we are now supporting carbon costs, so you can see what the carbon footprint of your Kubernetes workloads is. That's exciting.
Katherine Druckman: That is exciting. That brings it under the moral imperative.
Matt Ray: You might say saving money is a moral imperative.
Katherine Druckman: No.
Matt Ray: Luckily for most—
Katherine Druckman: It's a wonderful thing.
Matt Ray: ...for most people saving money and helping the environment, in this case, go hand in hand. We're looking for ways to optimize usage to use fewer resources, whether they're financial or environmental.
OpenCost's Origins and CNCF Contribution
Katherine Druckman: Tell me about the project maturity level, and history.
Matt Ray: When the project started, it was the open source engine of my employer, KubeCost. They kept the engine open source, but they contributed it to the CNCF in June of 2022.
Katherine Druckman: Okay.
Matt Ray: In conjunction with that, they put out a specification called the OpenCost Specification for how you measure the cost of Kubernetes resources, how you split those up across a node, how you say who has to pay for what, how costs get allocated, how you split memory versus how you split CPU, tying together requests, and limits versus actual usage. That's the specification. They worked with AWS and Red Hat, GCP, D2iQ, and a few others. That was the specification, and then the code. OpenCost started as the implementation of that.
OpenCost vs. KubeCost: Defining the Boundaries
Katherine Druckman: In a company like yours, how do you differentiate between the open source project and the business?
Matt Ray: There's a clean line for OpenCost and KubeCost in this case. A fair number of open source businesses are walking that line of asking what's commercial and what’s not. For OpenCost and KubeCost, OpenCost is monitoring. The goal of OpenCost is to find and get the best numbers, and the best metrics of the data we're trying to get. It includes parsing cloud bills and talking to Kubernetes and Cloud APIs to get their billing information.
What do you do with that data and those metrics? OpenCost does have a UI, but it's not giving recommendations. It's not autoscaling, machine learning, budgets, alerts, all the value-add on top of it. A lot of people don't need that out of their open source project. What OpenCost does is we want to be the best data source for costs and environmental monitoring and get embedded into other products.
KubeCost contributed it, and shortly after the project started, other companies and projects started picking us up and embedding OpenCost as their cost monitoring engine, too. For example, we're included in Grafana Cloud's Kubernetes cost monitoring, or Kubernetes Monitoring Suite.
Katherine Druckman: Right, okay.
Adoption and Integration of OpenCost
Matt Ray: Microsoft has embedded OpenCost in their billing dashboard. When you go to your Azure bill and say, "Oh, I want to see…" Well, here's AKS. Usually, most cloud vendors stop at that. Here's your AKS bill. And you say, "But what was going on in there?" Now you can expand that and see the cost if you break it down by namespace, each pod, or by the deployment.
Katherine Druckman: Interesting.
Matt Ray: Microsoft has become one of the maintainers of OpenCost.
Katherine Druckman: That was my next question. I did remember that there are multiple organizations represented among the maintainers, and I wondered what your contribution looks like these days. Is the bulk of the contribution still from within your company, or is it growing?
Matt Ray: We are currently a sandbox project, but we've filed for incubation. In May of last year, we filed to get promoted. Part of that is gathering up the metrics, showing the quality of community contributions, and how you're putting in the effort to become a diverse project with contributions from community members. We've had commits from about 150 individuals, and probably a third of that is KubeCost.
Katherine Druckman: Okay.
Matt Ray: That's a long tail, right?
Katherine Druckman: Absolutely.
Community Contributions and Project Growth
Matt Ray: There's a long tail of contributions, and I would say 85-90% of the contributed code right now is KubeCost, but our Azure maintainer has become quite active and all the ancillary stuff around it has taken off with community-led activities. What OpenCost has started to do is not what KubeCost cares about. KubeCost uses OpenCost to gather numbers and put them into their database. Then other people said, "I need my numbers in this format."
Katherine Druckman: Okay.
Matt Ray: They've shown up. We have a Parquet exporter. Parquet is a data format from Apache that…you know, it's CSV on steroids. Well, other things like to ingest that, so we got a community contributed Parquet exporter. Microsoft said, "It would be great if we had a CSV export." Their press release literally said, "You can open it with Excel…trademark." We've gotten different exporters and cleanup around performance. Grafana Labs contributes a lot of Prometheus enhancements to make it faster and better on Prometheus. Our data store for OpenCost is Prometheus, and they showed up with that. We're getting a lot more diversity, and this week I have someone who's volunteered to maintain our Grafana dashboards for us. We're getting new maintainers who don't work at KubeCost. We're getting a wide assortment of community contributors.
Katherine Druckman: How flexible is it then in terms of plugging into other people's applications, systems, and whatnot?
Matt Ray: We can work as a very basic Prometheus metrics exporter, where you run OpenCost and you don't have to run Prometheus for storage. You just run OpenCost, it connects to the cloud API, starts watching Kubernetes, then something else can scrape that endpoint, and put it into Grafana or another database. That is the simplest, smallest thing you can do as a straight Prometheus exporter.
We have a community-based Prometheus helm chart, of the Prometheus OpenCost exporter, if you want only that refined use case. Many people will install Prometheus with OpenCost to use as a data storage, but then forward it to other databases or business intelligence tools, and they don't use the UI and that's fine, too. That's what Grafana does.
We have users doing thousands of clusters with OpenCost, and forwarding them somewhere else, so it's got a small footprint for managing as an application, and it's easy to embed in other stuff. One thing I've been doing this week here at KubeCon is going to all the Kubernetes distributions and vendors saying, "Look, add this and your customers get cost monitoring. It's a CNCF project. We want you to engage, too." I've had really good feedback from that.
Flexibility and Use Cases of OpenCost
Katherine Druckman: I hear more conversations about cost monitoring and that type of observability, and I wonder, why am I hearing all of this right now? Why more today than a year ago, for example? Were we all going out willy-nilly and spending too much money for the last 10 years? Why is it such a conversation right now?
Matt Ray: There's been a post-COVID belt-tightening.
Katherine Druckman: It's the economy.
Matt Ray: I saw a statistic that said only 21% of software companies hit their sales targets last year, which means layoffs, spending cuts, and belt-tightening, and people start to ask, "Shouldn't we be saving money?"
Katherine Druckman: Yes, let's look at this more closely.
Matt Ray: You can use OpenCost to get those numbers. You can also use my employer KubeCost to get recommendations and budgets. We're seeing a lot of interest.
Katherine Druckman: There are a lot of opportunities for improvement, right?
Matt Ray: Absolutely.
Katherine Druckman: When you look at an entire system, it’s complex, right? There are a lot of moving parts to any software or application today.
Matt Ray: Yes.
Katherine Druckman: When you look at something like this, you're talking about deep granularity in terms of pinpointing where the cost savings are.
Matt Ray: Right.
Katherine Druckman: I'm wondering if there are commonalities. Do you see things that are consistently identified as pain points or things that people are wasting money on?
Matt Ray: The first thing that happens when people turn on OpenCost is they see how much their workloads are, and how much of the resources they've allocated are being consumed. We show users an efficiency number and show them an idle percentage. If you've deployed Kubernetes to 10 machines and are running workloads on it, we show you that of those 10 machines, you're only consuming 30% of them. This is common. People over provision by a lot. Even if you're on premise, that's consuming resources that you didn’t expect.
We show people how much they're wasting, and they can ask, "Are my containers set to the wrong requirements? Do I need to request fewer resources?" We also show people when things are not being used at all, and when there are dead jobs that don't talk to anything. OpenCost itself does not make recommendations. That's an exercise for everyone who uses OpenCost. You can use this as the fact finder and then optimize with that. Some people have used autoscalers built on top of OpenCost where they can look at costs, be more efficient, and change requirements to squeeze out more workloads. KubeCost, for example, will say, "You're deployed on this many nodes. Have you thought about reducing the number?" Or "You're on medium instances, have you thought about moving to extra-large, but fewer of them, and you'll save money." There are a lot of optimizations that come from just having this visibility.
Becoming a Committer and Maintainer
Katherine Druckman: In terms of growing the number of project contributors and graduating some of those contributors to new maintainers, how do you approach that? How do you approach encouraging new contributions?
Matt Ray: We've documented how you become a maintainer and committer and join the community. As soon as somebody shows up and says, "I want to get involved," we add them to our GitHub organization. It gives you a nice badge on GitHub that says OpenCost. There's not a lot of commitment there. We give them a badge for contributing comments on issues, opening issues, or even pull requests. The next step is to become a committer, which means you have permission to do reviews. Everyone can comment on issues, but approving a pull request is a member tier of our community called a committer.
We currently have five different repositories in OpenCost. Each of those has different maintainers. We have the main OpenCost with things like the Helm chart, Parquet Exporter, or documentation, each has different maintainers. I'm more than happy to escalate people to maintainership fast on everything but the main OpenCost project.
With our documented process to become a committer, another committer has to recommend you as a committer and all the other committers vote. We're still early on that. Now people just say, "Yes, that person should be a committer." But when it comes to the maintainers, we have a formal vote between the current maintainers and say, "Hey, this person has shown consistent quality in contributions." I opened a pull request to add Archer, our Microsoft maintainer. We opened a pull request and all the other maintainers had to approve it, and then he joined the maintainership.
That's our process. It mimics a lot of other CNCF projects. It's not particularly different. Most of our community action is happening in Slack and GitHub, and every two weeks we have a community meeting.
Katherine Druckman: Did you see an increase in participation?
Matt Ray: We've seen an increase in participation. It helped once we got a non-KubeCost maintainer.
Katherine Druckman: Then you become more legitimate. I get it.
Matt Ray: Part of that was trying to push conversations out into the open. It's not that KubeCost didn't want to do it, it’s just that they didn't have to.
Future Plans and Focus
Katherine Druckman: You're a new project and certainly new to the CNCF. If I talked to you a year from now, what do you hope to change in that time? What do you see happening soon?
Matt Ray: We have a lot of aggressive ideas about what we can offer. We are involved in the FinOps Foundation, which has a standard that they're building called FOCUS. It's the FinOps Open Cost and Usage Specification. They're trying to take all the cloud vendors and normalize their bills so when it comes time to pay your object storage bill, you can compare Google to AWS to Azure because they all have different names, they all have different ways of billing them, but this is trying to normalize them into a single standard.
In the next year, we will support FOCUS, which will be released, I think, in June of this year. [Note, this interview was recorded in March prior to the June release of FOCUS.] We can ingest all the FOCUS bills of different cloud vendors and show you apples to oranges.
You can compare your AWS to Azure bills inside of OpenCost and make informed decisions about your usage patterns based on placing the numbers next to each other.
We just announced the carbon cost. That's brand new. It's very rough. It only covers three cloud vendors right now, but we're going to make sure we get as many as we can into that. We are also announcing a new plug-in architecture for bringing in external cost providers. Our reference example is Datadog.
Katherine Druckman: Oh, okay.
Carbon Cost and Plugin Architecture
Matt Ray: Bringing in SaaS is an easy example. If you want to bring in bills for Datadog, Splunk, monitoring, or CloudFlare, bring in the data, see it in OpenCost, and put it behind our APIs. Then you'll be able to do things like correlate your CloudFlare and Datadog to a particular namespace and get the true cost of your application. You can also get the true carbon cost of your application. Then you could ask, "Well, what if I ran it on Azure versus GCP?"
Our vision is to bring all these things together to make informed decisions on things you care about.
Katherine Druckman: I wondered if you could tell us about your experience. I am guessing you were part of the process of initially becoming a sandbox project with the CNCF.
Matt Ray: I joined KubeCost right after they started a lot of the conversations. I had worked at another Kubernetes-related startup, and I've been working in open source for a long time.
Katherine Druckman: A long time. I understand. Like, let's stop counting.
Matt Ray: Yes, let's stop counting. That's one of the things that attracted me to working at KubeCost, was their open source involvement, and I knew that the CNCF work was in progress.
Katherine Druckman: Okay.
Matt Ray: Initially, I joined the company with a different role, but as soon as the position of community manager came up, I said, "I can do that. I've had a lot of open source experience over the years." I wanted to get involved and be the shepherd of getting this to a place where it can stand alone from KubeCost.
Katherine Druckman: They pay me.
Matt Ray: They pay my salary, and they're doing an excellent job as contributors and maintainers of the project. However, the goal of the CNCF is to have a diverse set of contributors and users.
Katherine Druckman: Emancipate the project a bit from the company. I wonder, what attracted you to that kind of work? Open source people are a unique breed, right?
Matt Ray: Yes, we are.
Matt’s Personal Journey in Open Source
Katherine Druckman: What is it that attracted you to that specific role?
Matt Ray: I've been involved in open source, and I participated in Red Hat’s IPO. That was a long time ago.
Katherine Druckman: Oh, lucky you.
Matt Ray: I have horror stories about that. I was in college, open source was new, and I was competing for a while, and started to realize that this is science. In science, you learn from other people's results, you share your successes and failures, and don't invent a rocket ship without having a long history of all the model rockets that came before it. These things we're seeing in AI now were not built from scratch. They are built on years of AI research, everyone else's experiences, and having all these beneficial things come together at one point.
That's how I always felt about open source. This is science. I want to move civilization forward and you don't do that by writing closed-door software and keeping it to yourself.
I worked at a large monitoring company that was closed source, and the product I worked on was making about $60 million a year in revenue. I was tasked with winding it down because they weren't making enough, but not telling the customers they were winding it down. We gutted out all the internals of anything expensive, outsourced a lot of the work, and laid off a lot of the team to squeak out another four or five years of revenue without paying for it. My employer said, "Hey, you're good at this. We'd like you to do some other projects." And I was like, "I'm done with this. Thank you very much."
Katherine Druckman: It doesn't feel good.
Matt Ray: I called up a friend of mine who was an industry analyst, and I said, "In the open source scene, who should I go work for?" He helped me and I've been doing that since.
Katherine Druckman: I appreciate you sharing your story. Is there anything you wish I had asked?
Matt Ray: No. I'm always happy to talk about OpenCost. I love talking about open source. I have strong opinions. Thank you for having me on the show.
Katherine Druckman: No problem.
You've been listening to Open at Intel. Be sure to follow us for more on LinkedIn and X. We hope you join us again next time to geek out about open source.
Guest:
Matt Ray has been active in open source and DevOps communities for over two decades and has spoken at and helped organize many conferences and meetups. He is currently the senior community manager at KubeCost for the CNCF sandbox project OpenCost. He has worked in and with enterprises and startups across a wide variety of industries including banking, retail, and government. He currently resides in Sydney, Australia after relocating from Austin, Texas. He co-hosts the Software Defined Talk podcast and is active on Mastodon, GitHub, and too many Slacks.
About the Host:
Katherine Druckman, an Intel open source security evangelist, hosts the podcasts Open at Intel, Reality 2.0, and FLOSS Weekly. A security and privacy advocate, software engineer, and former digital director of Linux Journal, she's a long-time champion of open source and open standards. She is a software engineer and content creator with over a decade of experience in engineering, content strategy, product management, user experience, and technology evangelism.