---
title: "Omer Levi Hevroni — K8s can keep a secret?"
url: https://appsecpodcast.com/omer-levi-hevroni-k8s-can-keep-a-secret/
date: 2019-05-01
duration_seconds: 2202
guests: ["Omer Levi Hevroni"]
topics: ["Cloud and Infrastructure", "DevSecOps and CI/CD"]
audio: https://www.buzzsprout.com/1730684/episodes/8122648-omer-levi-hevroni-k8s-can-keep-a-secret.mp3
transcript: true
---

# Omer Levi Hevroni — K8s can keep a secret?

*May 1, 2019 · 37 min*

with [Omer Levi Hevroni](https://appsecpodcast.com/guests/omer-levi-hevroni/)

on [Cloud and Infrastructure](https://appsecpodcast.com/topics/cloud-and-infrastructure/), [DevSecOps and CI/CD](https://appsecpodcast.com/topics/devsecops/)

[Audio](https://www.buzzsprout.com/1730684/episodes/8122648-omer-levi-hevroni-k8s-can-keep-a-secret.mp3)

## Show notes

Putting an application in Kubernetes does not solve the problem of getting secrets to it safely. Omer Levi Hevroni, a developer and security champion, explains the risks that led his team to create Kamus. He starts with Kubernetes basics and the broader responsibilities of developers who own applications through deployment and operation. The discussion then follows secrets from source control into the cluster, comparing native mechanisms with encryption and external key-management options. Omer walks through the Kamus workflow, including how an encrypted secret is tied to the application allowed to decrypt it. He also discusses the threats the design addresses and the value of documenting assumptions. This archive conversation captures a practical approach to secrets management and its tradeoffs.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
Security Journey provides application security education for developers and everyone in the software development lifecycle.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Omer Levi Hevroni:  
→ [Omer Levi Hevroni on GitHub](https://github.com/omerlh)

Mentioned in this episode:  
→ [Kamus — historical project repository](https://github.com/Soluto/kamus)  
→ [Kubernetes](https://kubernetes.io/)  
→ [Azure Key Vault](https://azure.microsoft.com/en-us/products/key-vault)

Chapters:  
00:00 Can Kubernetes keep a secret?  
02:38 From developer to security champion  
04:16 Learning to code and owning applications  
06:32 Kubernetes basics  
08:44 The challenge of handling application secrets  
10:34 Developers responsible for the whole lifecycle  
11:19 Keeping secrets usable and controlled  
14:18 Secrets accidentally committed to source code  
17:09 Options for secrets in Kubernetes  
18:58 Recovery and consuming secrets in application code  
21:08 Encryption and external key management  
24:22 Why the team built Kamus  
26:42 The Kamus encryption and decryption workflow  
28:21 Where encrypted secrets and keys live  
31:43 Threats addressed by the design  
34:22 Documentation, threat model, and further resources

## Transcript

*5,795 words · assemblyai*

**0:00 Chris Romeo:** Hey folks, this is Chris Romeo, CEO of Security Journey and co-host of the Application Security Podcast. On this episode of the podcast, Robert and I are joined by Omer Levi-Hevrani. Omer has written extensively on the topic of Kubernetes and secrets, and he's a self-proclaimed super dev. He's the author of a tool for secrets management in Kubernetes called Camus. Camus is open source, GitOps, Zero Trust Secrets Encryption and Decryption solution for Kubernetes applications. KMS enables users to easily encrypt secrets that can be decrypted only by the application running on Kubernetes. The encryption is done using strong encryption providers. They currently support Azure Key Vault, Google Cloud KMS, and AES. Kubernetes, can it keep a secret? I want to take a moment to introduce you to Security Journey. At Security Journey, we believe security is every developer's job. We work with our to help them build long-term sustainable security culture amongst all their developers. Our approach is to provide security education that is first conversational. We don't do lectures. Instead, we let the experts talk about what's important in security. The modules are quick, 10 to 20 minutes in length. They're deep. We've got some of the best security minds in the industry working to provide this content. And last but not least, fun. Training doesn't have to be boring. We make it engaging and fun for the developers. Visit www.security securityjourney.com to sign up for a free trial of The Security Dojo. The Application Security Podcast. Here we go. Hey folks, welcome to this episode of the Application Security Podcast. On this episode, we're going to take a dive into the world of Kubernetes, which this is the first time we've talked about this topic on the AppSec Podcast. And we're joined today by Omer, who is someone who knows quite a bit about Kubernetes and specifically about secrets management in Kubernetes, but we'll get into all that. But Omer, before we dive into the the, the world of Kubernetes, we got to know, how did you get your start in security? What's your security origin story?

**2:38 Omer Levi Hevroni:** So actually, I started as a developer, and today I'm working at Celuto, which is an R&D center for Asurion in Tel Aviv. And I started working there like 4 years ago as a software developer. And Asurion is a really big company. They have like offices all around the world. And at some point, they realized that They can't just control like everything from one place. The security team just don't scale. So they start a Maven program, or other call it security champions. And my manager offered me to join it because I already touched like security-related tasks, for example, authentication. And I got this opportunity with both hands. And I learned a lot in the last 3 years and it was amazing journey.

**3:24 Chris Romeo:** Amazing journey.

**3:26 Omer Levi Hevroni:** into all this AppSec world, and this is how I ended up here today.

**3:32 Chris Romeo:** So would you say you're a graduate of a Security Champion style program then? So when you first started, you were a developer, you were not carrying a security title when you first got started?

**3:44 Omer Levi Hevroni:** Yeah, and it was a really amazing journey and included a lot of like in-company training. We did like a full week of a pentest course, a SANS pentest course, and we have like a close group of all the Mavens from all around the world. And of course, a lot of OWASP materials like the local Israeli chapter, events, projects. I'm highly involved with OWASP, so it helped me a lot and it's also helped others.

**4:16 Chris Romeo:** Yeah, that's great. And it's just, I guess, for our listeners out there that are thinking about starting a Security Champion program, if you don't have one already, or deciding whether you want to invest in one, here's a case study for somebody who transformed themselves from a developer into a security person as a result of their interaction with a Security Maven or Security Champion program. So, that's very cool to hear that story. Omer, I also read in your bio that you started programming at a very early age. How old were you when you started coding?

**4:49 Omer Levi Hevroni:** I think it was 4th grade. Like, this is how it's called in Israel. I think it's 4th grade outside the States. I was like 10. My dad is also an autoprogrammer. It's not his official job. And he also always liked to like build computers at his home, so we always had all these kinds of parts all over the house and like building things. So he decided it's a good idea to teach me BASIC. It was horrible. Yeah, but it was enough to get me started. And I spent a lot of time trying to install Linux and all that fun.

**5:32 Chris Romeo:** Wait, now you didn't install Linux in the 4th grade, did you?

**5:36 Omer Levi Hevroni:** No, no, no. I think in 4th grade I still had Windows, but really soon I understood that Windows costs money, so I moved to Linux.

**5:44 Chris Romeo:** Yeah, I was going to be super impressed if you were like a Linux-running 4th grader, which that would have been pretty cool. But still neat that you started your journey into the world of development as a 4th grader learning the BASIC programming language. But before you finish school, what's the code that you're most proud of?

**6:08 Omer Levi Hevroni:** I did like a project at school that was horrible. I tried to compute polynomials in assembler, and you need to do like multiplies and all that, and power calculation and all that fun in assembler of really big numbers. So we need to do it in string, and it was like horrible, but kind of work.

**6:32 Chris Romeo:** Yeah, that's what I mean. You probably got some good experience, and there's not a lot of people that can actually legitimately say that they program in assembly. And can, you know, a lot of people can break things apart, but they can't actually write code from that perspective. So that's pretty cool. All right, so the topic for today is Kubernetes and secrets management. But I want to start by, Omer, asking you to give us a working definition of Kubernetes, because I don't know that everybody in our audience knows what Kubernetes is. I have a feeling most people do, but just in case somebody doesn't know what Kubernetes is, what is Kubernetes and what do I do with it?

**7:13 Omer Levi Hevroni:** Okay, so in high-level overview, Kubernetes is an orchestrator. Orchestrator is something that runs containers in production. Containers is the cool thing that everyone is using for the last 3 years, I think, to run things in production. I'm going to say something everyone is going to shout at me about saying it, but you can think about containers like VM. At least in a very abstract way, and no, containers is not VM or that, I know. But if you don't know what container is, you can think about container like a VM. It's a thing that runs your code. And Kubernetes lets you run this thing easily, relatively easily in production. And so it's a really sophisticated system for running this container, making sure they're alive, running multiple versions of the same container, high availability, storage, all the fun things that you need for a production application.

**8:07 Chris Romeo:** Okay, and so Kubernetes allows you to run across multiple nodes, so you can take your container and you can scale it to large internet-sized proportions, right?

**8:20 Omer Levi Hevroni:** Yeah, you can do all that, and of course it also takes care of everything else a container needs in production. Again, storage, memory, CPU, so you can define how much you need. It's a really mature project, and this is why a lot of people are using it. There were a few orchestrators before Kubernetes, but Kubernetes is the leading one these days.

**8:44 Chris Romeo:** Okay, so now that we have an idea what Kubernetes is, I— from what I understand, one of the bigger challenges that we face is— and this is bigger than just Kubernetes, this is almost across dealing with internet-based applications and web applications— is how do you handle the secrets? So let's start by just defining the problem. What is the problem that we that we have when trying to store and keep track of API keys and passwords and all that type of stuff in a container type of system.

**9:21 Omer Levi Hevroni:** So, I want to make the problem even more interesting, and I want to talk really quick about how we work at Zaloto. And so, the way we work at Zaloto, I like to call it super devs— developers who have full responsibility for the application. So, they're responsible, of course, to write the code, but the developer is also the one who is responsible for deploying it to production and deploying everything and also responsible for monitoring. So, the person who will wake up in the middle of the night, the one who carries the pager, is the developer itself. Now, when you add that to secret management, things get a lot more complicated because suddenly you have a lot of people. It is a very decentralized organization, and you need to make sure that all these people can create secrets, which are very, very, very sensitive type of information, and can do it securely without compromising the risk, without making— without compromising the secrets. And this is what makes it such a unique challenge and interesting. And of course, you also need to make it simple because you have a lot of developers and you need to make it decentralized.

**10:34 Chris Romeo:** So, a super dev. I've never heard the term a super dev. So, a super dev then, so you own whatever you're working on, you own from the kind of requirements and definition and design all the way through till end of life?

**10:55 Omer Levi Hevroni:** Yes. And of course, security, but we already passed that. Well, I hope. And so yeah, it's like a term I invented. We had an internal discussion trying to define how to call it. It was a pretty funny Slack discussion. People throw all kinds of ideas, but I didn't like any of them. So I go with super devs. It's not perfect, but it's something.

**11:19 Chris Romeo:** Yeah, no, it makes sense. Okay, so we've got this idea of super devs. They're fully responsible for all the code. So you're doing dev plus monitoring. monitoring plus security management. And so how did— how do secrets kind of play into that? What are— what is the real challenge that you face as a super dev trying to deal with secrets?

**11:43 Omer Levi Hevroni:** So, um, the challenge is combined from one, making sure we have a good and robust solution that devs can use it, and because, you know, they need something easy to use, otherwise they will spend all day trying to do things, and we don't want to do that. We want to have super devs who have time to write code. And second, we want to make sure that they have a secure solution. Otherwise, we compromise on security. And again, secrets are a really sensitive part of information.

**12:14 Chris Romeo:** Yeah. So, give us some examples of what developers are using secrets for inside of the applications that they write.

**12:21 Omer Levi Hevroni:** So, actually, almost anything you do, it's some point will require secrets. For example, you might need to call a third-party API. This third-party API is, I hope, authenticated. If it's not authenticated, it's a different problem. But if it's authenticated, you will have a certificate, API key, basic authentication, whatever. You might need to call a database. You might need to call another microservice. And this microservice is also authenticated.

**12:50 Robert Hurlbut:** Yeah.

**12:51 Omer Levi Hevroni:** And so these are usually the kind of secrets you see: API keys, tokens, certificates, secrets, whatever. This is usually the things, connection strings.

**13:03 Chris Romeo:** And so the point is that you need a— so the problem we're trying to solve is you need a secure way to store that information and make it available to your application without pasting it or without placing it inside of your code or your configuration files that are stored in your repository, your source repository.

**13:25 Omer Levi Hevroni:** I even phrase it as you want to make sure that only your application can have it and no one else beside your application. And you need to make sure that you pass the secret only to Kubernetes, and so only your application will have it but no one else. And you make— you need to do it easily so your team members can rotate it and all that.

**13:45 Chris Romeo:** Yeah.

**13:45 Omer Levi Hevroni:** And this is why we so much love GitOps. GitOps is the idea of doing everything via Git, um, or the idea of a configuration as code, infrastructure as code. So you have a code that describes how your production looks like, and this makes it very easy for super devs to work because they have code that describes everything, and it gives us audit and review and all that. But what you do with secrets, how you put secrets there.

**14:15 Robert Hurlbut:** Yeah.

**14:15 Omer Levi Hevroni:** This is the end side of the challenge.

**14:18 Chris Romeo:** Yeah, yeah. So Robert, based on your experience, how often do you see secrets misplaced in source code?

**14:25 Robert Hurlbut:** Oh yeah, quite often. I mean, you're trying to figure out where do I— how do I get it? When do I get it? Also, I see often, at least in the past, I've seen developers because it's a dev system. They're pretty lax on securing credentials and sometimes that will transfer over to production as well just because the solution works. So we continue to use this in production. I've even seen one system, I won't name names, where it was funny, they actually were more secure in development than they were in production. By the time it got to production, everything was wide open, all the passwords, everything was completely wide open. Mainly because it was easier for the scripting, the scripts that were used to deploy, to manage everything. It didn't know how to deal with any of the encrypted passwords and things like that. So that was a surprise.

**15:22 Chris Romeo:** Yeah, that sounds like not necessarily the best overall architecture play.

**15:28 Omer Levi Hevroni:** Yeah, but it might not be just the architecture, but it's also responsibility of DevOps or DevSecOps to provide easy and good tools, because if you don't have good tools, you will use what you have. And this is the challenge with super devs. You need to help them, I think, at least.

**15:49 Chris Romeo:** Yeah, I mean, I think that applies to every dev. I mean, every— and, and, Omer, you bring up a great point there about the fact that when we think about DevSecOps and, and how security should be integrated into DevOps, We want the devs to have the easy— devs need an easy solution, but also a secure solution. And shame on the security team if they don't provide the developers with a cookie-cutter approach to how they do secrets management that's enforced and checked and all that other type of good stuff.

**16:22 Omer Levi Hevroni:** And this is exactly where we was like 8 months ago when we started to— we had We spent a lot of effort on making it easy to deploy things to Kubernetes, but we had one small problem. We didn't solve the secret problem. So, everything was perfect and simple besides secrets.

**16:41 Chris Romeo:** Okay. So, 8 months ago, you began the journey of moving everything over to Kubernetes?

**16:48 Omer Levi Hevroni:** Yes. And we spent a lot of effort on making it easy because if you ever worked with Kubernetes, you probably know it's not easy. And so I would— I'm part of the DevOps team, and we as DevOps spend a lot of effort of trying to make it easy beside this problem of secrets, because it was a really hard one.

**17:09 Chris Romeo:** Okay, so we've identified kind of— we've defined Kubernetes, we've defined some of the challenges that the super devs and even regular devs are going to have in working with secrets management. What are some of the solutions then from a Kubernetes perspective, Omer, that we have to be able to help us solve this problem?

**17:30 Omer Levi Hevroni:** So, the most naive approach is using Kubernetes native object. Kubernetes has a native object called secrets, which like the name implied, like the name implied, should be supposed to store secrets. And the Kubernetes team spent a lot of effort to making this object secure and really good to use and There are a lot of mitigations to make sure that only your application will have that. But there is one tiny problem. If you want to write a manifest, a file that describes the secret, a JSON or YAML, the secret is encoded as Base64 in this file, which is not that good. It's not something you can commit to source control, and they even specify that in the source control— in the documentation.

**18:16 Robert Hurlbut:** Yeah.

**18:17 Omer Levi Hevroni:** So, using a Kubernetes secret is not easy with GitOps, and at some point we ended up— and I'm sure a lot of other companies are in the same situation— where you create the secrets directly in Kubernetes UI or with kubectl, but you don't store the secret anywhere else because it will be insecure. So, the secrets are stored directly in Kubernetes, and each time you need to modify a secret, you need to download it first, modify it, and then return it. And this is something I see also other companies doing because this is like the most naive approach using Kubernetes.

**18:53 Chris Romeo:** That sounds like it would be a bit of a management nightmare though.

**18:57 Omer Levi Hevroni:** Yeah.

**18:58 Chris Romeo:** If you have to constantly update any secrets directly through Kubernetes and you can't store them anywhere else. I mean, what happens if your cluster— I mean, granted, you probably have a number of high availability features and things, but what if for some reason your cluster was knocked out? you basically have to reset all of those secrets and everything else?

**19:17 Omer Levi Hevroni:** Yeah, and it's not just that. It's even worse because, um, you need— we need to talk about how you can consume these secrets in your code, right? So there are 2 ways to consume a secret into a code. One way is using environment variable, which some security experts will argue is not that good idea.

**19:35 Chris Romeo:** Did you say— was that an environment variable?

**19:37 Omer Levi Hevroni:** Yeah, so it's like Yeah, there is an open discussion if environment variable is a good or bad idea, but we can put that aside. So if you choose to go with environment variable, it's relatively simple. If you choose to go with a file, which is the alternative to environment variable, you probably end up having one JSON file. So I imagine you have a JSON file, and in this JSON file you have multiple keys like token A and API key B and the values.

**20:07 Chris Romeo:** Yes.

**20:08 Omer Levi Hevroni:** And then you take this file, do a Base64 encode because secrets are Base64 encoded, and you take this encoded value and put it in a secret. Now to update this secret, just imagine the flow. You need to get the secret from Kubernetes, decode the Base64, get the original JSON, modify the JSON, and encode it back and put it in Kubernetes.

**20:29 Robert Hurlbut:** So yeah, that sounds—

**20:30 Chris Romeo:** that doesn't sound very easy. That sounds pretty difficult from an overall day-to-day management perspective. Is that the case?

**20:37 Omer Levi Hevroni:** Definitely the case, and this is why we understood— we started with a very naive approach saying we can use Kubernetes secrets, but very soon we get to that situation where modifying secrets is hard and cumbersome. And we start to look at alternatives, and there are good alternatives that allow you to create encrypted secrets, which is basically native Kubernetes secret, just encrypted. So Theoretically, you can put them in Git and it solves some of the problems.

**21:08 Chris Romeo:** Now, is that a native— is that when you're talking about this encrypted secret, are we still within the native world of Kubernetes or have we branched out to something else?

**21:18 Omer Levi Hevroni:** No, it's like hybrid because these are external tools to Kubernetes. There are 2 popular tools, Helm Secret and cert-secret. But I call them hybrid because they ended up creating a native Kubernetes secret. So in your source code you have an object called encrypted secret, but in the end you will have on the cluster regular secrets object. So this is why I called it a hybrid approach.

**21:45 Chris Romeo:** OK, now you mentioned too that you talked about sealed secrets and Helm secrets.

**21:52 Omer Levi Hevroni:** Yes.

**21:52 Chris Romeo:** Tell me, I don't know, I don't know the difference between those, so maybe give me a little explanation of what a sealed secret is versus a Helm secret.

**22:00 Omer Levi Hevroni:** So both of the solutions are pretty similar. They both give you the same thing, a way to encrypt secrets. The difference between them is how you encrypt the secrets and when you decrypt them. Seed secret lets you decrypt the secrets in a very later stage, just when the secret is deployed to the cluster. So through the CI/CD pipeline, you have a seed secret, and then when it gets to Kubernetes, magic happens and a regular Kubernetes secret created from the encrypted secret, a decrypted version of it. So this is sealed secret. Helm secrets work a bit differently. It decrypts the encrypted secret at the deployment step outside Kubernetes, and then it creates a regular Kubernetes secret. So the difference between them is when the decryption is done, and it's also in how strong the encryption is. Sealed secrets work with one keeper in the cluster, which is not that secure. HashiCat is a bit better. It's using Mozilla SOPS. Mozilla SOPS is open source by Mozilla that has support for external KMS, things like AWS KMS or GCP KMS. So you have HSM support.

**23:09 Chris Romeo:** And KMS is Key Management Services, right?

**23:12 Omer Levi Hevroni:** Yeah, it's a product by AWS or Google cloud provider, AWS, Amazon, or Google. And it's both like REST API for key management that has HSM support. So HAM is a lot more secure than sealed secrets. Sealed secrets, again, you have one keeper in the cluster, not secure at all.

**23:33 Chris Romeo:** And OneKeeper, is that like a— is that the password manager OneKeeper?

**23:36 Omer Levi Hevroni:** No, Keepel, RSA Keepel.

**23:39 Chris Romeo:** RSA.

**23:40 Robert Hurlbut:** Oh, okay.

**23:41 Omer Levi Hevroni:** All right. I think it's RSA. I think it's RSA.

**23:44 Chris Romeo:** Got it.

**23:45 Omer Levi Hevroni:** Okay.

**23:46 Chris Romeo:** And so, Sealed Secrets and Helm Secrets, these are not built into Kubernetes. These are external things that were created after, that were created by outside non-Kubernetes people to solve the problem.

**23:59 Omer Levi Hevroni:** Yes. But again, these tools are imperfect. Sealed Secrets, I don't like it from a security point of view. And Helm Secrets enforce you, require you to use Helm. Helm is package management for Kubernetes. And so if you don't use Helm, or if you sometimes want to stop using Helm, you will need to find an alternative. And this is why I don't like these 2 tools.

**24:22 Chris Romeo:** All right, so we know we don't like built-in secrets, using the built-in secrets from Kubernetes, because they're hard to manage. And we've talked about why sealed and Helm secrets are not necessarily the best overall solution. Is there any answer? Like, what is— where did you land in your deployment?

**24:43 Omer Levi Hevroni:** So, we did another quick look on HashiCorp Vault, which I'm sure a lot of the audience are familiar with. And if you're not familiar with HashiCorp Vault, it's basically a key-value store that is secure. And it's really interesting, but it's not GitOps, which is the main reason we decided not to use it. And at that point, at least I was very confused because like I looked all over the place and there was no existing solution that I could feel could work with our workflow. Again, the super devs workflow. And it was a really hard situation to be in because I know we don't have a good solution for secrets, but there was no good solution that I know exists out there. I was a bit disappointed and didn't know exactly what to do. Then one day while working on a totally different task, I encountered Travis encrypted secrets solution. If you never worked with Travis, it's a really neat solution. It basically lets you encrypt a secret for a specific Travis repository that exposes RSA public key for every repository. So you can encrypt the secret for a specific repository, and then only this repository builds can decrypt it. And when I saw it, I was really impressed because if you think about it, it's really simple but really secure. And I started to look on how I can do the same thing to Kubernetes, and this is how Kamos born. And Kamos is an open source I built at Zaluto, which basically lets you do the same thing as Travis encrypted secrets.

**26:25 Robert Hurlbut:** Okay.

**26:25 Omer Levi Hevroni:** It lets you encrypt the secrets for a specific application on Kubernetes, and only this application can decrypt it. So if you go back to the super devs, the way Kamas works is that anyone can encrypt a secret, but once the secret is encrypted, no one can decrypt it.

**26:42 Chris Romeo:** If I have— let's say I have a secret, I'm writing a new piece of code. What are the steps that I go through as I interact with Kamas?

**26:51 Omer Levi Hevroni:** You use Kamus. Kamus has a CLI which basically interacts with Kamus API. So you use the CLI and you need to give the API 3 things. You need to give— you need to tell the API what you want to encrypt. You need to tell the API to which application you want to encrypt and in which Kubernetes namespace this application lives. The way Kamus identifies application is using a Kubernetes object called service account. Service account is an object, Kubernetes object, that provides identity for a pod.

**27:19 Chris Romeo:** Okay.

**27:20 Omer Levi Hevroni:** It has a token and application can use this token to interact with Kubernetes API or with any other API, and you can give permissions to this service account. It's a really neat feature. So you need to give Kamuz these 3 parameters. Again, the secret, the application or the service account used by the application, and the namespace. Kamuz encrypts the secret for you and returns it for you. Now the developers can put the secret in Git Twitter, Facebook, whatever you're using because it's strong encryption, so put it wherever you want. Then the regular CI/CD magic happens. Your code gets into Kubernetes and when your application starts, before it starts, it needs to interact with Kamos, provide the service account token and the encrypted secret. Kamos performs authorization and if the secret indeed encrypted for this application, the secret is decrypted and the application gets the decrypted value back, and then it can use it to perform its application.

**28:21 Chris Romeo:** Okay. So where are my secrets stored then, even when they're in that encrypted format? Is SKMS something that's using— is it providing or storing my— just in files, or is this like a cloud service? What am I interacting with? Where does this API actually live that I'm interacting with?

**28:44 Omer Levi Hevroni:** So, Kamos does not handle the part of storing the secret. You can store the secret wherever you put other manifest files. So ideally you can store it in the same place where you store all the other files used to deploy your application to production. So it could be Git, it could be file server, it could be blob in the cloud, whatever already works for your company. Kamus is responsible for encrypting and decrypting the secret, and it's an API running inside Kubernetes, and it needs to run inside Kubernetes so it will be able to authorize applications based on the service account token.

**29:22 Chris Romeo:** Okay, got it. So then, so then it keeps its keys inside of Kubernetes?

**29:29 Omer Levi Hevroni:** No, Kamus has a pluggable encryption system. So currently Kamus supports in-memory AES key, which is the most insecure installation of Kamus, but it has support for Azure Key Vault and GCP KMS. Both are very strong solutions, provide HSM protections, and Azure also provides IP filtering. And we hope in the future to add support for AWS KMS and maybe other solutions like native HSM or something. It's really pluggable and it's really easy to add new encryption systems to Komoot.

**30:12 Robert Hurlbut:** So it becomes, it sort of becomes an interface to these other, I know Azure Key Vault and other solutions like that, so it sounds like it will allow you to use those other mechanisms as well pretty seamlessly.

**30:27 Omer Levi Hevroni:** Yes, the idea is to make it easy for application to talk with this storing encryption solution and solving the— Kamus solves the layer of authentication and authorization. This is what Kamus solves. It does not handle encryption itself.

**30:43 Chris Romeo:** Okay, so Kamus is really the middleware between Kubernetes and Azure Key Vault or Google Cloud KMS, and I see you're going to do in the future AWS KMS. So, it becomes like that middleware layer that just handles the connection between Kubernetes and the storage and the secure secret storage.

**31:07 Robert Hurlbut:** Yes.

**31:09 Omer Levi Hevroni:** And we do hope like in the very, very future, I see Cammus as something that can be relevant also outside of Kubernetes. Secret is not a problem that is unique on Kubernetes. And ideally, at some point, we can say, for example, I have a Lambda running in AWS and I use the identity of the Lambda to authenticate to Kamus and get my secrets for Kamus. But this is like very far in the roadmap, but I definitely see it's going there because it can solve a lot bigger problem than just Kubernetes secrets.

**31:43 Robert Hurlbut:** Okay, so what are some of the things that it can help you with in terms of threats that the solution will help you with?

**31:52 Omer Levi Hevroni:** So I think the main reason Kamos is superior to other solutions is the very simple permission model. The fact that no one can decrypt the secret, you have one-way encryption, makes things a lot more simpler because now the attack surface of where the secret lives is only in the application itself and of course on the computer where the developer generated it. But the attack surface is a lot smaller than in other models where you have a role that lets you read the secret or decrypt it or something like that. And I think this is the main security feature of Cammus. And once you understand that and you understand that in Cammus there is no permission management because there is one flat permission management, it improves the security significantly. And we put a lot of other mitigations, for example, We make the CLI secured. It's enforced HTTPS by default and it's allowed to support certificate pinning. For example, we make sure that the API has security tests covering all the code entering into it. We have 2 different APIs actually of Commons, one handling encryption and one handling decryption. So once you have 2 different microservices, you can expose only one of them to the internet. The one that handles the caching does not need to be exposed to the internet. And all of these things reduce the attack surface significantly. And to make it clear to everyone, we created a public threat model. We use a format called Jerkin. I took the inspiration from OWASP Cloud Security Project.

**33:39 Robert Hurlbut:** Okay.

**33:40 Omer Levi Hevroni:** It's a nice OS project that tries to document different threats and mitigations of cloud providers. And so they use JARCAN format and we use it also. So you can check it out, it's on Komoot repo, and you can see there the different threats and mitigations we think about when we did the threat model to Komoot. And so all those things I talked about and a lot more is covered there.

**34:04 Robert Hurlbut:** Okay.

**34:07 Omer Levi Hevroni:** So this is from a security point of view, and of course we have SecurityMD to make it easy to perform responsible disclosure because I'm sure we didn't cover everything and there are things we missed. So if you think about something like that, in the repository it specifies how to report it.

**34:22 Robert Hurlbut:** Okay, good. So let's say we want to follow up and, and get some more information. Where, where should we go to, to get some more information on Kubernetes and secrets and Kuma and other kinds of solutions?

**34:37 Omer Levi Hevroni:** So I wrote a blog post on the Celuto engineering blog. The blog post covers pretty much the things I talked about and a lot more. And it also covers how you can start with Kuma. It's really easy. It's one Helm command. We released a Helm chart to make it easy to install Kuma. So you can read it, go over it, and then get into the repo. We have detailed installation details explaining how to install Kuma securely. And we have a Slack so you can come talk with us on common Slack. And if you have any question, come ask. We will be more than happy to answer it and help other people use it.

**35:14 Robert Hurlbut:** OK, and you have a link to that threat model you were talking about as well in the blog?

**35:18 Omer Levi Hevroni:** Yeah, and put it— we will— I will send it to you so you can provide it to the audience.

**35:25 Robert Hurlbut:** OK, yeah, we'll make sure it's available.

**35:28 Omer Levi Hevroni:** OK, great.

**35:30 Robert Hurlbut:** All right, any, um, any last thoughts? Uh, thanks, Amar, for, for joining us today, but any last thoughts for our listeners?

**35:36 Omer Levi Hevroni:** Um, I, I think the most important point is what we talked about at the beginning, thinking about how we as security engineers can make the world of— the work of developers easy and help them choose the right and most secure solution. And this was the main motivation, motivation behind Camus. Making it a lot harder to make mistakes and expose secrets to the world. Yeah.

**36:09 Robert Hurlbut:** Said very well. Thank you very much for joining us.

**36:12 Omer Levi Hevroni:** Thank you very much for having me. It was a pleasure. I really enjoyed it. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.

---

Source: https://appsecpodcast.com/omer-levi-hevroni-k8s-can-keep-a-secret/