Omer Levi Hevroni — K8s can keep a secret?

0:00Chris RomeoHey folks, this is Chris Romeo, CEO of Security Journey and co-host of the Application Security Podcast. On this episode of the podcast, Robert and I are joined by Omer Levi-Hevrani. Omer has written extensively on the topic of Kubernetes and secrets, and he's a self-proclaimed super dev. He's the author of a tool for secrets management in Kubernetes called Camus. Camus is open source, GitOps, Zero Trust Secrets Encryption and Decryption solution for Kubernetes applications. KMS enables users to easily encrypt secrets that can be decrypted only by the application running on Kubernetes. The encryption is done using strong encryption providers. They currently support Azure Key Vault, Google Cloud KMS, and AES. Kubernetes, can it keep a secret? I want to take a moment to introduce you to Security Journey. At Security Journey, we believe security is every developer's job. We work with our to help them build long-term sustainable security culture amongst all their developers. Our approach is to provide security education that is first conversational. We don't do lectures. Instead, we let the experts talk about what's important in security. The modules are quick, 10 to 20 minutes in length. They're deep. We've got some of the best security minds in the industry working to provide this content. And last but not least, fun. Training doesn't have to be boring. We make it engaging and fun for the developers. Visit www.security securityjourney.com to sign up for a free trial of The Security Dojo. The Application Security Podcast. Here we go. Hey folks, welcome to this episode of the Application Security Podcast. On this episode, we're going to take a dive into the world of Kubernetes, which this is the first time we've talked about this topic on the AppSec Podcast. And we're joined today by Omer, who is someone who knows quite a bit about Kubernetes and specifically about secrets management in Kubernetes, but we'll get into all that. But Omer, before we dive into the the, the world of Kubernetes, we got to know, how did you get your start in security? What's your security origin story?

2:38Omer Levi HevroniSo actually, I started as a developer, and today I'm working at Celuto, which is an R&D center for Asurion in Tel Aviv. And I started working there like 4 years ago as a software developer. And Asurion is a really big company. They have like offices all around the world. And at some point, they realized that They can't just control like everything from one place. The security team just don't scale. So they start a Maven program, or other call it security champions. And my manager offered me to join it because I already touched like security-related tasks, for example, authentication. And I got this opportunity with both hands. And I learned a lot in the last 3 years and it was amazing journey.

3:24Chris RomeoAmazing journey.

3:26Omer Levi Hevroniinto all this AppSec world, and this is how I ended up here today.

3:32Chris RomeoSo would you say you're a graduate of a Security Champion style program then? So when you first started, you were a developer, you were not carrying a security title when you first got started?

3:44Omer Levi HevroniYeah, and it was a really amazing journey and included a lot of like in-company training. We did like a full week of a pentest course, a SANS pentest course, and we have like a close group of all the Mavens from all around the world. And of course, a lot of OWASP materials like the local Israeli chapter, events, projects. I'm highly involved with OWASP, so it helped me a lot and it's also helped others.

4:16Chris RomeoYeah, that's great. And it's just, I guess, for our listeners out there that are thinking about starting a Security Champion program, if you don't have one already, or deciding whether you want to invest in one, here's a case study for somebody who transformed themselves from a developer into a security person as a result of their interaction with a Security Maven or Security Champion program. So, that's very cool to hear that story. Omer, I also read in your bio that you started programming at a very early age. How old were you when you started coding?

4:49Omer Levi HevroniI think it was 4th grade. Like, this is how it's called in Israel. I think it's 4th grade outside the States. I was like 10. My dad is also an autoprogrammer. It's not his official job. And he also always liked to like build computers at his home, so we always had all these kinds of parts all over the house and like building things. So he decided it's a good idea to teach me BASIC. It was horrible. Yeah, but it was enough to get me started. And I spent a lot of time trying to install Linux and all that fun.

5:32Chris RomeoWait, now you didn't install Linux in the 4th grade, did you?

5:36Omer Levi HevroniNo, no, no. I think in 4th grade I still had Windows, but really soon I understood that Windows costs money, so I moved to Linux.

5:44Chris RomeoYeah, I was going to be super impressed if you were like a Linux-running 4th grader, which that would have been pretty cool. But still neat that you started your journey into the world of development as a 4th grader learning the BASIC programming language. But before you finish school, what's the code that you're most proud of?

6:08Omer Levi HevroniI did like a project at school that was horrible. I tried to compute polynomials in assembler, and you need to do like multiplies and all that, and power calculation and all that fun in assembler of really big numbers. So we need to do it in string, and it was like horrible, but kind of work.

6:32Chris RomeoYeah, that's what I mean. You probably got some good experience, and there's not a lot of people that can actually legitimately say that they program in assembly. And can, you know, a lot of people can break things apart, but they can't actually write code from that perspective. So that's pretty cool. All right, so the topic for today is Kubernetes and secrets management. But I want to start by, Omer, asking you to give us a working definition of Kubernetes, because I don't know that everybody in our audience knows what Kubernetes is. I have a feeling most people do, but just in case somebody doesn't know what Kubernetes is, what is Kubernetes and what do I do with it?

7:13Omer Levi HevroniOkay, so in high-level overview, Kubernetes is an orchestrator. Orchestrator is something that runs containers in production. Containers is the cool thing that everyone is using for the last 3 years, I think, to run things in production. I'm going to say something everyone is going to shout at me about saying it, but you can think about containers like VM. At least in a very abstract way, and no, containers is not VM or that, I know. But if you don't know what container is, you can think about container like a VM. It's a thing that runs your code. And Kubernetes lets you run this thing easily, relatively easily in production. And so it's a really sophisticated system for running this container, making sure they're alive, running multiple versions of the same container, high availability, storage, all the fun things that you need for a production application.

8:07Chris RomeoOkay, and so Kubernetes allows you to run across multiple nodes, so you can take your container and you can scale it to large internet-sized proportions, right?

8:20Omer Levi HevroniYeah, you can do all that, and of course it also takes care of everything else a container needs in production. Again, storage, memory, CPU, so you can define how much you need. It's a really mature project, and this is why a lot of people are using it. There were a few orchestrators before Kubernetes, but Kubernetes is the leading one these days.

8:44Chris RomeoOkay, so now that we have an idea what Kubernetes is, I— from what I understand, one of the bigger challenges that we face is— and this is bigger than just Kubernetes, this is almost across dealing with internet-based applications and web applications— is how do you handle the secrets? So let's start by just defining the problem. What is the problem that we that we have when trying to store and keep track of API keys and passwords and all that type of stuff in a container type of system.

9:21Omer Levi HevroniSo, I want to make the problem even more interesting, and I want to talk really quick about how we work at Zaloto. And so, the way we work at Zaloto, I like to call it super devs— developers who have full responsibility for the application. So, they're responsible, of course, to write the code, but the developer is also the one who is responsible for deploying it to production and deploying everything and also responsible for monitoring. So, the person who will wake up in the middle of the night, the one who carries the pager, is the developer itself. Now, when you add that to secret management, things get a lot more complicated because suddenly you have a lot of people. It is a very decentralized organization, and you need to make sure that all these people can create secrets, which are very, very, very sensitive type of information, and can do it securely without compromising the risk, without making— without compromising the secrets. And this is what makes it such a unique challenge and interesting. And of course, you also need to make it simple because you have a lot of developers and you need to make it decentralized.

10:34Chris RomeoSo, a super dev. I've never heard the term a super dev. So, a super dev then, so you own whatever you're working on, you own from the kind of requirements and definition and design all the way through till end of life?

10:55Omer Levi HevroniYes. And of course, security, but we already passed that. Well, I hope. And so yeah, it's like a term I invented. We had an internal discussion trying to define how to call it. It was a pretty funny Slack discussion. People throw all kinds of ideas, but I didn't like any of them. So I go with super devs. It's not perfect, but it's something.

11:19Chris RomeoYeah, no, it makes sense. Okay, so we've got this idea of super devs. They're fully responsible for all the code. So you're doing dev plus monitoring. monitoring plus security management. And so how did— how do secrets kind of play into that? What are— what is the real challenge that you face as a super dev trying to deal with secrets?

11:43Omer Levi HevroniSo, um, the challenge is combined from one, making sure we have a good and robust solution that devs can use it, and because, you know, they need something easy to use, otherwise they will spend all day trying to do things, and we don't want to do that. We want to have super devs who have time to write code. And second, we want to make sure that they have a secure solution. Otherwise, we compromise on security. And again, secrets are a really sensitive part of information.

12:14Chris RomeoYeah. So, give us some examples of what developers are using secrets for inside of the applications that they write.

12:21Omer Levi HevroniSo, actually, almost anything you do, it's some point will require secrets. For example, you might need to call a third-party API. This third-party API is, I hope, authenticated. If it's not authenticated, it's a different problem. But if it's authenticated, you will have a certificate, API key, basic authentication, whatever. You might need to call a database. You might need to call another microservice. And this microservice is also authenticated.

12:50Robert HurlbutYeah.

12:51Omer Levi HevroniAnd so these are usually the kind of secrets you see: API keys, tokens, certificates, secrets, whatever. This is usually the things, connection strings.

13:03Chris RomeoAnd so the point is that you need a— so the problem we're trying to solve is you need a secure way to store that information and make it available to your application without pasting it or without placing it inside of your code or your configuration files that are stored in your repository, your source repository.

13:25Omer Levi HevroniI even phrase it as you want to make sure that only your application can have it and no one else beside your application. And you need to make sure that you pass the secret only to Kubernetes, and so only your application will have it but no one else. And you make— you need to do it easily so your team members can rotate it and all that.

13:45Chris RomeoYeah.

13:45Omer Levi HevroniAnd this is why we so much love GitOps. GitOps is the idea of doing everything via Git, um, or the idea of a configuration as code, infrastructure as code. So you have a code that describes how your production looks like, and this makes it very easy for super devs to work because they have code that describes everything, and it gives us audit and review and all that. But what you do with secrets, how you put secrets there.

14:15Robert HurlbutYeah.

14:15Omer Levi HevroniThis is the end side of the challenge.

14:18Chris RomeoYeah, yeah. So Robert, based on your experience, how often do you see secrets misplaced in source code?

14:25Robert HurlbutOh yeah, quite often. I mean, you're trying to figure out where do I— how do I get it? When do I get it? Also, I see often, at least in the past, I've seen developers because it's a dev system. They're pretty lax on securing credentials and sometimes that will transfer over to production as well just because the solution works. So we continue to use this in production. I've even seen one system, I won't name names, where it was funny, they actually were more secure in development than they were in production. By the time it got to production, everything was wide open, all the passwords, everything was completely wide open. Mainly because it was easier for the scripting, the scripts that were used to deploy, to manage everything. It didn't know how to deal with any of the encrypted passwords and things like that. So that was a surprise.

15:22Chris RomeoYeah, that sounds like not necessarily the best overall architecture play.

15:28Omer Levi HevroniYeah, but it might not be just the architecture, but it's also responsibility of DevOps or DevSecOps to provide easy and good tools, because if you don't have good tools, you will use what you have. And this is the challenge with super devs. You need to help them, I think, at least.

15:49Chris RomeoYeah, I mean, I think that applies to every dev. I mean, every— and, and, Omer, you bring up a great point there about the fact that when we think about DevSecOps and, and how security should be integrated into DevOps, We want the devs to have the easy— devs need an easy solution, but also a secure solution. And shame on the security team if they don't provide the developers with a cookie-cutter approach to how they do secrets management that's enforced and checked and all that other type of good stuff.

16:22Omer Levi HevroniAnd this is exactly where we was like 8 months ago when we started to— we had We spent a lot of effort on making it easy to deploy things to Kubernetes, but we had one small problem. We didn't solve the secret problem. So, everything was perfect and simple besides secrets.

16:41Chris RomeoOkay. So, 8 months ago, you began the journey of moving everything over to Kubernetes?

16:48Omer Levi HevroniYes. And we spent a lot of effort on making it easy because if you ever worked with Kubernetes, you probably know it's not easy. And so I would— I'm part of the DevOps team, and we as DevOps spend a lot of effort of trying to make it easy beside this problem of secrets, because it was a really hard one.

17:09Chris RomeoOkay, so we've identified kind of— we've defined Kubernetes, we've defined some of the challenges that the super devs and even regular devs are going to have in working with secrets management. What are some of the solutions then from a Kubernetes perspective, Omer, that we have to be able to help us solve this problem?

17:30Omer Levi HevroniSo, the most naive approach is using Kubernetes native object. Kubernetes has a native object called secrets, which like the name implied, like the name implied, should be supposed to store secrets. And the Kubernetes team spent a lot of effort to making this object secure and really good to use and There are a lot of mitigations to make sure that only your application will have that. But there is one tiny problem. If you want to write a manifest, a file that describes the secret, a JSON or YAML, the secret is encoded as Base64 in this file, which is not that good. It's not something you can commit to source control, and they even specify that in the source control— in the documentation.

18:16Robert HurlbutYeah.

18:17Omer Levi HevroniSo, using a Kubernetes secret is not easy with GitOps, and at some point we ended up— and I'm sure a lot of other companies are in the same situation— where you create the secrets directly in Kubernetes UI or with kubectl, but you don't store the secret anywhere else because it will be insecure. So, the secrets are stored directly in Kubernetes, and each time you need to modify a secret, you need to download it first, modify it, and then return it. And this is something I see also other companies doing because this is like the most naive approach using Kubernetes.

18:53Chris RomeoThat sounds like it would be a bit of a management nightmare though.

18:57Omer Levi HevroniYeah.

18:58Chris RomeoIf you have to constantly update any secrets directly through Kubernetes and you can't store them anywhere else. I mean, what happens if your cluster— I mean, granted, you probably have a number of high availability features and things, but what if for some reason your cluster was knocked out? you basically have to reset all of those secrets and everything else?

19:17Omer Levi HevroniYeah, and it's not just that. It's even worse because, um, you need— we need to talk about how you can consume these secrets in your code, right? So there are 2 ways to consume a secret into a code. One way is using environment variable, which some security experts will argue is not that good idea.

19:35Chris RomeoDid you say— was that an environment variable?

19:37Omer Levi HevroniYeah, so it's like Yeah, there is an open discussion if environment variable is a good or bad idea, but we can put that aside. So if you choose to go with environment variable, it's relatively simple. If you choose to go with a file, which is the alternative to environment variable, you probably end up having one JSON file. So I imagine you have a JSON file, and in this JSON file you have multiple keys like token A and API key B and the values.

20:07Chris RomeoYes.

20:08Omer Levi HevroniAnd then you take this file, do a Base64 encode because secrets are Base64 encoded, and you take this encoded value and put it in a secret. Now to update this secret, just imagine the flow. You need to get the secret from Kubernetes, decode the Base64, get the original JSON, modify the JSON, and encode it back and put it in Kubernetes.

20:29Robert HurlbutSo yeah, that sounds—

20:30Chris Romeothat doesn't sound very easy. That sounds pretty difficult from an overall day-to-day management perspective. Is that the case?

20:37Omer Levi HevroniDefinitely the case, and this is why we understood— we started with a very naive approach saying we can use Kubernetes secrets, but very soon we get to that situation where modifying secrets is hard and cumbersome. And we start to look at alternatives, and there are good alternatives that allow you to create encrypted secrets, which is basically native Kubernetes secret, just encrypted. So Theoretically, you can put them in Git and it solves some of the problems.

21:08Chris RomeoNow, is that a native— is that when you're talking about this encrypted secret, are we still within the native world of Kubernetes or have we branched out to something else?

21:18Omer Levi HevroniNo, it's like hybrid because these are external tools to Kubernetes. There are 2 popular tools, Helm Secret and cert-secret. But I call them hybrid because they ended up creating a native Kubernetes secret. So in your source code you have an object called encrypted secret, but in the end you will have on the cluster regular secrets object. So this is why I called it a hybrid approach.

21:45Chris RomeoOK, now you mentioned too that you talked about sealed secrets and Helm secrets.

21:52Omer Levi HevroniYes.

21:52Chris RomeoTell me, I don't know, I don't know the difference between those, so maybe give me a little explanation of what a sealed secret is versus a Helm secret.

22:00Omer Levi HevroniSo both of the solutions are pretty similar. They both give you the same thing, a way to encrypt secrets. The difference between them is how you encrypt the secrets and when you decrypt them. Seed secret lets you decrypt the secrets in a very later stage, just when the secret is deployed to the cluster. So through the CI/CD pipeline, you have a seed secret, and then when it gets to Kubernetes, magic happens and a regular Kubernetes secret created from the encrypted secret, a decrypted version of it. So this is sealed secret. Helm secrets work a bit differently. It decrypts the encrypted secret at the deployment step outside Kubernetes, and then it creates a regular Kubernetes secret. So the difference between them is when the decryption is done, and it's also in how strong the encryption is. Sealed secrets work with one keeper in the cluster, which is not that secure. HashiCat is a bit better. It's using Mozilla SOPS. Mozilla SOPS is open source by Mozilla that has support for external KMS, things like AWS KMS or GCP KMS. So you have HSM support.

23:09Chris RomeoAnd KMS is Key Management Services, right?

23:12Omer Levi HevroniYeah, it's a product by AWS or Google cloud provider, AWS, Amazon, or Google. And it's both like REST API for key management that has HSM support. So HAM is a lot more secure than sealed secrets. Sealed secrets, again, you have one keeper in the cluster, not secure at all.

23:33Chris RomeoAnd OneKeeper, is that like a— is that the password manager OneKeeper?

23:36Omer Levi HevroniNo, Keepel, RSA Keepel.

23:39Chris RomeoRSA.

23:40Robert HurlbutOh, okay.

23:41Omer Levi HevroniAll right. I think it's RSA. I think it's RSA.

23:44Chris RomeoGot it.

23:45Omer Levi HevroniOkay.

23:46Chris RomeoAnd so, Sealed Secrets and Helm Secrets, these are not built into Kubernetes. These are external things that were created after, that were created by outside non-Kubernetes people to solve the problem.

23:59Omer Levi HevroniYes. But again, these tools are imperfect. Sealed Secrets, I don't like it from a security point of view. And Helm Secrets enforce you, require you to use Helm. Helm is package management for Kubernetes. And so if you don't use Helm, or if you sometimes want to stop using Helm, you will need to find an alternative. And this is why I don't like these 2 tools.

24:22Chris RomeoAll right, so we know we don't like built-in secrets, using the built-in secrets from Kubernetes, because they're hard to manage. And we've talked about why sealed and Helm secrets are not necessarily the best overall solution. Is there any answer? Like, what is— where did you land in your deployment?

24:43Omer Levi HevroniSo, we did another quick look on HashiCorp Vault, which I'm sure a lot of the audience are familiar with. And if you're not familiar with HashiCorp Vault, it's basically a key-value store that is secure. And it's really interesting, but it's not GitOps, which is the main reason we decided not to use it. And at that point, at least I was very confused because like I looked all over the place and there was no existing solution that I could feel could work with our workflow. Again, the super devs workflow. And it was a really hard situation to be in because I know we don't have a good solution for secrets, but there was no good solution that I know exists out there. I was a bit disappointed and didn't know exactly what to do. Then one day while working on a totally different task, I encountered Travis encrypted secrets solution. If you never worked with Travis, it's a really neat solution. It basically lets you encrypt a secret for a specific Travis repository that exposes RSA public key for every repository. So you can encrypt the secret for a specific repository, and then only this repository builds can decrypt it. And when I saw it, I was really impressed because if you think about it, it's really simple but really secure. And I started to look on how I can do the same thing to Kubernetes, and this is how Kamos born. And Kamos is an open source I built at Zaluto, which basically lets you do the same thing as Travis encrypted secrets.

26:25Robert HurlbutOkay.

26:25Omer Levi HevroniIt lets you encrypt the secrets for a specific application on Kubernetes, and only this application can decrypt it. So if you go back to the super devs, the way Kamas works is that anyone can encrypt a secret, but once the secret is encrypted, no one can decrypt it.

26:42Chris RomeoIf I have— let's say I have a secret, I'm writing a new piece of code. What are the steps that I go through as I interact with Kamas?

26:51Omer Levi HevroniYou use Kamus. Kamus has a CLI which basically interacts with Kamus API. So you use the CLI and you need to give the API 3 things. You need to give— you need to tell the API what you want to encrypt. You need to tell the API to which application you want to encrypt and in which Kubernetes namespace this application lives. The way Kamus identifies application is using a Kubernetes object called service account. Service account is an object, Kubernetes object, that provides identity for a pod.

27:19Chris RomeoOkay.

27:20Omer Levi HevroniIt has a token and application can use this token to interact with Kubernetes API or with any other API, and you can give permissions to this service account. It's a really neat feature. So you need to give Kamuz these 3 parameters. Again, the secret, the application or the service account used by the application, and the namespace. Kamuz encrypts the secret for you and returns it for you. Now the developers can put the secret in Git Twitter, Facebook, whatever you're using because it's strong encryption, so put it wherever you want. Then the regular CI/CD magic happens. Your code gets into Kubernetes and when your application starts, before it starts, it needs to interact with Kamos, provide the service account token and the encrypted secret. Kamos performs authorization and if the secret indeed encrypted for this application, the secret is decrypted and the application gets the decrypted value back, and then it can use it to perform its application.

28:21Chris RomeoOkay. So where are my secrets stored then, even when they're in that encrypted format? Is SKMS something that's using— is it providing or storing my— just in files, or is this like a cloud service? What am I interacting with? Where does this API actually live that I'm interacting with?

28:44Omer Levi HevroniSo, Kamos does not handle the part of storing the secret. You can store the secret wherever you put other manifest files. So ideally you can store it in the same place where you store all the other files used to deploy your application to production. So it could be Git, it could be file server, it could be blob in the cloud, whatever already works for your company. Kamus is responsible for encrypting and decrypting the secret, and it's an API running inside Kubernetes, and it needs to run inside Kubernetes so it will be able to authorize applications based on the service account token.

29:22Chris RomeoOkay, got it. So then, so then it keeps its keys inside of Kubernetes?

29:29Omer Levi HevroniNo, Kamus has a pluggable encryption system. So currently Kamus supports in-memory AES key, which is the most insecure installation of Kamus, but it has support for Azure Key Vault and GCP KMS. Both are very strong solutions, provide HSM protections, and Azure also provides IP filtering. And we hope in the future to add support for AWS KMS and maybe other solutions like native HSM or something. It's really pluggable and it's really easy to add new encryption systems to Komoot.

30:12Robert HurlbutSo it becomes, it sort of becomes an interface to these other, I know Azure Key Vault and other solutions like that, so it sounds like it will allow you to use those other mechanisms as well pretty seamlessly.

30:27Omer Levi HevroniYes, the idea is to make it easy for application to talk with this storing encryption solution and solving the— Kamus solves the layer of authentication and authorization. This is what Kamus solves. It does not handle encryption itself.

30:43Chris RomeoOkay, so Kamus is really the middleware between Kubernetes and Azure Key Vault or Google Cloud KMS, and I see you're going to do in the future AWS KMS. So, it becomes like that middleware layer that just handles the connection between Kubernetes and the storage and the secure secret storage.

31:07Robert HurlbutYes.

31:09Omer Levi HevroniAnd we do hope like in the very, very future, I see Cammus as something that can be relevant also outside of Kubernetes. Secret is not a problem that is unique on Kubernetes. And ideally, at some point, we can say, for example, I have a Lambda running in AWS and I use the identity of the Lambda to authenticate to Kamus and get my secrets for Kamus. But this is like very far in the roadmap, but I definitely see it's going there because it can solve a lot bigger problem than just Kubernetes secrets.

31:43Robert HurlbutOkay, so what are some of the things that it can help you with in terms of threats that the solution will help you with?

31:52Omer Levi HevroniSo I think the main reason Kamos is superior to other solutions is the very simple permission model. The fact that no one can decrypt the secret, you have one-way encryption, makes things a lot more simpler because now the attack surface of where the secret lives is only in the application itself and of course on the computer where the developer generated it. But the attack surface is a lot smaller than in other models where you have a role that lets you read the secret or decrypt it or something like that. And I think this is the main security feature of Cammus. And once you understand that and you understand that in Cammus there is no permission management because there is one flat permission management, it improves the security significantly. And we put a lot of other mitigations, for example, We make the CLI secured. It's enforced HTTPS by default and it's allowed to support certificate pinning. For example, we make sure that the API has security tests covering all the code entering into it. We have 2 different APIs actually of Commons, one handling encryption and one handling decryption. So once you have 2 different microservices, you can expose only one of them to the internet. The one that handles the caching does not need to be exposed to the internet. And all of these things reduce the attack surface significantly. And to make it clear to everyone, we created a public threat model. We use a format called Jerkin. I took the inspiration from OWASP Cloud Security Project.

33:39Robert HurlbutOkay.

33:40Omer Levi HevroniIt's a nice OS project that tries to document different threats and mitigations of cloud providers. And so they use JARCAN format and we use it also. So you can check it out, it's on Komoot repo, and you can see there the different threats and mitigations we think about when we did the threat model to Komoot. And so all those things I talked about and a lot more is covered there.

34:04Robert HurlbutOkay.

34:07Omer Levi HevroniSo this is from a security point of view, and of course we have SecurityMD to make it easy to perform responsible disclosure because I'm sure we didn't cover everything and there are things we missed. So if you think about something like that, in the repository it specifies how to report it.

34:22Robert HurlbutOkay, good. So let's say we want to follow up and, and get some more information. Where, where should we go to, to get some more information on Kubernetes and secrets and Kuma and other kinds of solutions?

34:37Omer Levi HevroniSo I wrote a blog post on the Celuto engineering blog. The blog post covers pretty much the things I talked about and a lot more. And it also covers how you can start with Kuma. It's really easy. It's one Helm command. We released a Helm chart to make it easy to install Kuma. So you can read it, go over it, and then get into the repo. We have detailed installation details explaining how to install Kuma securely. And we have a Slack so you can come talk with us on common Slack. And if you have any question, come ask. We will be more than happy to answer it and help other people use it.

35:14Robert HurlbutOK, and you have a link to that threat model you were talking about as well in the blog?

35:18Omer Levi HevroniYeah, and put it— we will— I will send it to you so you can provide it to the audience.

35:25Robert HurlbutOK, yeah, we'll make sure it's available.

35:28Omer Levi HevroniOK, great.

35:30Robert HurlbutAll right, any, um, any last thoughts? Uh, thanks, Amar, for, for joining us today, but any last thoughts for our listeners?

35:36Omer Levi HevroniUm, I, I think the most important point is what we talked about at the beginning, thinking about how we as security engineers can make the world of— the work of developers easy and help them choose the right and most secure solution. And this was the main motivation, motivation behind Camus. Making it a lot harder to make mistakes and expose secrets to the world. Yeah.

36:09Robert HurlbutSaid very well. Thank you very much for joining us.

36:12Omer Levi HevroniThank you very much for having me. It was a pleasure. I really enjoyed it. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.