José Carlos Chávez - When Museums Get Hacked: OWASP Top 10 Lessons from Heists

0:00Chris RomeoJosé Carlos Chávez is a security software engineer at Okta, an OWASP Karaza co-leader, and a mathematics student at the University of Barcelona. He enjoys working in security, compiling to WASM, designing APIs, and building distributed systems. When not working with code, you can find him sipping kombucha or enjoying time with his children. Quick note, we're opening up a few sponsorship spots on the Application Security Podcast. If you want to get in front of real AppSec practitioners, the folks actually making decisions, this is a great place to do it. Just reach out to me, Chris Romeo. I'd love to chat. You can find me on LinkedIn. Hey folks, welcome to another episode of the Application Security Podcast. My name is Chris Romeo, general partner at Curve Ventures, and co-host of said podcast, as always, joined by Robert Hurlbut, who's in an undisclosed location at the moment. So we cannot tell you where he is because we don't know. All right.

1:10Robert HurlbutHey, Chris. Yeah, Robert Hurlbut and threat modeling architect as well as trainer. And glad to be here and to talk about application security as always.

1:19Chris RomeoYeah, we're going to get into the OWASP Top 10 for 2025 for the first time somehow. We got to 2026 before we talked about this, but before we get there, we're joined by Jose Carlos Chavez, who is going to educate us about this new OWASP Top 10. But first, Jose, let's, let's get into your security origin story. How did you get into the world of security and then ultimately application security?

1:46José Carlos ChávezSo yeah, that's, that's a very— oh, first, thanks for having me here. I'm so happy. to be here and very excited about this. So it was by chance, actually. I was working as a software engineer for, I don't know, 12 to 13 years. And then actually most of, well, most of my career was working on observability from the very beginning of observability when observability was mainly theoretical, let's say. And then at some point, I joined a company called Traceable, which we were doing security on top of observability, which was a very innovative concept. And I started to get interested in security. So I was working on security features as a software engineer. Um, and then in 2024, um, I was kind of like trying to find my way. And, and, and then I, I saw this opportunity at Okta as a security engineer. And, and I was like, okay, I, I know something about security. I'm not a security engineer. I, I wasn't familiar with all the acronyms in security, you know, SAST.

3:11Chris RomeoYeah.

3:13José Carlos ChávezDAST, CISO, I mean, you name it. So that was my first approach to security, trying to, you know, get along with all these concepts and then say like, okay, let's try it. And ended up joining Okta as a security software engineer. And still, my mindset is still as a software engineer. So, I would say I am a very hands-on security engineer, mainly because I come from the other, well, not the other side, but from the engineering side, right?

3:49Robert HurlbutYeah.

3:51José Carlos ChávezBut so far I'm liking it. I have really learned a lot and I always try to bring all my software engineer practices into the security world. And my first, or the first way for me to connect was through automations, right? Because I, you know, I needed repetitive, deterministic processes. And then I saw that security has a lot of processes, but then many of them are usually manual or ad hoc. And I was like, it didn't fit in my software engineer mindset, let's say. So I, at the beginning, I started like, you know, we can automate this, we can automate that. And on the way, I started learning the problems. Okay, what is vulnerability management? What is platform security? Why is this important? And how can we make sure that we have a consistent, repeatable, and deterministic security posture? So, yeah, that's mainly how I got into security.

4:54Chris RomeoSo, you mentioned observability in kind of the beginning of your origin story here. Observability is that— I'll tell you what I think it is, and you can tell me what it actually is. Observability, when I think observability, I think some type of software that's watching what the software is doing kind of from an internals perspective. Is that— am I on the right path?

5:22José Carlos ChávezYeah, yeah, yeah, totally. So, observability is about understanding the system state through its output. Like through its metrics, through its logs, like, because, you know, we're talking about dynamic systems and then understanding the state of the current state of a system is very complicated because you have many pieces moving. And then observability aims to, you know, make sense out of all the moving pieces. And then how can you, what is the state of the system right now? Well, you have this component, this component, this component, but what is the state? Like, give me a number, right?

6:00Robert HurlbutYeah.

6:01José Carlos ChávezAnd then observability helps you to understand, okay, how the data is flowing, what are the bottlenecks, what are the source of errors, what are the chain of dependencies in components, and then be able to pinpoint the failures and say like, okay, you know, this is failing because this specific component that is being called by this upstream service is failing. I used to give talks about distributed tracing, for example, using the analogy of a pipe system. And then the talk was something like, okay, imagine you have a pipe system, you open the tap, and then the water is flowing. And then let's think about it as a 200. And then let's say next day you open the tap and the water is not flowing. And then, you know, behind the scenes, you have several pipes being connected to components that regulate the water flowing and everything. But when it be nice if one of the pipes yelled at you and say, hey, I'm clogged. That's why the water is not running. And this is basically the whole concept of observability, that you can pinpoint the failures, pinpoint the bottlenecks from a specific component that overall makes the system fail or have a deviation from the normal behavior.

7:19Chris RomeoYeah, it's interesting.

7:26José Carlos ChávezAnd then this concept of building security on top of observability was mainly like, okay, if we have all this data from observability, that is usually used for operational concerns, can we use that data for security concerns, right? Because one of the main goals of observability is to be able to react. And then to react, you need to make sense out of the data, right? And then convert this data into information and that information into action.

7:55Chris RomeoSeems like it's even more important now in an AI world where we've got code coming from who knows where that was created by who knows what model, and now it's in production. Seems like observability is a— I know we're gonna talk about the OS Top 10, but I'm kind of, I'm pulling on a thread here that's interesting to me. It seems like observability would be like, Something that would be crucial as a feature to have better understanding of what AI-generated code is doing.

8:29José Carlos ChávezYeah, yeah. Because also you have many intrinsic pieces that you don't know how they work, right? You have no clue how the models work. You have the theory on how, like in generative AI specifically, how the data is being generated. probabilistically and everything, but actually what is happening, you don't know. And there are a lot of unknowns in AI. The thing is that nobody has time for that now because everyone is trying to deliver features and, you know, I want to do it faster and faster and make it open to more use cases, but nobody, well, there might be some companies, but I haven't seen a strong position around, okay, first, Let's make sure that we understand how all the pieces work and what are the bottlenecks in my pieces. What are the, or even things like understanding, like, what is actually my model doing in the internals, right? How do I, how can I trust that the outcome is the right one if I don't understand the process, right? And observability is more or less like that. So I have been away from observability for a while. I don't know what people is doing in observability now for AI specifically. I guess everything now is related to tokens. So I see there might be a lot of interest on, okay, where am I spending most of the tokens? But I guess it will be really cool to step back and say like, okay, Let me see where all, what are the strongest points or the stronger points through all this reasoning, and how can I pinpoint where are the weak points in all this process of generating code or generating outputs?

10:22Chris RomeoMakes sense. All right, Robert, you better get me back on track here, or I'm gonna pivot this entire episode to AI and observability. Okay.

10:32Robert HurlbutUh, well, actually, I'm just curious about, I think you, uh, did a talk on museum heists. And so getting us back on track on the, on the top 10, but there's one particular top 10 category, uh, that you picked and I think mapped to this heist. Um, just curious what made that click for you?

10:50José Carlos ChávezUm, so the story is very, um, incidental again. I was giving a talk at the, at what, Rocher Labs, you know, about OWASP Top 10. And that was, my talk was, I think it was 2 days or 1 day after the Louvre heist. So I was reading about the Louvre heist and I said like, I was reading the news. And the first thing that clicked on my mind was like, oh, there are cameras, but cameras are not pointed specifically to that part where the robbers got into. And I was, immediately my mind connected to, you know, lack of observability or monitoring failures. And then as I was reading through the news, I started connecting dots with like, yeah, this is, like security vulnerability in a way. And then in the end, I catch like 4 or 5 of them and I said like, this is very similar to OWASP Top 10. And then I started connecting the dots and doing some more research because my talk is not only about Louvre, or was not about Louvre specific, or most of the Louvre because it caught the interest, but like I had to dig to get some others, right? Because Lubri wasn't enough, although it's a very, very big case. And yeah, I had to, you know, change my slides in the last day to make it more appealing because OWASP Top 10 is very interesting, but it's also very dense. So if you present it from technical side only, people is gonna, you know, at some point you're gonna lose them.

12:43Robert HurlbutYeah.

12:44José Carlos ChávezBut if you present a story behind it, and more so if it's an interesting story, a fresh story, like it was Louvre at that point, then it's easier to get people connected. So that was how I connected to museum case.

13:02Chris RomeoYeah, I think it's a brilliant way to approach a topic that's gotten a lot of attention and had a lot of talks. It's probably more talks in AppSec on OWASP Top 10. in the history of AppSec than any other topic. But one thing I do have to check in with you real quick, Jose, you've never been one of the people who stole something from the museum though, right?

13:26José Carlos ChávezNo, no, I just check. I had to confirm. Yeah, I might have stole some books from library. I mean, forgot to return, but never anything from a museum.

13:39Chris RomeoNothing. This is— so this isn't based on your experience. You're not like using the Louvre as an example and like, well, it just happened to be I was there when this happened.

13:46José Carlos ChávezNo, no, no, no, no, no, no. I'm just kidding.

13:50Chris RomeoSo top 10, 2025 is the new version, last updated 2021 before this. Anything big or anything that caught you off guard when you saw the 2025 and what came out in this new version?

14:05José Carlos ChávezSo, um, things changed based on, on many, um, I mean, the last 2 years have been very hard for supply chains, for example. And the new OWASP Top 10 was reflecting that because on top of all the vulnerabilities we had, which were like very oriented to the code and, you know, on the system and on everything we had in the last 2 years, we had a lot of incidents and a lot of buzz around stolen accounts and then attackers pushing malicious code into open source libraries. Last year, we had a lot of CVEs related to libraries being compromised. I remember, I don't remember the month, but there was a month where we had one, every weekend we had one CVE went zero-day, and we had to react to that. So I think the WAS Top 10 evolved on that line, and you can see the supply chain failures, for example, that became one big chunk. But on the other hand, I read, so, I started getting involved into OWASP Top 10 recently, but before I was trying to prepare OWASP Top 10 for service mesh. And then I was doing a lot of reading on that and security reports. Like I was reading all the security reports that were going out, try to grok the information. And one of the things I read and I couldn't find, I think it was one synopsis. Yeah. Report or somewhere was saying that although all the sophistication that we might have now, you know, we have AI wasn't a thing at that time, but we have the open internet and we have open source libraries and we have more powerful machines. The attacks are, or the core principles of the attacks remain the same. And also attackers are exploiting more old vulnerabilities rather than new. So there are a lot of attention, you know, there are a lot of blog posts being written about CVEs or new CVEs or zero days or new vulnerabilities. But the majority of the attacks are, according to that report, were around old libraries, old-fashioned attacks. So I think in that sense, you will see that, you know, OWASP Top 10, the difference from the 2025 to 2021 are that, And new concepts emerge based on latest incidents, but also the solid, like the core of the incidents remain the same, which is, but BOLA, for example, broken object level access, for example, remains the same. Injection is still powerful. So yeah. There are changes, but I would say the usuals never get old-fashioned.

17:31Robert HurlbutSo, broken access control is still sitting at number 1 for 2025 OWASP Top 10. We've got mature tooling now, but why is it still number 1? What are your thoughts on that? Is it a tooling problem or an org problem?

17:55José Carlos ChávezI think it's both. There is this law that I cannot remember the name now that says that systems reflect the organizations, right? How the information flows across the system and how the components are connected is a reflection of how the company works. And I think you can have one without the other. Most of the problems with broken object-level access is first that although you might have new tooling now, things are changing fast. And it's hard to catch up with new roles, new capabilities, new services. That's one thing. On the other hand, there is an overhead of reconciling all the time these new things. Like I was working, for example, in a project which was basically trying to take a RBAC approach. RBAC is relational-based access control. And the idea is that you have now the access decision based on relationships between nodes, right? I can access this, not because I have this role, but mainly because I am connected in a graph. And that part is easy, right? Saying who can access what. It's easy. Keeping that up to date, it's hard. And then leveraging a consistent policy on how to enforce that is even harder, right? For example, at Okta, we have a lot of tooling that we build in-house to make sure that we can enforce these policies. Because you can have these policies, you can define them. It's hard to keep them up to date. You have to all the time be revisiting, okay, now we have these new resources, now we have these new services, now we have these new permissions. But then reconciling like, okay, what are the teams that have access to this? Who is the manager? Who, who, how do you inherit permissions? And then this concept of temporal access, which is new, that wasn't part of any specification of access control, is also something. that is required now. Now more people talk about break glass solutions, right? Whenever you need to access production system with lots of permissions because of an incident, because if it's a very specific case, and then who is in control of revoking those permissions? And then you have a lot of over-provisioning now as well because Yeah, companies usually don't have this tooling for, okay, these permissions were granted to you, but do you still need it? Do you actually have access to it? Like, should we revoke that? Companies are not looking into that because, of course, the way I think is that It's just, there is still this mindset and I'm coming from, as I told you, coming from software engineering, but I still see this mindset where people tend to confuse security and compliance and then say like, okay, we have these checks, right? We implemented these policies, check, check, check. Okay, we are safe. But then that doesn't mean that you are you have a good security posture. That means that you fulfill certain things, that you fulfill certain policies and certain requirements, but that doesn't mean that things are safe now. So when I was working at something called Kubernetes Security Observability at Okta, and then I was doing a lot of research, which the idea was that you will understand from Kubernetes events, you will understand your security posture and then propose actionable improvements. And I was reading this book from, I think it was around eBPF or something that was saying, you can implement all the controls and then hope for the best. Because hope isn't a strategy, right? You have to implement the controls, but then monitor Um, how is it working? Like how many denials did you have? How many, um, permission grants did you have? And then what is the baseline? What are the deviations? You know, who are the outliers? And then that requires again observability. So that, that also made a click on me because yeah, you can just leverage the policies and then say, okay, we are safe. Or, or, you know, enable Falco in your cluster and say like, we got Falco, right? Mm-hmm. Whenever I need to know something, I will go and check the logs. But that's not the way to look at this. The way is like I implement Falco and I harden my environment. So I have a very steady baseline of events. And then I can set up alerting and say, whenever we receive an alert, it's because something is happening. Right. But I see in many cases that people implement tools and then Okay, they say, okay, we got the tools, you know, but nobody's checking like, okay, how many logs did this tool produce? How many false positives? So I think that's one of the challenges. And then again, the other challenges are the practical stuff, right? How do you ensure that all your employees go through Artifactory instead of going directly to npm?

23:38Chris RomeoYeah.

23:41José Carlos ChávezThis requires tooling and the tooling has to be up to date, has to cover several use cases so you don't break your users. So you have to invest a lot on that as well. And I think that's also another challenge. And if we talk about corporates that probably have contracts with other corporates or contracts with governments, then okay, they have to do it. But if we talk about startups that are you know, fighting to survive one more month, then all these things are just burden. So they do their best, but, you know, they cannot, like, secure everything because it will be a lot of effort and a lot of time and then compromise the revenue.

24:25Chris RomeoYeah, I haven't seen a startup yet that has implemented some type of internal repository for open source. You just don't, you don't have time. Like npm is where you get JavaScript packages from when you're building something at a high rate of speed. So that's, that's a, that's a good point that you made there. How about injection? So this is the one that I don't know. I've been around security for a while. Robert has as well. And it just seems like we should have solved this one already.

24:59Robert HurlbutYeah.

24:59Chris RomeoLike, why is it still number 5 on the list? We've had parameterized queries forever. We have ORMs for being able to access databases securely. And yes, they still have raw functions that you can do if you really want to, you know, use raw SQL, you can still do it, but at least you have to make a decision to do it. Like you have to go out of the way. So I don't know, this, this is just the one that I don't, I don't get why we haven't solved this one yet. It feels like this is, this was solvable. 5 years ago. What do you think, Jose?

25:32José Carlos ChávezUm, so yeah, I think one of the big problems of injection is that, um, first, yeah, you can prepare your statements, you can, as you say, parameterize your queries and do everything, but then you don't know what is going on in your underlying, um, let's say in, in, in your underlying infrastructure. First, there are a lot of dependency chains depending on the ecosystem. In ecosystems like JavaScript, you will pull a lot of underlying libraries, and that might contain old software that might be susceptible to injection. That's the first point. The second point is that there is a lot of legacy. And then, for example, I am a co-leader for OWASP Coraza, WAF, which is a WAF compatible with core ruleset. And then we have an extensive set of rules for injection. And when I started working on Corasta, I was like, but is this truly needed in 2000? Back then it was 2022. In 2022, do we still have injections? And you will be very surprised how many CVEs related to injection we had in 2024, 2025. And we're talking CVEs, which means that those are open source libraries. So to me, this is the main source of problems when it comes to injection. Like open source libraries are still susceptible to injection, not only to SQL injection, but you can do code injection. I remember we had, I saw an incident like last year because you could render malicious JavaScript in a page because the underlying library wasn't sanitizing the HTML. It was rendering a text that will also render HTML. So yeah, and here we are talking open source libraries. Imagine also proprietary code. We tend to believe like, oh yeah, we sanitize all the inputs, we validate everything. But do we? Like, I hope with AI, this is gonna— one of the good things about AI is that at least from the reviewing code standpoint, like if you have an AI-based reviewer, these kinds of things can get caught. But before that, people will write code the way they will write code and not necessarily adding these guarantees. So yeah, as surprising as it sounds, open-source libraries still are prone to injection.

28:25Chris RomeoYeah. And to your point, it seems like AI should be able to understand SQL injection and search a codebase for it. Because it's not like it's that complicated of— there's not like 100 different ways that you can introduce SQL injection in 100 different categories. It's, it's, it's at the same point where data access is being done. It's a failure. Like we said, it's a failure in its, you know, raw SQL being, you know, not parameterized and, and, um, being manipulated without input validation. So yeah, I'm with you. It seems like AI should be able to find these things and somebody needs to start a foundation and then just buy a whole bunch of tokens and call it the SQL Injection Removal Foundation.

29:23José Carlos ChávezSo there is an initiative, I think it started by ChainGuard. I have no association with ChainGuard, by the way, but I remember And I saw this as a positive thing, like they will start a program to protect open source software, to analyze, to use AI to analyze open source software and do patching, which is great. I think at least these are for private companies or, I mean, yeah, corporates as well to try to keep the open source system, the open source ecosystem like in a decent point or in a better point. Still, it's never enough for an open source project, right?

30:05Robert HurlbutLet's talk about software data integrity failures and supply chain and how those 2 may overlap in the real world. When you're triaging an actual incident, how do you tell which one you're looking at, either those software data integrity failures or supply chain? How would you tell?

30:31José Carlos ChávezYeah, it's complicated. I remember last year we have, or was it 2024? I don't remember. I think it was 2025, the TJ Actions incident, where they will basically infect a well-known, widely used action for checking changed files in GitHub Actions. When we were alerted about the problem, we tried to go to GitHub to see, because from the GitHub security advisor, it said like all versions are vulnerable. And when we tried to get into the source code to analyze what was going on, the repository was gone for a day. So we couldn't tell whether this was a mutation of the tag. and then all versions were vulnerable because all versions were mutated, or if the vulnerability was from the very beginning and it was just discovered, right? So this is, I think, one thing we should do better as an ecosystem or as an industry is to pay attention to our dependencies. We, I've spent a lot of time trying to get deterministic builds, immutable builds, through pinning dependencies in the very— because in language, when you build, let's say, when you build an application, you're kind of forced into doing supply chain well, right? Into, okay, if it's npm, I will use Yarn or npm or pnpm, and then I will have a package.json. And then a package-lock.json, which will tell me what dependencies am I using exactly on what commit and all that. That's kind of the default, right? And same in Go. Go is even more strict on that. But other things that are around, and this is also related to the new DevOps approach, like you have now CI and CD. And then things like GitHub Actions, people usually don't pin So a mutation and attack will expose you. Things, easy things like Dockerfiles, you might see Dockerfiles, plenty of Wget or curl where they are pulling down artifacts directly from GitHub host. And then with no integrity check, no checksum, Nothing, just curl and then directly execute the bash script or unzipping the content and then dumping into the Dockerfile, into the Docker image. These things nobody looks into. Makefiles that do the same. So yeah, GitHub Actions, Dockerfiles, Makefiles, Jenkinsfiles, like all this around the way we build applications is still in security debt for supply chain. We don't. And now, what is more concerning now to me is like, people run skills and then who guarantees that your skill is what you are expecting to run, right? You just run the skill and then what if other skill mutated your skill? What if other skill mutated your scripts, right?

33:58Robert HurlbutYeah.

34:00José Carlos ChávezWe don't have anything like package-lock.json for skills. So basically you trust whatever it says. Nobody reads the skills. You just trust it and then hope for the best.

34:11Chris RomeoAgain, you don't even, you don't even, we don't even know who wrote them. I've been playing around with Claude and looking at the different plugins and things they have. And like, I click on the, on the catalog and look at them. I'm like, who made these things? It's not really very clear who made them. Much less can I check the integrity of a skill that I downloaded to see, is this actually the skill from the people? I can't even see who did it, nor can I check the integrity of it. Yeah.

34:40José Carlos ChávezAnd yet you grant them all the permissions, right? Because you allow them to run anything, to run any tool, to run any binary. You just dangerously allow every execution, or you don't even read the permission. And even if you read the permissions, like, Nobody tells you, I need read because of this, and I need write because of that. It's just, okay, you list the permissions, and who knows what is going on. Yeah, there is a nice project.

35:07Chris RomeoA hidden— yeah, this is like a— this is something that people aren't talking about enough right now.

35:13José Carlos ChávezNo.

35:13Chris RomeoThis idea that we've basically thrown a lot of basic security stuff out the window With the speed with which AI is moving. I hadn't even thought about this and I have, you know, loaded skills and stuff, but Jose, until you mentioned this, like I've been sitting here, you know, as you were talking, I'm like, wait a second, I'm guilty of this too because I want to go fast. I want to have Claude do all these things for me. Wait, why does Claude have control of my computer all of a sudden? Like we've, I've given Claude access to everything, including the ability to drive my laptop.

35:47José Carlos ChávezYeah, yeah, exactly. So there are a few things here. First, I wanted to talk about it. I submitted a couple of talks. I don't remember which conferences around SEAL. SEAL is a project from a colleague of mine, which basically teams the skills bundles or the cloud bundles like skills, hooks, agents. So whenever you, there is a mutation, then there is a failure on checking like, okay, this is still prompt doesn't match what it's expected to match, right? This is a nice project that he is starting, just briefly collaborating with him. But then there are other approaches like Nono, for example, which has a sandbox approach for Claude. So basically you You put a sandbox on Claude and then you control most of the things. But again, like if you don't check the integrity, although it's sandboxed, you can still run malicious things, right? You can still take malicious actions. So it's a combination. But I think one of the problems that I see in this kind of corporate-driven open source or corporate-driven specifications is that they try to get features as fast as possible, and then they try to regulate or write standards based on nothing yet but a promise that this feature, this is what my customers need, this is what my customer has to do. I see the same in observability, basically, with OpenTelemetry. You will see the standard being driven by major vendors, but then not based necessarily on user experience or use cases, but more like, I promise to deliver this feature, so I need to modulate this into the spec, but I don't know yet how it works. I see the same in cloud. For example, when I was trying, when I was building my very first skills, the standard wasn't clear. And there was no strict standards around how a skill can be built, how the permissions are declared. And I was like, okay, you just, you know, release this spec because there was a pressure, because, you know, everyone wants to release the latest and greatest, but you haven't considered this, this, this, this use case. And then—

38:29Chris RomeoYeah.

38:29José Carlos ChávezIt's half-baked. That's what I am seeing now. And then, for example, easy things like, okay, you have a manifest that declares what, or hashes your hooks, your agents, your skills. There's nothing like that, which is something basic from security. So yeah, I mean, I think this is, This is normal in a corporate-driven open source, let's say.

38:58Chris RomeoLet's deal with logging. Logging's always the forgotten item, and anytime you make a top 10 list, it always drifts to the bottom. No one's ever made logging number 1, like a lack of logging. But you said in your talk that logs need to be immutable, reliable, and cannot be manipulated. And that seems like a pretty tall order. And I agree with you. I don't disagree with you, but I think it's a challenge to have truly immutable logs. I think we've been searching for that for 30, 40 years, and I don't know that we've ever found it. So, I'm curious to get your take on this, the way you describe this, immutable, reliable, cannot be manipulated.

39:44José Carlos ChávezYeah. So, the idea is that logs are, your source of truth, or, I mean, the way I see it is like, if you think about events, the promise is that from an event, you can reconstruct the reality from the events, right? You basically aggregate them and you can reconstruct the state. And for that, you need first that all the events are registered, that all the events are recorded, more so if they are important, if they are sensitive actions. That's number one. So, and then you need them to be reliable in a way that whenever I retrieve an event, that's exactly a reflection of what happened. And for that, you need to take the same warranties as you will do with any database, for example, like encryption at transport, encryption at rest, and then permissions for writing, or even better, like immutability.

40:48Chris RomeoRight?

40:49José Carlos ChávezFor example, I remember back in the days when we were trying to deal with events and then we were looking at how to use Kafka. Kafka is basically a log stream. You would have events or you would have messages there. And then the idea is that messages cannot be mutated. And if you need to mutate it, then okay. Because we cannot mutate the records in Kafka because they are, let's say, the next state depends on the previous one. So you cannot basically update the previous one because otherwise you will break the whole chain. The same pattern should apply for our logs, right? That you will have aggregates that are basically declared a linear story. And then you cannot screw with an intermediate because then you will break the whole story. And this is important for many things. First, for forensics, of course. If there is an incident, you want to go through the logs and recreate the past and recreate what happened, right? That's number one. Number 2 is that logs will tell you things, right? Logs can lead to actions. And this is why you need logs to You need to record everything. You need to record every action that is important for the user. And number 3, logs are still data and you should treat them as every other data that you treat customer data. You know, even they might have, might include sensitive data. So you also have, it also has to go through all the care that you will give to your databases. or your other data sources, but also have the consideration that they might include sensitive data and that they should be reviewed and anonymized before they get into the persistent layers. So yeah, again, for me, it's just natural to emit logs, to do audits because of my background. But And also, now, whenever you deal with things, I saw a lot of inclination towards trying to get reports, or like, for example, there are vulnerabilities, but then now, because of cloud, people assume that they can get any kind of report and say like, okay, tell me how many of these vulnerabilities are applicable. Tell me how many of these vulnerabilities are minor upgrade, major upgrade. Our third-party services, tier 1, you know, and then for that event, you need very, very solid logs, right? So cloud can go and slice and dice the data and produce you meaningful outputs.

43:39Chris RomeoOkay.

43:46Robert HurlbutSo let's talk about vulnerabilities. You mentioned in your talk vulnerabilities evolve, but root causes stay the same. If you had to name 1 or 2 root causes that were sitting under most of those top 10 categories for the OWASP Top 10, what are they? What are those 1 or 2 root causes?

44:06José Carlos ChávezI would say the first root cause for me is lack of maintenance. Like, we have layers over layers of data or of libraries. And then when you build software and you depend on library A, and then library A depends on library B and so on, and then 10 layers deep, you will stumble upon Lodash, you will stumble upon LeftPad, you will stumble upon FormData, you know, and those libraries are either poorly maintained because they are just small libraries that cover very small use cases, but are that are widely used, then you got that problem. And then the second, I would say, development practices are still a problem. People still build software and then not necessarily including the practices that you might have seen. Like, I remember in 2000, I think it was 2003, 2015, everything was about microservices. Everyone was talking about microservices. And then people, you know, wrote any sort of microservice. 2016, everything was about Lambdas. And then all the conferences were talking about Lambdas. And then it was a very innovative concept, like, let's make everything a Lambda. And then when you look what a Lambda is and you look what people write in code as a Lambda, it's just a very amateur JavaScript file that people execute because all the scaling power, all the operational power is abstract. So I would say bad development practices are still a problem. And that led us also to have very insecure systems. And then number 3, which is now that I will say it's recent or new, is that software has evolved, right? Now we are in the AI age. And that, I wouldn't say democratizes software, But now more people are into software because there is this selling idea that you can build anything with AI. And again, you are just live coding or using, you name it, the vendor to build your own, to pitch your idea and then build your POC. And then you are basically conferring all your superpowers to agents, to AI models and everything. But nobody should receive enough security education for that, right? People who is basically giving the entire, the key to the realm to Claude are not aware of what are the risks, right, of that, because there is no, there was no any requirement to start using these tools. You just, Yeah. Because it was massive now, and then everyone could access to it. It just went— before, let's say, software was more like a niche thing, right? If you were a software engineer and you had certain knowledge about security, then you will build software and maybe operate the software and maybe deploy it somewhere. And then you have basic sense of what security measures you needed. But now, because it's massive, It was given to the people and then everyone is building their own applications. I can write a Node.js application tomorrow. I have no idea what Node.js problems in security are, what Node.js considerations do I have to do to take, right? Um, but still I can build it and then I can run it and I can deploy it. Um, so I think that's, that's another, another problem, like lack of security education and On the other hand, I get a point, right? What is security location, right? You watch a video, you pass an exam, but then in the day-to-day, how do you make sure that this is enforced? And I think security policies supported by tooling is very important, but then who has time for that, right? Who has time? Okta is a big company, so Okta prepares very nice videos and trainings that we have to go through. And then they spend a lot of effort in building tooling that enforce all our security policies. But other companies just don't have time for that. And then they just don't want to hook into another vendor, right?

49:06Robert HurlbutYep.

49:08Chris RomeoYep, that's true. All right. Well, Jose, what would be like a single key takeaway that you wanna leave our audience with?

49:15José Carlos ChávezSo I would say the very first is security is everyone's responsibility. That's, everybody says that, but nobody wants to take the ownership of that, right? We security teams are not the only ones responsible for security. And one of the things I saw towards the time was that when I started in software, there was a developer's team and then there was a QA team. And then developers will develop and QA will do the QA. Then slowly the engineering teams started getting the mindset of quality assurance, started writing their own unit tests, started writing integration tests, started catching more regressions in end-to-end tests. Then CI came. And now you run all your tests on CI and then you catch, you have less bugs, right? By the time it reaches production or staging, let's say, in those times we were using those environments. You will see less bugs because now you have the possibility of running all your tests on CI before you merge anything. To the point that, for example, I was at a company like 3 years ago where we were running Istio on CI, right? Which is a monster. Istio is a big stack, right? And I hope the same thing is going to happen to security, that slowly engineering teams are going to catch up. And say like, okay, adopt this mindset of security to be embedded into their development, right? Security develop— secure software development lifecycle is an old concept that I hope is now getting more and more, being a common thing in the software industry. So yeah, my main takeaway will be that, like, security is everyone's responsibility. Security needs you. And ownership or security, for security being a thing, for security being effective, for security being tangible and possible to roll out across an organization, you need a strong sense of ownership. And ownership is crucial. for security to succeed. So, that will be my another takeaway. Invest in ownership, like, focus on ownership, deliver ownership, and live for the ownership.

52:07Chris RomeoThat will be— That's good. So, Jose, thanks for sharing your expertise with us, and I can't wait till the video for the talk is available. Jose, for those in our audience, this— he did this talk at OWASP Global Vienna. And as of right now, the video is not yet available, but it will be soon. So check that out. And Jose, thanks for being a part of the show.

52:33José Carlos ChávezThanks for inviting me. It's been very fun.