We Told the Agent No 11 Times. It Did It Anyway

0:00Rock LambrosWe have that Replit incident from last year, July of last year or something like that. And, you know, we've beat that one to death. Uh, you know, the, the CEO of, uh, what's Saastr? S-A-A-S-T-R. I have no idea how he pronounces it. Um, he was building an app using the Replit coding engine, right? But he put the project in a code freeze and by his own count, told the agent 11 times, don't touch anything. And it did anyway.

0:26Chris RomeoRock Lambros is Director of AI Standards and Governance at Zenity, where he helps organizations navigate AI security and governance. A recovering CISO with 30 years of experience, he has led security programs for multi-billion-dollar organizations across industries. Rock co-leads the OWASP Top 10 for large language models and contributes to several major AI security initiatives. He also co-authored The CISO Evolution and writes Rock Cyber Musings. A newsletter breaking down AI security developments for executive leaders.

0:59Robert HurlbutYou have reached your final destination. Well, not your final destination. You've reached your destination here on the Application Security Podcast.

1:06Chris RomeoThis is Chris Romeo, joined as always by my good friend Robert Hurlbut, who is a threat modeling legend. I'll give him that title. He'd never call himself that, but I'm going to call him that as he is embracing the world of AI and everything in the threat modeling space right now. And so we're joined by Rock Lambros. Rock, I want to jump right in here. And we normally ask security origin stories, which our audience is used to, but I'm going to ask a more kind of scoped question because there's a, there's a jump in your career that I'm really curious and I want to understand in more depth.

1:39Rock LambrosYep.

1:39Chris RomeoAnd so you wrote this, or co-wrote a book, The CISO Evolution, and now you're deep in agent controls at Zenity. And doing a lot of work in the world of OWASP, in AI, agentic, everything else. Like, so my question is, how'd you end up here? How'd you end up making that transition from CISO to now being in the deep end of the pool of this agentic world?

2:06Rock LambrosYeah. I mean, first of all, I would say that, you know, since 2018, I'm a recovering CISO of an oil and gas company. And since 2018, I was off on my own. Rock Cyber doing, you know, virtual and fractional CISO type of stuff. And, you know, the same old tried and true challenges were always the case, right? How do you articulate cybersecurity risk to the board? How do you get funding? All that kind of stuff. And then I sat down in front of a terminal for the first time in probably November-ish 2022 and used ChatGPT. And was like, holy crap, this is awesome. And this is going to go off the rails fast. And, you know, with regards to use within your organizations, because everybody's going to find this like, you know, like thinking it's a superpower. And if, you know, if we go back, you know, a whole like almost 4 years ago now, which is a lifetime ago, the models were pretty bad. Yeah. I mean, they hallucinated a lot. They were just flat, flat wrong a lot. Right. So, and so I had the realization that people were going to become very over-reliant on them. And so I set out to learn frankly everything I can about it. I'm even wrapping up a master's in data science and AI right now at the University of Denver. And, you know, I have like a month and a half left, but who's counting? But so, I mean, so, and I mean, I'm not going to lie. I was a little burned out. And, you know, the same old conversations, you know, get our SOC 2, get our ISO 27001. We got to pass it. Oh, we're going to be underfunded. What can we do to just check the boxes? And AI really gave me something new to dig my teeth into, which is why I jumped into a WASP. My dear friend Sandy Dunn dragged me in. And so if there's a— if you don't like the work that I do, go blame her. And, you know, I just immersed myself and now I'm eyeballs deep.

4:18Chris RomeoThis episode is brought to you by Origin. Quick question, would you hire 1,000 people without knowing what they can access or what they're doing all day? Of course not. But that's how most companies roll out AI agents. Today, you get the prompt and the final answer. Everything in between is gone when the terminal closes. Origin finds the agents on your endpoints and rebuilds their work. Every command, every file, every handoff between agents on one timeline. Origin Endpoint AI Observability. See what they found in their own traces at originhq.com/origin. That's originhq.com/appsec. All right, great. Well, Rock, let's answer the question, the title question that hopefully with a simple answer, who tells the agent no?

5:18Rock LambrosRight. So, let's start with the whole story. So, I mean, we have that Replit incident from last year, July of last year or something like that. And, you know, we've beat that one to death. Uh, you know, the, the CEO of, uh, what's Saastr? S-A-A-S-T-R. I have no idea how he pronounces it. He was building an app using the Replit coding agent, right? But he put the project in a code freeze and by his own count, told the agent 11 times, don't touch anything. And it did anyway. And then the agent told him that the rollback was impossible, which turned out to be wrong. And so the reason why I do keep coming back to this story, even though we've beaten the hell out of it, is it does answer the title of this episode. Because who told the agent no? The human did, the human in the loop, but via a prompt, right? 11 times. And a prompt is something the model weighs against everything else in its context, right? It's like having comments in the code. So the challenge was nothing in that stack sat between the agent and the database with the power to say, don't do this. And, you know, I mentioned that I'm a recovering oil and gas CISO, and in a gas plant, we never trusted the control system to shut the plant down safely on its own. The SIS, the safety instrumented system, Runs on separate hardware, separate control plane, separate logic, and it doesn't care how confident the control system actually is that things may be going well at the time. Once the safety system trips, it goes. And I think agents need that same split, right? So the short answer to my long answer, I think agents need that same split. Most of the deployments out there really don't have it.

7:08Robert HurlbutSo is an agent almost like a child that doesn't listen?

7:12Rock LambrosIt's a child that will take your suggestions, but it's a child that also wants to please you first and foremost before anything else, right? Meaning it will always accomplish its task, right? That is the function of how a transformer model operates in the LLMs that are running underneath these agents. They will complete an output, whether it's correct, whether it is within the boundaries or the guardrails that you set for it, distant secondary. But it wants to complete its mission, right? It wants to complete its goal. And because of the non-determinism of those transformer models, we've gotten better, certainly, but it's a, it's a crapshoot if it does it the way you want it to do it. Yeah. They're becoming more, I would say, correct in the output, but it's still a crapshoot on if they do it within the boundaries that you set. If you're just using prompt-based guardrails.

8:14Robert HurlbutWould you go as far as to say the prompt is advisory?

8:18Rock LambrosOh yeah.

8:19Robert HurlbutThat gets sent to a model?

8:21Rock LambrosAbsolutely. A prompt is loaded into context. And for those listening who don't understand what context is or a context window is, You've probably at least heard the term because, and now it's a marketing term. Claude gives you a million token context window before you hit that compacting conversation message in your sessions, which is frankly your cue to get up and go do a workout, grab coffee, go to the bathroom, because it takes a few minutes, right? Every time you see that, right? And so it loads the prompts into context and every time You hit enter on that keyboard, or every time there's an agent turn, it's sending the entire context to the model, bringing it back. And there is a fun phenomenon called context rot that researchers have determined. And God, I don't remember the body offhand, but you know, so what does that mean? The first token in the context is worth much more than let's say the 125,000th token in the context, right? So the fidelity becomes less and less the larger that context window gets. So just because we have a million token context windows doesn't mean that we should rely on those outputs all the time. I think, you know, we'll still see today long-running conversations going back and forth in Claude, ChatGPT, whatever. that they kind of start to degrade, right? Until maybe you start a new session. And that's, that's exactly the reason why. That's exactly the reason why.

9:58Robert HurlbutYeah. I find myself clearing context when I go from like with Claude, they've got that nice, I've got it in my status bar. It goes green to yellow to red.

10:08Rock LambrosYeah.

10:09Robert HurlbutAnd because the turn costs are going up too, right? Like the more context that you're packing into a turn, the more you're paying for for an individual, uh, you know, agent turn back and forth.

10:22Rock LambrosYep.

10:22Robert HurlbutAnd so I've, I've, I've started clearing context a lot more just as a cost savings measure because I found my usage on my plans were running out by Wednesday and they reset on Sunday. And I'm like, I don't wanna pay API charges because, you know, I don't, that's like, that's like a boat payment.

10:42Rock LambrosOh man. I don't even wanna tell you. Ugh. I am feeling the token burn so bad right now between a couple school projects and a presentation, a demo for a presentation that I'm working on, boat payment, mortgage payment. Yes. Yes. And that's even with leveraging prompt caching, right? So most of the models, I think Anthropic started it, GPT follows now. They'll, if you can reuse the same prompt or most of the components of a prompt, they'll consider it cache and only charge you at like, I don't know, 10, 20% of what it normally would be for that input, right? The output tokens are the same, but at least for that input. So if you understand how to leverage that, like within your coding mechanisms, that certainly helps. But no, burn, burn, baby, burn. Token inferno. That's how I feel.

11:41Robert HurlbutOn an episode with Jim Manico a few weeks ago, he was, he introduced something that one of the guys that he works with, the concept of a token furnace.

11:54Rock LambrosYeah.

11:54Robert HurlbutWhich, I mean, we laughed initially, like right when he mentioned it. Ron Paris that works with him was the originator of this term, and he was telling us how they couldn't stop laughing. when Ron first said it, but then you start thinking about how this stuff works and it's like, yeah, you know, if you don't have these things set right, or if you aren't very good at prompting and setting scope, it is ultimately a token furnace that's burning through all your available capacity.

12:22Rock LambrosThis is why you start hearing the, you know, the terms prompt engineering doesn't matter anymore. It's all about context engineering. And I tend to agree. I mean, prompt engineering is still important. Because if you, if you go look at the prompting docs for each model generation from Opus 4.8 to Opus 5 to Fable, um, or ChatGPT, you know, the 4th generation to the 5th generation to now Aster 6, there's the models like interpret your prompt differently. It's laid out there very explicitly. So prompt engineering is still important. But context engineering, super important.

13:01Robert HurlbutI find that people that are very new on their AI journey may look at a prompt like write secure code as something that the model, a state that the model can get into. And I just don't see that as where they are at this point. Like, so I think you do have to back up I think there is something to prompt engineering in that you need to scope things and ensure that you're putting the right other controls around how you're building software. It's not as much about the individual prompt. It's about don't just start up a session and say, build a feature that does X, Y, and Z.

13:45Rock LambrosYeah.

13:46Robert HurlbutGo through a process of figuring out the scope of what you're trying to build, Determining a number of specs that are gonna be driven by it, perform a threat model against those specs. I mean, I'm still doing all these things with the agents, but then from there you can start to build with some other constraints around, you know, other rules and things that you plug into that the agent's gonna, you know, should hopefully follow. No guarantee that the agent's gonna follow any of your guidance that you put in AgentsMD. AgentsMD is not a, uh, it's not a file that, uh, it's not guaranteed. Like, I don't think AgentsMD is a trust builder in the things that I build. It's once again, back to the original concept of the prompt is advisory. Things like AgentsMD, that's advisory guidance to the model. It can decide to ignore as much of it as it wants. So—

14:44Rock LambrosAbsolutely right. Because again, when you put guidance in AgentsMD or ClaudeMD or whatever, It's prompts, right? So it's strong suggestions.

14:54Robert HurlbutSo you and Steve Wilson, I guess, as you were working on the LLM top 10, and I wanted to hear this story about prompt injection because I think there's still a thought in the industry that prompt injection is a bigger problem than I think the data led you to. But walk us through this story, kind of take us through how this broke down.

15:17Rock LambrosYeah. So the headline, right, if you guys have been following, was that prompt injection fell out of the top 10, but it didn't, right? It's clearly still number 1 on the 2026 iteration of the top 10 for LLMs. Because what happened was this year was the first year that we really had a corpus of reported incident information By which to bounce our assumptions off of. So I mentioned earlier that I'm in a data science and AI master's program. So I put some of that data science to use. I took the community vote that we put out there that said, hey, kind of help us rank these top 10s. Do you think these, these, the incumbent top 10s are still relevant? You know, help grade them, submit your own new top 10s. So we took that community vote. Yeah. Um, and pegged it against the data. And, and prompt injection fell out of the top 10. It fell into number 12. And, but, right, the devil's in the details. There was a wide confidence interval there. It could have fallen from 4 to 18. But it was awesome, frankly, the conversation that it drove within the working group. Um, when I like dropped that nugget into the Slack working group, and then I almost like stepped away and just started eating popcorn. It was, it was awesome, the conversation that it drove. And at the end of the day, we came down to, it would be negligent for us to leave prompt injection off the list. Part of the issue or the challenge is the way we use words, meaning like it's a top 10 vulnerabilities list. Is prompt injection a vulnerability or is it, you know, an attack factor? Right. And it is an attack factor. It is the method by which most of the other instances of the of the top 10 are realized or actualized, but then also in how the incidents are reported, they're also reported, it seemed like based on the analysis, based on the outcome, right? So the outcome was sensitive information disclosure or hidden context exposure or misinformation. Versus prompt injection was the thing that led to it. So while prompt injection was kind of scattered throughout, it wasn't really reported as such. So we took, we ended up weighing the human vote across the top 10 because other things shifted as well by like 75, 25, right towards, towards the human vote. Because the practitioners, the people who voted, the practitioners in the working group, who see this day in and day out from a red team and blue team perspective, right? That judgment is still super valuable. And the data, it's the first year that we have it. There were over 6,600 incidents that we analyzed, but it was still very noisy.

18:18Robert HurlbutSo does that mean that prompt injection has been solved from the model provider side, or just that It's not a separate category. It's something that is distributed across those other items, the outcome-based, like, is it still a thing?

18:35Rock LambrosLet me be very clear because, um, I think it was Boris Cherney had a tweet about this a few weeks ago.

18:43Robert HurlbutWho's the head of, he's the guy who created Claude Code.

18:46Rock LambrosHe's the guy who created Claude Code saying we have solved prompt injection at Anthropic. It has absolutely not been solved. It is mathematically impossible to solve prompt injection. So there's great work out there by the folks of like Apostle Vassilev and Shoshana Cox around the proofs, uh, mathematically around that. If you want to really nerd out, the latent space is way too big. We're talking about trillion, 2 trillion parameter matrices, right? And it takes one token to compromise for that prompt injection to land. So it's, um, no, it has absolutely not been solved, which is why it is still absolutely on the list. And actually, if you read the, um, the, the leader from the leads or the letter from the leads, uh, in the top 10 for LLMs this year, um, you'll see that we say assume prompt injection. Now build everything else around that, right? Which is really no different messaging than we've been, than we've been giving and have been given for the last 2 decades almost around cyber resiliency, right? Build around the when, not the if. And prompt injection is effectively that for LLMs. Just assume it, assume it's going to happen, and build your controls around that.

20:15Chris RomeoThis episode's sponsored by Corgia. Design it, build it, ship it. Corgia secures it. Corgia is an AI-native app security platform covering your whole software lifecycle, from the first architecture diagram to code in production. It brings design reviews, AI-powered code scanning, and autonomous pen testing into one platform, so your team spends less time chasing noise and more time fixing what matters. Visit corgea.com. C-O-R-G-E-A.com. Corgia, one security platform for the entire SDLC.

20:49Robert HurlbutLet's shift a little bit. Excessive agency went from 6th to 3rd on the 2026 list. What moved it?

21:00Rock LambrosI think reach, right? The ultimate reach moved it. And what I mean by that is that the flaw behind excessive agency, that didn't get worse, but the reach of insurance all did. So I think in 2025, LLM apps, you know, they pretty much answered questions, right? You ask it a question, it can give you back an answer. 2026 has been the year of the agent. So now they open pull requests, they call APIs, they hold deployment tokens. And so that entity, if you go read it, It describes, I think, effectively 3 defects, right? That, you know, is there too much functionality? Are there too many permissions? And is there too much autonomy? And now, frankly, that's wired into far more systems. And it actually brings up a point. So we tried to set out with a hard line in the LLM for Top 10s to say, like, how do we How do we carve off the difference between the LLM for top 10s and the agentic top 10? And the LLM top 10 owns the risk when the model is a component inside your application. The agentic top 10 owns the risk when that model becomes an actor with tools, memory, and otherwise downstream consequences. But I think what we found this year is that like Venn diagram is getting— that middle part of the Venn diagram, that overlap part of the Venn diagram is getting bigger and bigger. So I could foresee a world in the next couple years where we just have a top 10 list as a result of that. And now if we look at it from the data perspective that, you know, that I mentioned before, the practitioners, the vote raised it 4th in the vote. The incident record itself came in 7th, right? And again, that confidence interval that I mentioned, Ranked it from 2nd to 14th. And so, the data's pretty consistent, actually, with the MOVE when you factor everything in. And the reason why, like I've mentioned, such wide confidence intervals when we leverage it against the data is because we don't have a good taxonomy. A risk isn't a risk across the board. A vulnerability isn't a vulnerability across the board. Agency. isn't even talked about, like, could be agency or autonomy, right? Depending on how it's used, right? So trying to do that semantic analysis on the entries versus how the incidents were reported and written up was super difficult, which is why, like, we see these confidence intervals so broad, and which is why we're still leveraging heavily on the human vote. Absolutely data-informed, But the humans still, still have to say.

23:52Robert HurlbutIsn't excessive agency where almost like— I shouldn't, I shouldn't use absolutes. Almost every developer laptop in the world today is, is in the midst of excessive agency because, I mean, that's, that's what I see the defaults for how these things are set up and deployed. And as I've been studying this space in more depth, There are certainly controls and things you can use, but there's so, I mean, this whole thing is so new.

24:27Rock LambrosYeah.

24:27Robert HurlbutLike it was last December when everybody went on Christmas or holiday, you know, winter holiday breaks and came back in January and said, these things can code.

24:37Rock LambrosYeah.

24:38Robert HurlbutAnd then there was an explosion. Like it's, it's only been 8 months since that kind of eureka moment across our industry. And so there are controls and things we're going to get into more of the details about those controls that people can use. But like, Rocco, are you seeing this in the same way I am? Like, is our, the average developer laptop, is the average developer laptop right now just running with this excessive agency as kind of the default policy?

25:04Rock LambrosI think, um, like, you know, every developer, I mean, you, I'm sure the 3 of us have always run into environments where developers have local admin and can do whatever they, whatever they want, right? That's auth, right? That's authentication. I think the difference between authentication and agency is authentication is, can the entity, the person, the agent, whatever, do the thing? Can it do it? It's pretty, pretty, pretty strict yes or no, right? With, with auth scopes and that sort of thing. Agency is more, should the thing be taking this action at this time?

25:39Robert HurlbutOkay.

25:40Rock LambrosRight? So, there's decision-making authority there, which by definition, an agent needs to have, or frankly, it doesn't need to be an agent. That's automation that we've had for like, I don't know, 5, 6 decades, right? So, if it doesn't need to make kind of these decisions on the fly beyond if-then-else statements, then it doesn't need to be an agent. And that's really the difference is the authentication being the can, the agency really being the should, right? And because it may, a decision may be valid in one condition and set of circumstances, but the same decision may not be valid in another set of conditions and circumstances. And that is the challenge with agents.

26:30Robert HurlbutSo if we think about the controls, if we, we stay on this excessive agency kind of path here. You've got controls that exist within the clients. If people are using Claude, Codex, Grok, whatever, as their primary client interface, you've got the provider controls and permission lists and things that can be adjusted. So that's obviously part of the excessive agency control path. It is, in my mind, agent-controlled if it wants to. So I don't put a lot of faith in what can happen on the developer's workstation. Like, I don't trust the developer's workstation because the agent is operating there and it could make adjustments to a permissions file if it went rogue, for example. But like, what are the other controls then on top of that? If, if We start with kind of that very basic thing that the providers give us. What are the other controls that we can layer on top to try to, try to, you know, prevent the agents from running away?

27:42Rock LambrosYeah. So the different providers provide us different things, right? From hooks to compliance APIs and everything in between, from spawning agents in the same process as the master agent or as completely separate subprocesses. And so another project that I lead over at OWASP is the Agent Control Standard. And our mission there is to almost create this like control middleware layer to where it doesn't matter what model or harness you're running underneath. You develop your controls once because Copilot exposes telemetry differently than Claude. And hell, Claude Across Claude AI, Claude Code, and Claude Cowork expose things differently in different ways. So by and large, so I'm a Claude Code junkie, right? A self-professed one. I use the hook ecosystem, right? So there are many hooks that Claude Code exposes to where you can inject Python, right? Deterministic mechanisms. What happens when an agent spawns up? What happens when it spawns up a sub-agent? What happens when that sub-agent completes a task? What happens when it tears down? What happens when it makes a tool call, right? I leverage in my harness, in the way I've configured my harness, those heavily. And I think that's why I'm excited about the agent control standard, is that hopefully we can simplify that across the board. And then it doesn't matter if you're running Claude underneath or something on Bedrock, or you're building on Agent Force, or within Copilot Foundry or whatever that is.

29:25Robert HurlbutYou mentioned something about the telemetry on Claude. And so I'm, it just made me start thinking, given that Anthropic is obviously embracing agentic development as the people that are building it, right? Is that a tell, the fact that their telemetry doesn't line up across, it doesn't work the same way across 3 different products within the same company that's only existed for a couple of years. Like, is that a tell about agentic development?

29:59Rock LambrosI don't know, because I mean, I think if we look at, we go back and look at Copilot and Gemini, right? Copilot, the Copilot ecosystem, and also the Gemini ecosystem, uh, over at Google, were different products that got acquired and like, patched together effectively, right? So forever, and even still now in some cases, they are different, they are completely different products underneath with just the same name attached to it. I mean, hell, I remember Google Gemini when it was Google Bard, right? Or one of the components was Bard, right? So I don't know if that is necessarily an agentic tell. It certainly could be. I think it certainly could be. But I think when you get organizations and product lines essentially racing, right, to get to market, I think you're probably gonna see that split anyway. Because I don't sit inside Anthropic, but I would bet the remit is go forth and prosper, various product groups, fail fast, fail often, right? Our cliché. and go and by the way, kind of let us know what you're doing, right? And maybe we could, we could share some stuff. I bet there's a lot of that going on still. Now, now pour in agentic coding on top of that. Maybe that just adds fuel to the fire. I don't know.

31:24Robert HurlbutThat's where I was kind of going because we don't have a lot of datasets for pure agentic companies other than, I mean, there's probably a handful of companies that you could say are operating at that level. right now where they've been, you know, where they're writing everything with agents and, you know, it's the model providers. There's a few other startups that are, you know, agentic software factory kind of focused and, you know, it doesn't matter who they are, but it just got me thinking like, is there a case study that we'll see in the future that will help us to understand what it looks like when you go pure agentic?

32:06Rock LambrosI don't know, man. I do scroll LinkedIn or Instagram enough, and I keep seeing these 2-person billion-dollar companies. Um, but then I don't really, I don't see it in reality. Uh, and I would love to see the first, I would love to see an actual unicorn exit that was purely run by agents. Like, like you said, like maybe 1 to 5 people kick it off, grows it to be a unicorn, exits. It's an actual legit product with customers and ARR. And, you know, has a, has a Wiz type of exit, right?

32:46Robert HurlbutYeah.

32:46Rock LambrosAnd, I think we're still a ways off from that. Um, I think the model, the underlying models need to get better first. I don't know. I kind of feel like I kind of feel like they're getting dumber with each iteration. And by dumber, I mean, you have to be much more specific. Going back to that prompt engineering, you have to be much more specific into the way you instruct it. Then it could be really good, but I'm seeing it infer things more versus validating data. And I think that's maybe hallucination by another name. Maybe it's hallucination by educated guess. And even when I challenge, let's say Claude on it, it'll say, oh, sorry, I inferred that based on XYZ. It wasn't actually something that I validated. Sorry, let me go validate this. So you have to be, I think it's becoming, you need to be very explicit into having it validate itself and having almost these cross-attacking agents. Right. To, I've got a set of skills set up myself that'll go in like adversarial, I call them pre-mortems, right? Like tell me how this failed 6 months from now. Um, and attacks it from different angles. And that does a pretty decent job. But again, there's token burn there. There's a lot of token burn there. So previously we had, uh, Rob Van De Veer on a, on an episode.

34:13Robert HurlbutAnd it was said that, uh, nobody agrees on what it means to sandbox an agent. So, could you tell us what should it mean and where does that control sit?

34:23Rock LambrosYeah. I love Rob. He's a dear friend. Like, I work with him over at the OWASP AI Exchange and he is doing the Lord's work over there. And I think the first thing that I keep coming back to is like, what is it? What's the AGIS code inside of a— how do you keep it within a box, whether they be a container, a VM, a You know, with no host mounts, because I've tried that, right? I've tried running my cloud code strictly in Docker and I've had to punch so much Swiss cheese into that Docker container back to the host to get it to be worth a damn to do anything that it's, that's almost worthless. Right. But really it comes down to isolation. We know how to do it, but because of those factors, I think not as well as we think. Um, I read some research papers this summer that kind of tested, let me think about this, tested the frontier models against these sandboxes, container sandboxes, and they found that they escaped through ordinary misconfigurations, like a privileged container, a writable host mount, stuff like that. Uh, or an exposed Docker daemon even. So like, I think humans are going to keep making human mistakes and then the agents are really good at exploiting those human mistakes. As we saw with the OpenAI Hugging Face incident. But then I think the second job is outside of isolation is deciding what the agent can actually reach and do from inside of that box. And that's, again, that's where the incidents happened this year. So no, nobody, like no human broke out of anything. No exploit attack like happened. But specifically in the OpenAI Hugging Face case, they, they, they determined that it would be easier to cheat on the test than to solve the test. So let's go find the answer. And they found a package proxy that they could get out on to go do it. Right. I would argue that's not bad agentic design. That's bad architecture. Um, now caveat, the models, like guardrails absolutely had to be lifted for this x. gym exercise that they were undergoing to see truly what the capability of those models were. We'll never see like how those guardrails were lifted, what the models were actually told to do, what little guardrails they were actually given. But regardless, that wasn't, in this case, the agents trying to be malicious. They just got out. They just found a way out. Now, when they got out, they started doing some really interesting things. Found a chat board, started communicating with each other on a German chat board, noticed that they were being, uh, maybe discovered, buried themselves deeper in that chat board, found a, found a, essentially, uh, I think a Kubernetes, like, management pod or something like that. that had, again, zero autonomy, but that had a lot of authority because it had the keys to the kingdom there. And it found those in it. So I would argue bad architecture on both sides.

37:34Robert HurlbutAnd the tension that exists here is we need the agents to still be productive and innovative.

37:42Rock LambrosRight.

37:43Robert HurlbutWe can't just put them in a container with no commands and say, do your thing. Like invent commands if you think you need one to be able to achieve some goal, you know, with a gateway available for TCP/IP travel. Well, you have to invent TCP/IP to be able to communicate. Like, we can't do that because— Right. They are so excellent at what they do and they're so innovative and they provide such great results. They have to be able to continue that. And that's going to be the tension that comes in when we start, when we try to plan how to control them. Is we have to keep 'em in the box, but not let them run away.

38:28Rock LambrosAbsolutely. I mean, you know, we play with clock code. How sick do you get? Like, you walk away, you think the agent's doing something, you come back an hour later or overnight and realize that an hour into the thing, it's sitting there on a prompt for you. Right. That's super frustrating.

38:46Robert HurlbutI've had that happen in the last couple days, so it's very frustrating. I

38:50Rock Lambrosagree. And even running it in YOLO mode now, right? Dangerously skip permissions mode now. Um, it'll still, it'll still trip on some things. And, and, and some of my hooks are set up such to where it's like, don't write outside the repo. And I've had to loosen those because it'll try to write to like, you know, /private or /tmp. And it's like, ah. Bad, but I can also see valid. And like, I just, I just want to get, I just want to get my work done. So will I accept that risk? And I'm sorry to say, 9 times out of 10, it's yeah, I'll accept the risk. I'll loosen the, I'll loosen the guardrails a little bit and maybe I'll get bit someday. And then maybe I'll learn my lesson and then maybe I'll realize it was a once in a 10-year event and be like, ah, I'm fine. Learn my lesson, tweak some things and keep going. I don't know, but that's the same approach we've taken with cybersecurity risk for decades, right?

39:45Robert HurlbutSo the OWASP Agent Control Standard, I'm assuming it includes, I haven't, I haven't done a deep dive on, I'm going to, as a result of our conversation here, containerization or sandboxing is a core component. What are the other pieces? That the Control Agent Control Standard is going to define?

40:07Rock LambrosActually, sandboxing isn't really a core component. You can run it like it's— so the way we've designed Agent Control Standard is we're very opinionated on the spec. Um, like what that control spec should look like. We're not very opinionated on how you implement it. So you want to continue, you want to sandbox your agents, you don't want to sandbox your agents. We're leveraging OpenTelemetry, right, for the observability piece. We're leveraging, I think we're up to 18 or 19 hooks that we have found common across the model providers right now, across the board that we're leveraging. We have also a more deterministic layer on top of an LLM as a judge model. using policy as code, like Rego and Cedar, to be able to give you rules. So that way, when an agent takes an action, like triggers on any one of those hooks, it'll, you know, come up with a determination, right? Allow, deny, modify type of scenarios. And then we're building upon that. So I want to be very clear. Yeah, we're getting a lot of interest in it, but it's still kind of a v1 standard. It's like a public preview standard. Um, there's a lot of activity going on in the repo right now. I would not experiment with it. Please go break it. Tell us how to fix it. Issues are welcome, but don't like go throw this in your, in your production environment yet.

41:40Robert HurlbutSo the end result of this work though is an open source project that will do these things. for you? It's not just a— or is it a spec? Is it something that's like a set of guidance that I use, or is there actually something I can install later when you get further down the road?

41:59Rock LambrosIt'll be something that you can install later down the road. We actually have a— if you go to the repo now, there is a reference implementation that we built on top of the Microsoft Agent Governance Toolkit because they both attack kind of different layers in the stack. So there's an actual working model in video of how that works in the repo right now. We're going to build an SDK around it as well, right? So you'll have like a Python installable package. So yeah, yeah, I mean, there's a lot on the roadmap. I'm pretty excited about it. I'll just say stay tuned and get involved.

42:34Robert HurlbutOkay. And I want to, I want to make sure we kind of wrap up our conversation here with something that's very actionable. for our listeners. And so we have the OWASP Top 10 for LLM. We have the OWASP Agentic Top 10. If we've got a listener who's an AppSec person, maybe they're running the team, maybe they're part of the team, where would you recommend that they jump in across these different things? Like where, where can they get a quick win? I guess.

43:04Rock LambrosThis is going to sound boring. And familiar. Inventory. Know what agents are running in your environment and know what every agent can reach. Understand, I call it like a reachability budget, or it's a blast radius. Understand what an agent can actually do, what decisions it can actually make, not what you think they can do and think they can make, right? And, you know, do you actually have a kill switch? that you have tested, right? That will shut down the entire line. What happens? You kill switch an agent. What happens to its sub-agents that are spawned off and operating and other sub-processes? Right? Like, so like, uh, just like, you know, the back tried and true inventory of what you have. And then I would say, look into putting deterministic policy checks in front of the tool call, right? That you keep that like are critical and that you cannot undo. Outside of the agent's process, right? And this is where the agent patrol standard, I think, is very beneficial because it rides almost as a sidecar, if you will. Not a technical sidecar, but almost like a sidecar. It does not ride within the agent process. It is coming outside of the agent process. So the agent can't tell it to ignore itself, right? And then The LLM risks, the LLM list is not the end-all be-all. And while we did rank it kind of 1 to 10 by what we think is, you know, priority severity, it ultimately comes down to your organization and your organizational context, right? Rank it with what you're seeing within your environments. Determine which of those entries lead to the worst outcomes. for your specific use cases, and then go invest in those. Again, with the assumption of prompt injection is just going to happen.

45:03Robert HurlbutYeah, it's very helpful to kind of put that in perspective and give people some things they can start with. And yeah, I think most organizations do not have that inventory at this point because, I mean, this is the Wild West that we live in and it's only roughly 9, 10 months old, whatever it is at this point. So, Rock, I could literally talk to you for hours. I just learned that's what I learned in this conversation. And so thanks for joining the podcast. Thanks for sharing your wisdom. And we'll definitely have you back on in the next 6 months just to pick up where we left off, because I know you got a lot more to share with our audience and we look forward to that.

45:39Rock LambrosAnd there'll be a lot more in 6 months to share. I promise. If the last 6 months have taught us anything, will be that we are in for a ride over the next 6 months.

45:48Robert HurlbutYeah, hold on.

45:49Rock LambrosOr we'll be dead. AI will have killed us all by then.

45:52Robert HurlbutI hope not, but hold on, I guess. Hold on, folks, because it's going to be a wild ride and the next 6 months are going to be probably even more wild than the last 6.

46:03Rock LambrosYep. Thank you, guys.

46:06Chris RomeoThanks for listening to the Application Security Podcast. If you enjoyed this episode, subscribe, leave us a review, and share it with a friend or colleague who would enjoy the conversation. We'll see you next time.