Skip to content
AppSec PodcastThe Application Security Podcast — home
52 minSeason 13, episode 21

Trust No AI: Johann Rehberger on Breaking, Watching, and Killing Agents

With Johann Rehberger

Security TestingAI and LLM Security

Johann Rehberger has spent years breaking AI agents, and he still won't let one run autonomously on his own computer. The red teamer behind Embrace The Red joins Chris Romeo to explain the rule he lives by: trust no AI.

Audio hosted by Buzzsprout. Nothing loads until you press play.

Sponsored byOriginEndpoint AI observability. From prompt to outcome.Learn more ↗
Episode chapters · 20 chapters
  1. 00:00:00- Cold open: assume breach, trust no AIAudioVideo ↗
  2. 00:00:17- Meet Johann RehbergerAudioVideo ↗
  3. 00:00:48- Welcome, and Johann's AI security origin storyAudioVideo ↗
  4. 00:05:38- How much should you trust AI agents?AudioVideo ↗
  5. 00:07:20- Models, providers, and where your data ends upAudioVideo ↗

About this episode

Johann Rehberger has spent years breaking AI agents, and he still won’t let one run autonomously on his own computer. The red teamer behind Embrace The Red joins Chris Romeo to explain the rule he lives by: trust no AI. Johann traces his path from building red teams at Microsoft and Uber to the Machine Learning Attack Series, then makes the case that models are probabilistic systems you have to evaluate, isolate, and monitor. He explains why the classifiers inside coding agents still miss real accidents, how he caught a GLM 5.3 agent trying to SSH out of its sandbox at 3 a.m., why a separate sentinel should watch the logs, and why Docker was never a sandbox. Plus: four things every CISO should do before handing developers coding agents, and why the normalization of deviance is AI’s quiet risk.

This episode is sponsored by Origin. Let your agents work. Keep them in sight.

About Origin
Origin is an endpoint AI observability platform that inventories, observes, and measures the usage of AI agents at scale. It gives organizations a record of what their agents do across endpoints, so security teams can monitor agent activity, investigate unexpected behavior, and enforce their policies.
→ Learn more about Origin

This episode is sponsored by Corgea. Design it. Build it. Ship it. Corgea secures it.

About Corgea
Corgea is an AI-native application security platform that secures software from design to production. It brings together security design reviews, AI SAST, dependency and IaC scanning, code quality checks, and autonomous pentesting—helping security and engineering teams find risk earlier, fix what matters, and ship securely.
→ Learn more about Corgea

Connect with Johann Rehberger:
→ Johann Rehberger on LinkedIn
→ Embrace The Red
→ Cybersecurity Attacks: Red Team Strategies

Resources
→ Machine Learning Attack Series
→ Agent Security is a Systems Problem
→ Trust No AI: Prompt Injection Along the CIA Security Triad
→ The Normalization of Deviance in AI
→ Redesigning source control at Anthropic (Scion)
→ The Grok Build CLI upload incident
→ GLM 5.3 from Z.ai
→ NorForge: Inside Cairn

Actionable

From this conversation

  1. Re-run security evals against your own code whenever models change

    Build baseline test cases from your own past security issues and re-run them every time a model is upgraded, downgraded, or rolled back, since a newer model will not automatically perform better inside your harness and context.

    10:47
  2. Fix security fundamentals before layering AI defenses on top

    Get basics like MFA, egress control in production, and binary allow-listing in place first, then add AI to scale monitoring and detection work — adversaries and agents alike will try the easy paths first.

    17:19
  3. Treat model output as untrusted and control the worst downstream outcome

    Approach AI with an assume-breach mindset: treat the model's output as untrusted and place effective, understood controls on the downstream actions so you bound the worst possible outcome rather than relying on prevention alone.

    23:22
  4. Isolate autonomous agents and export their logs for outside review

    Run serious agent work on an isolated machine instead of your primary computer, and ship logs out of the VM so a separate monitoring process can analyze behavior — a watcher agent with scope boundaries can even terminate an agent that goes off track.

    31:52
  5. Roll out coding agents with inventory, paved road, monitoring, and red teaming

    Before letting coding agents loose, inventory what is already running, build a secure-by-default paved-road option that is the easy choice, invest in monitoring that observes agent behavior over days or weeks rather than single actions, and use a red team to validate your assumptions.

    46:05
Transcript · 52 min conversation

0:00Johann RehbergerMy core phrase is always, like, coming from assume breach mindset, like, trust no AI, right? You know, the output is untrusted, and the downstream implications, you need to control those with, like, effective controls that you understand, so you control the potentially worst outcome.

0:17Chris RomeoJohann Raeburger is a red teamer and security researcher with more than 20 years of breaking things for a living. He became a security engineer at Microsoft, where he built the Azure Data Red Team. He went on to build a red team at Uber and to serve as red team director at Electronic Arts. He wrote the book Cybersecurity Attacks: Red Team Strategies. While writing about attacking AI for that book, he realized he didn't really know how machine learning worked, and he has focused on AI security since 2020. His rule for AI is simple: trust no AI.

0:48Robert HurlbutWelcome to another episode of the Application Security Podcast. This is Chris Romeo, and, uh, joined today by Johann Rehberger. And this is going to be a real treat because I had a chance to dive into a lot of the things that Johann's done over the last couple of years. And I'm going to be honest, I'm super excited to get some perspective on the things that he's done, things that he's learned and whatnot in the world of AI. So Johann, I'd love to start by your AI security origin story, which I realize is going to flash forward a little bit from probably the total length of your security origin story, but I'm just curious how you how AI got your attention and then what the path has been from there.

1:33Johann RehbergerYeah, sure. First, yeah, thanks a lot for having me, Chris. Excited to be on. My origin story in AI. So let me first tell you one sentence about my security origin story that goes back like 20 years, but started at Microsoft. That's sort of where I became a security engineer. My machine learning security origin story is actually a lot more recent. It started In the summer of 2020, when I was like, I had written a book about red teaming, sort of putting all my thoughts into how to build a red team, like an internal red team within a company, which I did at Microsoft and Uber. And there was one part in the book that I had written about AI, that AI can be used for attacks, but also for like, you know, AI itself can be attacked. And then I realized, oh, this is the one thing I've never really understood in my career, right? How does machine learning actually work? So then I went off in a long journey, taking a lot of classes and courses and reading books about understanding machine learning, the way it actually works internally a little bit better. And then I realized that there's a big gap between the machine learning world and the security world, that a lot of the machine learning researchers and so on, they do not really care about security. And sort of, I realized that there's this gap in between. And then I started writing a blog series called the Machine Learning Attack Series. I think this is what really kicked off my passion around sharing what I learned in that space. And there were a lot of fascinating things I learned. The one big one that just blew my mind back in the day was like an attack called image scaling attacks, where you take a large image, making it by compressing it or making it smaller, rescaling it, it becomes a fundamentally totally different image. And these things are just like really, really fascinating. That was part of that series. And then pickle deserialization and just regular threat model things that you wouldn't, machine learning researchers usually wouldn't think about, you know, how you protect your model, that you should probably be signing your model before loading it in production, right? Did you know you're loading the the correct one and so on. Yeah, I think that is sort of my origin story in the machine learning world. Back then, there was not that much talk about large language models. Back then, it was all more about image classifications and so on.

4:02Robert HurlbutAnd then how did you make that jump to the large language model space? Like, where were the models when you made that jump? Were you there right at the very beginning? of the GPT kind of series, or did you get in a little bit later?

4:16Johann RehbergerYeah, like I was there before ChatGPT. I wrote one blog post with GPT-3 maybe 6 months before ChatGPT came out where I wrote the entire blog post with GPT-3 talking about how AI can now be used for basically creating phishing emails, and it sounds— it seems really realistic, right? You would not know the difference. And actually the blog post at the very end, the last sentence was, None of this was written by me. And that was actually 6 months before ChatGPT released. So the model GPT-3 was already really good.

4:48Chris RomeoThis episode is brought to you by Origin. Quick question, would you hire 1,000 people without knowing what they can access or what they're doing all day?

4:59Robert HurlbutOf course not. But that's how most companies roll out AI agents. Today, you get the prompt and the final answer. Everything in between is gone when the terminal closes. Origin finds the agents on your endpoints and rebuilds their work. Every command, every file, every handoff between agents on one timeline. Origin endpoint AI observability. See what they found in their own traces at originhq.com/appsec. That's originhq.com/appsec. Nice. So I know that you've spent a lot of time looking at agents, breaking agents, and so I wanted to start at a, at a pretty high level here and just get an understanding for how much are you trusting these things right now in towards the end of 2026?

5:56Johann RehbergerThat's a good question. As a security engineer, I trust nothing. Yeah, me neither.

6:03Robert HurlbutMe neither.

6:04Johann RehbergerBut that doesn't get you very far, right? I think that's always the dilemma, right? I think you need to be skeptical, aware, and fundamentally realize that we do not understand how these systems actually work, right? And even though the robustness over the last 4 years, the robustness in large language model has increased drastically, right? They appear more trustworthy, but that also is something that I call a normalization of deviance, that we potentially started trusting them where we shouldn't actually trust them. So the core principle, I think, still remains for me that I don't trust them or have them run on a dedicated machine, not having access to my own information without explicitly granting it access and so on. So that's my stance, and I don't think I will probably change range that stands for a long time, until there's really some breakthrough happening in the machine learning space that demonstrates, or yeah, some breakthrough that just makes it more trustworthy rather than just empirical repetition and normalization that we trusted more over time.

7:19Robert HurlbutSo is it the model itself that kind of holds you up from being able to say, like, I'm going to go, I'm just going to fully trust it? Is it the model provider? Like, what's the concern on your side right now?

7:35Johann RehbergerThat's a good way to look at it. I think there's multiple facets to that, right? First, you have the model itself. So even if you would have the model hosted yourself and run it yourself, right? At least you would know the model itself doesn't change, which is currently a big problem, right? You might run an inference and you, if you use a SaaS provider, right? If you use OpenAI or Anthropic, you actually don't know in reality. And I think we have sometimes observed that even that models might be busy or one model is busy, so they serve the request with a different model. And so you don't really, really actually even control which model is actually going to be used in the end because you have no insight, right? I think it's stopping a lot of that. And then the data control, like who actually looks at the data, right? You can see that every single prompt you have is probably reviewed by some agentic system for certain things, right? And we don't know about where the data ends up, where it is stored, how long it is actually retained, right? Companies always give these promises, right? 30 days or something, but in reality, we don't know. And with less mature companies, it's also more likely that there's some problems in the whole process of these having these security controls actually in place, right? I think that's also one example, for instance, with SpaceX AI, right? They had this bug where they would just upload— if you launch the command line tool for Groq, it would upload your entire— I think it was not just even the folder you were in, even data outside the folder you were in, but they just uploaded everything to SpaceX AI. And that was just a bug, right? This was not intentional. I believe that it was actually not intentional because it's just probably how fast, when systems get built really quickly, right? And pushed out with a lot, with less quality assurance, then you encounter bugs like that, right?

9:32Robert HurlbutYeah, because I mean, these systems are being built by the same systems. Like, it's, you know, we've seen a bunch of things in the last couple of weeks, you know, Anthropic comes out with their new Scion source control system that they've stated that Git isn't fast enough for them. So they took influence from Meta and Google and came out with this new system. But they said they built the whole thing on Claude with 7 engineers. Something that you look at the scale of, like how many hundreds of engineering years go into making a source code control system that runs across the network. and doesn't just screw everything up.

10:16Johann RehbergerYeah.

10:16Robert HurlbutLike thousands of engineering years went into that and they did it with a team of 7.

10:20Johann RehbergerYeah.

10:21Robert HurlbutBack on the point that you said about the models though, it just made me think about, I saw as like some of the providers are launching new, getting ready for new models, they start serving up the new models like in a family, they serve up a later version of it, but it doesn't, it doesn't tell you that it's using a later version of it. So to your point, like we're not even really sure what we're using. on the other side. There's no checksum that we can get back.

10:47Johann RehbergerYeah. And some of that can even be exploited by an adversary, by an attacker. Like, even like, this is one thing I really want to highlight why the guardrails and the downgrading is really problematic because you can, as an attacker, you can have the model get downgraded and then your attack succeeds because it works against the weaker model that is less robust, right? So these downgrades are actually very dangerous, I think, even. They do not provide the security guarantees you might want to have, right, in certain And additionally, even, or security guarantees is maybe even the wrong word, but they don't have the quality guarantees that you would expect or the robustness guarantees. But I think really what it comes down to, too, especially for security engineering and how our world is changing, is that we are entering a world that we are all probably less familiar with, which is sort of this world of we need to explore and experiment. Meaning, if you have a CI/CD pipeline and you have a model in that pipeline that does a review, right, how often are you actually going to run that review? Is it just one time, or are you going to run an experiment, you run it 100 times and see what is the difference? And then when a model change is happening, just an upgrade or downgrade, are you going to rerun your experiments, your evaluations, right? Do you get a better performance or a weaker performance, right? There's a lot of baselining, I think, that everybody needs to to do to get really the best out of these systems and not just say, oh, the next model is going to be better because it might, however your harness works and so on, the surrounding controls, the context you provide, right? A newer model might not always work better, right? Or even the same model might be degrading over time if they change something or roll back to a previous checkpoint and so on. Yeah. So there's a lot of considerations, I think, for, but this is really how our role as security engineers is going to change, having that thinking for experimentation and evaluations. I think that is something we all have to get more mature in.

12:45Robert HurlbutAnd that's not our normal approach to the world historically. I mean, I've been in security—

12:51Johann RehbergerWe're used to running a Semgrep and it's going to give you the same output.

12:54Robert HurlbutWe like our determinism, right? Like, we want to run it and get the same result every time. And, you know, I've been doing this for a long time and I can't think of a time more filled with disruption than any time in my 30-plus-year career.

13:12Johann RehbergerYeah.

13:12Robert HurlbutLike, you can say cloud was a disruption, but it was a slow-burn disruption because we didn't all jump on board with it in a 12-month time period to the point where companies are pushing code into production now from agents at high speed that last Christmas they didn't even have an agent. They didn't have any license for the tools. Yes. And now here we are towards the end of 2026, and people are pushing into production at full speed with code that hasn't been touched by humans. And so, yeah, this is the fastest. But yeah, to your point, as security engineers, we gotta find our place. We have to fight for our place, because if we don't, we're gonna get pushed off to the side. And security is the people that are standing up and saying, like, to your point, like, understanding evals, That's something that is a, you gotta, you gotta put some time on the machine. It's not something you just read a blog post and go, okay, I understand evals now. No, you gotta run your own. You gotta, you gotta test these models in different scenarios. You gotta look at data. You gotta become a partial data scientist, partial infrastructure engineer, AI engineer. Like you gotta, plus being a developer, you gotta put all these things together into one place.

14:29Johann RehbergerYeah. Especially like, within a company, right? Because your own code, right, that's what you need to evaluate against, right? Like, you have like the top 10 past security issues and which model can find it, will it find it again in the future, and so on, right? There's just basic test cases basically to rerun when you adopt a new model. But at the same time, right, it's all super exciting and I think we all really love diving into like exploring what agents can do, what problems we can solve with them, right? I think There's a lot of upsides too, to that whole story. And I think especially for security, it can be a big uplevel, like in all aspects, right? One, unfortunately, we don't talk as much, which is the defense aspect, right? But I think it can help a lot in the defense space, which we should talk a lot more about. I think everybody's focused on attacking with AI, but I think defending, usually that's normal because attack always drives defense, right? That's kind of happening typically.

15:25Robert HurlbutThat's been the cycles that we've seen. And I mean, I was around when the first pen testing wave happened, where pen testing became the most important thing to almost, to a lot of people all the way up into leadership. It became the thing that I used to play this fun game where I'd ask like a computer science student studying security, you know, what do you want to do? What's your, what's the job you want when you finish university? And for many, many years. Yeah. The answer was always, I want to break stuff. I want to be a pen tester. I want to be a red teamer.

15:59Johann RehbergerIt's a good pick.

16:01Robert HurlbutBut it's always led. It's always been the leading, like, no, we never had a time in our, in security's history where it's like, people were like, I want to be a defender. I want to do the things that you have to do right all the time to make sure the door doesn't ever open for the human, the model, the agent, you know, whatever we're dealing with into the future. So yeah, defense is I think it's going to lag behind. I think it is lagging behind right now, but there'll be a catch-up period.

16:29Johann RehbergerYeah. I was posting about this the other day. I was like, where is the frontier blue team, right? I'm really kind of hoping one of the labs is at one point standing up a frontier blue team, right? That shares and highlights playbooks and runbooks and how they build out the defenses and monitoring and sandboxes. I mean, there was recently Anthropic did actually share some insights, I think.

16:50Robert HurlbutBut yeah, I mean, from your perspective, is, is how do you keep up with an agentic group of, you know, as the attackers go agentic, we're already seeing them, right? It's happening. It's happened already. It's, it's going to only rocket ship from here forward. Like, how do you use this new technology on the blue team side to, to counter this, what I think is a tidal wave coming towards us. We just don't know it yet.

17:19Johann RehbergerYeah, I think the important thing to say is to not just jump on AI and have AI do everything, right? I think there's a lot of basics that every company should look into first, right? You have MFA, you have egress control in production, do allow only allow binaries to run in production, right? There's a lot of basic security controls, I think, that you need to have in place. And then when you have a very strong foundation, then when it comes to monitoring and detection, that's when you can then add on. This is what I want to really highlight. You need to have the basics, and then you add on AI on top of it to scale the work, right? Because if you have an agentic attacker, right, the logs are going to be so massive, right, that it's going to be difficult to process and handle the logs. But then you're— so, defendant, this is really the things that need to be figured out, right? How can you actually defend your company if these hosted models refuse your requests for defense, right? You ask to analyze a log file, the log file contains some CBRN content or whatever, right? And the model will just stop responding. Or even easier, right? Just a security topic, right? It will just stop responding, right? And that is not helpful if you are in a crisis, right? If your company is currently breached and the adversary is moving laterally, right? you need a defense capability that is guaranteed to function, right? And I think this is a big problem currently, but even just over the weekend, right? I think it was Open Code. They had the same problem that they could not use OpenAI, I think GPT. And then they got luckily daybreak access, so they were unblocked to some extent, but still not, it's just still not like 100% work all the time, right? So there's a lot to figure out and what it means for open weight models too, right? Or obliterated models that can actually— and this is weird to say, right? That you need to then go somewhere to a potentially less trusted source. Who knows, right? You get an obliterated model that might have all sorts of backdoors. Who knows? That does invoke certain tools on your machine if certain conditions are true, right? It's not a good place right now. Where I think open weight models should probably be looked into a lot more that we get a better defender ecosystem that is healthy and not just depending on centralized choke points.

19:46Robert HurlbutDo you think the open weight models are like 6 months behind, like I've heard a lot of other people claim, or are they better than people are letting on?

19:59Johann RehbergerI think like GLM 5.3, right? A lot of people say it's probably 6 months behind or so, right? I don't have like a, I never really ran an eval myself on that, but it's pretty good, but it loses if the context window gets large, it just starts malfunctioning. It's just, it's not robust. So you cannot actually, that is my experience on how I used it, right? Yeah. that then eventually it starts just derailing and getting off track if the context window grows large. But these are the kind of things you need to know, right? How it impacts your environment, your data, how that impacts the model performance, actually. Yeah.

20:38Robert HurlbutSo specifically on the topic of steering models, I'm seeing a lot of vendors who are building steering products right now. There's certainly a lot of vendors that are building governance through various different approaches. But the one that I'm curious from your perspective is this idea of, if, is, can we steer the agent towards good security by, you know, however we're able to inject ourselves into that process? Like, is that, is this even possible based on what you've seen?

21:13Johann RehbergerI think you can steer it towards like better security, but it doesn't mean it's like 100% better all the time. But I think you, there's products, right? If you inject yourself in certain parts of the pipeline, right? The outcome is going to be better from a security point of view. If you then can somehow validate, ideally in a deterministic way, that the outcome is actually what you want it to be, right? That would be ideal. That you directly have the agent plugged in somewhere in the CI/CD pipeline or even earlier on the developers' Actually, now we shouldn't even say anymore on the developer's machine, but on whatever the coding agent's machine is, right? If you have that plugged in, right, it might be able to allow only code to get out in the first place that has passed certain basic security properties, right? Or has certain properties in place. And I think that is really an interesting place to look into, actually, right? There's often, speaking more philosophical, something I think a lot about as a red teamer too, right? world is always to think about the world upside down, like the things that you just cannot comprehend. They just need to come up with these scenarios. But I sometimes wonder, is there a world where the model is going to be a better security control than anything else, right? In my point of view right now, I cannot see that world, but could such a world exist somewhere down the road? I have no idea. I mean, I don't think so currently. But who knows, right?

22:46Robert HurlbutIt's a good, it's a good thought experiment though. You'd have to have, it'd have to be a lot more performant to be, you'd have to have near, you know, real-time response and whatnot, which I don't see any reason why we're not going to get there, where things aren't going to speed up over time to the point where I'm not spending time watching the pinwheel spin, waiting for the agent to come back. And so we're going to get to the point where it's like, bing, bing, bing, bing. It can go faster than we can type, faster than we can think. So that's kind of the first hurdle to a fully model-driven security control.

23:22Johann RehbergerYeah. It's a world that is possible, but also scary, right? I think this is the big thing that, because of the lack of understanding how these systems work, right? This is always where I come back to, where I have to remind myself, right? My core phrase is always coming from assume breach mindset, like trust no AI, right? Just to highlight that the output is untrusted and the downstream implications, you need to control those with effective controls that you understand. So you control the potentially worst outcome, right? And that is the thinking that drives my strategies.

24:01Robert HurlbutI think that's the best possible way we can deal with these things at this stage. I am also baffled by the fact that I can't find anybody who can explain how these things work to me.

24:16Johann RehbergerYeah, it's fascinating. It's less— more like, it's a lot more like discovering something than actually inventing something, right? I think there's a big difference between computers, hardware that we know and we engineered, versus in this case, we are discovering a new kind of technology, right? It's not something that we invented in that space, right? It's literally like fire. We see fire and then we start playing around with fire. I mean, and then figure out how to control fire, right? To make it useful.

24:48Robert HurlbutYeah. Nobody wrote the code for fire.

24:51Johann RehbergerNobody wrote the code for fire.

24:53Robert HurlbutTo make it start. It was, uh, it's an organic kind of process there.

24:59Chris RomeoThis episode's sponsored by Corgia. Design it, build it, ship it. Corgia secures it. Corgia is an AI-native app security platform covering your whole software lifecycle, from the first architecture diagram to code in production. It brings design reviews, AI-powered code scanning, and autonomous pen testing into one platform, so your team spends less time chasing noise and more time fixing what matters. Visit corgia.com. C-O-R-G-E-A.com. Corgia, one security platform for the entire SDLC.

25:34Robert HurlbutSo, all right, let's talk about classifiers. So, I made a statement on a recent episode talking to a guest about AI, some other facet of AI. We were talking about the classifiers, and I think I inadvertently or accidentally made the statement that the classifier would catch a delete everything. And so I know you've, you've had some experience working with Claude Code's auto mode, you know, steps that looked like they should be harmless, ultimately turning into harmful actions and using ChatGPT together with Claude to trick stuff in. Like, what is the state of the classifier right now? Like, like, should I trust that it's going to block a delete everything, like an agent trying to do some massive deletion operation, or how much should I trust it?

26:25Johann RehbergerYeah, that's where my trust is maybe a little less strong at the moment. And that's why I still don't fully autonomously run an agent on my own computer, right? You've just seen recently, there's just even without an adversary that tries to do something bad, but you see these accidents, maybe, or mistakes where it's just— I think there was the last example I saw was because I guess a lot of the training data is more focused on Linux or macOS, I don't know, but it turns out that on Windows, I think, I don't know which model it was now, which agent actually, I don't remember that, but there was a quoting issue in a PowerShell script. We just used the wrong quote and then it terminated a string and then it ended up, I think it was probably Program Files and then terminated somehow the string and then it just deleted the whole, like, something way more than it should have. And it was literally a quoting problem, right? And so these are accidents where the classifier also did not catch it. And with the classifiers, I think now that we see a lot of— if there's anybody listening right now that is looking for a research project, I think we have not really seen evals being run against the decision models, how resilient they are to attacks like indirect prompt injection, right? If you think about where we were 3.5 years ago with indirect prompt injection, there was literally no robustness. All the models really just ignore previous instructions. In any really frontier model, this is becoming a lot more difficult. So there's a lot of work with all the sometimes embarrassing examples the industry demonstrated to the model providers. invested a lot of time and effort to make that more robust. So we have improved there a lot. Still not perfect, but I wonder right now how the decision models actually— how resilient they are to indirect prompt injection. I don't think— I'm not aware of anybody having run a broad-scale eval on that, taking one of the benchmarks from the regular LLMs and then trying to adapt them to decision models and see what the outcome is, right? Because a lot of— I think a lot of people will adopt those decision models now because they're really fast, they're really good, right? Really quick. You get a yes/no answer or whatever you want to kind of select as the answers. And they're ideal for classifiers. So I'm very curious about how that space is going to develop in the future.

28:57Robert HurlbutAnd so in the paper, newer IPS paper that you were talking about, treating the model as untrusted, And we've certainly had, we've certainly discussed that a couple of times here. It's, it's kind of come into the conversation. I'm just curious, like, how do you practically put that into that, that concept into life? And I know you've said you don't run agents on your primary computer, but what else do you, what, what else are the things like from any facet, red team, blue team, AppSec, how do you treat it as untrusted? but still get value from it?

29:35Johann RehbergerSo going back to that paper and the positioning paper, that really kind of all of that thinking even goes back to my realization that I had 6 years ago when I started looking into ML, where I realized that machine learning research is focused on the model and only on the model itself. That was like this whole thing, but whenever somebody takes a model and builds a product, what actually matters is not the model by itself, but the system, right? This is really the thinking needs to be on the system level. Like, what is the input? How you manage? How do you deploy? How to patch, right? How to take it online, offline? How to, yeah, just all the ecosystem around it, right? And I think this is a big part of why security engineers are ideally positioned because we usually think in like systems, right? Especially red teamers too, we usually think in a holistic view beyond just the core piece of the system, right? Yeah. And then to your question around using agents practically, right? I do use agents actually on my main computer, but I usually do not have them running autonomously, right? I kind of observe what they're doing, Or I put them in certain sandboxing modes where over time I kind of have developed certain monitoring capabilities too, where I could watch what happened and analyze what happened and so on. But that use case, usually use case is very limited. When I really do anything serious, I just have it isolated. I just have a computer where I just have actually multiple agents installed and they run sometimes concurrently. But I just have them run and then work on that. And the way I communicate to them is either by sending messages to them, or, you know, sometimes I have like certain communication channels that I do allow to have being open. So it's a, in a way, it becomes organically, you adapt and adjust the security controls around it to fulfill your needs.

31:47Robert HurlbutSo yeah, it's kind of your own personalized security program?

31:52Johann RehbergerYeah, like a very interesting thing. I've been experimenting a lot with like monitoring of agents and it happened like when I started using GLM 5.3 more that I was like, okay, I need to have some kind of monitoring in addition to all like, you know, the security controls I have in place. But I want to also see, right, theoretically the agent should not be able to get out and laterally move to another machine, right? That should be blocked because there's no SSH, it doesn't have the keys to to move around and so on, but who knows, right? So I was like, okay, let me just put— this is very naive and basic in the beginning, right? I think this is what I mean, but we need to experiment right now a lot, figuring out what works, what doesn't work. But I just had a Codex session then open and then ask it to look at the logs of GLM and tell me if it ever goes off track. I gave it sort of the in-scope IP addresses. And then it noticed one moment where it actually attempted to SSH to the gateway, which was very interesting, very out of the blue. It might have just been a hallucination or some glitch because that's when I noticed GLM becomes— the performance of GLM becomes less robust the larger the context window is. And that's sort of where I was like, okay, this is interesting that There's a way, of course, if you look at the logs, you would also detect that there's an attempt happened. But in this case, you just then in addition to that, you have an additional system that I would call it like a sentinel, right? That just looks at the logs. And of course, this system can be improved a lot. And I really hope that we have products in this space at one point where the logs need to go out, they need to be signed. Of course, all the prompts should actually be signed, that then on a different place, you can actually— consume them and analyze them out. You shouldn't be analyzing the logs necessarily within the same VM where they were generated, right? You want to get them out, then have a holistic process outside the VM that reviews what actually happened and so on. I think there's a lot of innovation that is still needed to build such monitoring and observability observing systems.

34:03Robert HurlbutSo is there at this time, When you have an agent potentially misbehaving, you have a different agent that's completely isolated from that agent who's acting as your sentinel. I don't feel like there's a big threat of corroboration between those 2 at this stage.

34:27Johann RehbergerI mean, these are the things we need to really— these are the right questions, right? I think this is actually the thing where if you get too red-pilled with AI, you just try to solve every problem with AI and that eventually will probably not be the right thing. And I think this is exactly one of these scenarios. Could one actually then tell the other one, hey, do something totally different, or hijack the other one basically? And these are real concerns we should actually be having. And so that's why I really like that question because that really highlights an understanding of the complexity and us not understanding how these systems actually function. And the action, for instance, the one action that the agent in my case here is allowed to take is terminate the other agent. When it sees—

35:17Robert HurlbutSo the sentinel can send a kill switch basically.

35:20Johann RehbergerYeah.

35:21Robert HurlbutOkay. And now did it send a kill switch on that?

35:24Johann RehbergerYeah, it terminated the— this is actually how I noticed because this happened at 3:00 AM in the morning. I was like, So in the morning, I go to the computer, just check what's happening with that research, and then I see that GLM is not running anymore. And then I asked ChatGPT or Codex, why is GLM not running? And then it told me because it went off, out of scope. So that makes you also realize how these systems can, like, could function, right? But there's always the concern that there is some, there's just a lot we don't understand, right? That's right.

35:54Robert HurlbutDid you ever, were you ever able to draw a conclusion as to why that particular agent went off the rails?

35:59Johann RehbergerIt was, my understanding is having looked at exactly the commands it sent, like leading up to that one SSH command, it was that it just slowly started getting off track where it just made, it started hallucinating things. And then it started actually trying to probe local files on the computer itself, like folder names that made no sense, like 200-character long, like home names of user folders and so on. And yeah, and then it just went totally like off track. And then until it then attempted to do something off the machine, right? But locally on the machine, it was already doing like strange things for quite a while. For like the last 10 prompts were probably already all, I would say garbage actually, like totally gibberish.

36:47Robert HurlbutDo you see the same behavior with the frontier models?

36:50Johann RehbergerI have, this is a good question. I have not, not to that, to my memory. I sometimes see strange things happening, but I have not seen, I mean, this happened a long, long time ago. There was really one really funny thing that was like, oh my gosh, obviously this would happen, but it was just like, I had a VM set up somewhere and I think even had maybe the agent create the VM or something and then had the other agent just seeing if it can like It was about a Linux kernel exploit I wanted to kind of research and build. And then I gave the goal, hey, become root, right? And it's just the first thing. And it was totally unexpected to me, but it was really hilarious in a way because it just brute-forced the password rather than having the goal. I gave it the goal to, you know, use this sandbox escape that I had found in PipeWire. And then it just actually just started brute-forcing root. And it was like, comes back an hour later, I'm root. I found the problem. I found— And what happened? It was like, oh, the password was really weak. So that's like really just an unexpected path that led to the same goal, like to the same objective. Right.

38:00Chris RomeoAnd that's a fact.

38:01Robert HurlbutThat's just a fascinating case when something like that happens, because the general public will immediately go to, oh, it was malicious in nature. But we know it's not malicious because it doesn't have the ability to be malicious. It's goal-driven.

38:19Johann RehbergerExactly right. It is a machine and it does not have this ultimate objective. It just computes, right? And that's what happened here too. It just computed what it thought was the best path.

38:32Chris RomeoYeah. And that was the least path to resistance, which honestly, we've been doing, humans have been doing that for a lot longer than machines have. Yeah. From a renting perspective, if you have 2 options, one is going to be easy, one is going to be complicated. You're not taking the complicated path. You're booking the win. Let me just take it, book the win, and be done.

38:54Johann RehbergerAnd this is really true and sometimes also dangerous right now, I think, with all of the news, but we all jump really a lot on AI news, a lot, right? But I think adversaries usually always take the easy path, right? And there's a lot of really easy ways to do things often, right? And we should not lose track of, that's why earlier I said, right, first get your regular security like in good shape, right? The things we know already that actually help and secure systems, right? Get those in place because, you know, when an agent comes, then it will try those things first because that's what everybody is going to do because that's the easiest path to get into computers, right? And the very really What worries me a lot is the ordinary small business, smaller companies, right, that do not have big security budgets that might be eventually run over by accident, right, which we have seen in the past already, right, with drive-by attacks and so on. But that might just be when the— Mm-hmm. When less sophisticated adversaries or have means to do certain things, right? They might actually do it, right? More like we, in the past, we talked about like script kiddies, right? But I think that is a real like empowerment for people that just play around and maybe even by accident then cause big damage, right? And AI is definitely something like that, that could easily, even by accident, lead to a big problem.

40:31Chris RomeoI mean, as a script kiddie, you had to have a little bit of knowledge. You couldn't be a script kiddie with zero knowledge. And I think the challenge we face now is you can be a script kiddie and only know with just typing to a mod, an agent somewhere. Like, you don't have to understand any of the technical background. I mean, even script kiddies had to get Perl running or on a machine with Perl where they could execute the Perl command or whatever that they were trying to use, right? Yeah. You don't need that anymore.

41:02Johann RehbergerYeah. I mean, a good example was actually just today in, I think it was in Reuters, where it was discovered that a threat actor used, I think it's called Canes, a pen testing system that has a GUI, a UI, and multiple threat actors actually used that, stood it up on the internet, And then started hacking companies and breaching companies. And all of that is discoverable. And a startup in Silicon Valley, actually, I think, I forgot the name now. I think Northforge is the name. Northforge. And they discovered, with Shodan, discovered those panels from these pen testing platforms. And then they could observe which companies were exactly compromised. how they, as the threat actors, actually operated, how they used the AI to achieve objectives and so on. And I think even like OPSEC-wise, you can tell now, right, there's definitely a lack of understanding on how computer systems generally work, I guess, right? That really just shows us how less sophisticated threat actors can now also have very powerful capabilities, right?

42:17Robert HurlbutSo I want to come back around to the sandboxing things. I know you mentioned the kernel vuln you found. via your sandbox work, but like, are sandboxes like mandatory for the future of agents in some capacity?

42:35Chris RomeoBecause I know some people are doing containers, some are doing micro VMs. I'm sure somebody's inventing something I'm not even aware of yet that's going to isolate them even more into a process and somehow wrap them.

42:46Johann RehbergerWhat do you think? I think this is really the first challenge because I think Like, sandboxing, right, used to mean you have hypervisor isolation, at least to me, that's what a sandbox is. Docker, for me, was never really a sandbox, right?

42:57Chris RomeoYeah.

42:58Johann RehbergerBut you have hypervisor, I mean, Docker can be run with hypervisor isolation, but usually used to be process isolation only, at least to my vague memory now. I might be a little bit off here. But the thing is, right now, sandboxing has a lot of meanings. And I think what we generally mean is some form of containment. that takes place, right? But, and right, when you have agents do work autonomously, right, for long periods of time, there's just no way you should just have them run in an infrastructure that is like fully open or something, right? It's just gonna go off track eventually or by accident, right? And not even— It itself doesn't know it's doing something malicious, right? It's just pursues a goal, right? So I think sandboxing is one of these things that we— it has to be a lot easier and out of the box, right? I think that, you know, you just— that's actually, there's one thing I like about Muse from Meta is it has a lot of these like capabilities baked in where it doesn't talk outside the world unless you kind of approve it, allow it, and then it can going forward, it talks like, can talk outside, but it is sort of has a— a good user experience, I feel, where you kind of, as a user, you know what you, over time, that you kind of open things up, right? There might be a reminder that you open certain things up, maybe or not. But if you then look at other things, like egress control is often really, really not handled well. And I think egress control is one of these key points that, you know, that's where agents, as we've seen, in the real world now too with Hugging Face and other scenarios where egress control is really how agents then can go way out of bounds over time. Yeah. So sandboxing, I think, is definitely a critical part of securing agentic infrastructure. And my view has always been, and I think Michael Balguri and I were, to my knowledge, the first ones that always talked about this, that these agents like Copilot and so on, these are like, you need to think about as a company, that they're like a malicious insider potentially, right? And we have dealt in security with malicious insider always, right? It's not a new threat, right? So that should make us all feel very good about this problem, right? Because, oh, malicious insider, every company has them, at least when you have a certain size or number of employees, right? And how can you find them? How can you detect them, right? There is the big difference now with AI is, and that's becoming very obvious, right? It's the velocity and the speed. That's just gonna be the challenge that we have to adjust to, right? Velocity, speed, and sophistication, probably even.

45:48Chris RomeoSo I guess if you had to, you know, talk to a CISO tomorrow and they're saying, hey, my developers need coding agents, what are the— what would you advise that CISO?

46:00Robert HurlbutAs far as, okay, here's the things you got to have figured out before you let these things loose.

46:05Johann RehbergerI think I would tell him 4 things. The first one is you already have them running. Reality. The first thing you need to do is an inventory, right? Figure out actually what is already running. I think it's the very first step. These are all things in security that are not new, right? Security, we have done this our whole lives, right? Get the inventory. Where are you talking to, right? handle what you actually run, right? Then build the paved road, make the one that you secure and put the money into securing, make that the easy choice for everybody, right? That, you know, you have a one-click agent and it's the easy choice. It's secure by default, right? You, that's the control aspect, right? You know where it's going to talk to by default, what systems or tools it has access to, right? What skills you might give it. And then the third aspect is you need to monitor. This is maybe where you need some new investments in addition also, right? How can you monitor what the agents are doing? As we mentioned before, right? Can you have— can you get the prompts? And again, I want to repeat this because I think it's a feature that the labs or AI companies need to add to the prompting experience is some way of signing prompts. Yeah. That you actually know in a way later that the prompt was really the one you thought it was. And then you need to build monitoring around that, right? And the monitoring might be very different than you used to do monitoring because an agent might do an objective over a long period of time, right? So just using a monitor that just looks at each individual action is not going to be enough, right? You need something that looks at it maybe daily, weekly, monthly, and observes a trend potentially, right? So there's new New innovations. And also, again, I think we, as security engineers, we enter a new world of experimentation, exploration, right? We need to figure these things out, what will actually work the best. So that's the third thing, is the monitoring observability. And then the fourth, and this is where I come into the picture, my whole passion comes into the picture, is that you need to test, right? All the things you think are true are usually not true, and that's why you have a red team. And I always encourage CISOs, if you have a certain number of employees or certain number of organizational size, that you have an internal red team, right? That challenges what you think or what everybody is telling you what the truth is. And the red team will help you validate that, right? And yeah, that is the same is true here with agents, right?

48:34Chris RomeoVery cool. Well, where can folks find you, Yohan, online if they want to?

48:39Johann Rehbergertrack your work? I think the easiest is my blog. There's some contact information. It's embracethereds.com. So where I usually try to get a post out every month about some of the research and work I've been up to. Yeah, I think that's the easiest way to reach out to me.

48:58Chris RomeoOkay. We'll make sure that gets—

48:59Johann RehbergerOn X, I guess, as well. But all that information is on my blog.

49:03Chris RomeoOkay. Very cool. And I guess if you had one final thought, Give us a final thought to take us out.

49:09Johann RehbergerA final thought. Oh my gosh. A final thought. Now I try to come up with something grand. Now you caught me off guard. A final thought. I think there's 2 things that should be one thought.

49:25Robert HurlbutI can do 2. I'll take 2.

49:26Johann RehbergerI think one I mentioned already is this normalization of deviance, which is sort of, I think, often an organizational problem. where we need to question certain things more often, right? Like, are we actually— what we are doing now, if we would've 2 years ago, if somebody would've told us that we are doing this now, they would tell us that we are insane, right? Having these reality checks sometimes, right? When things go over long periods of time, they just might slowly deviate from what you think is normal. That is something I always think about a lot. But I think the main takeaway for me is still that we should treat machine learning models as untrusted. And I think my key point is always like assume breach as a red team. Of course, I'm big with assume breach, right? That preventive measures alone are not enough. And so that's why I feel like trust no AI is a good term to take that assume breach mindset, the assume misalignment mindset into into the era of AI.

50:33Chris RomeoYeah, I like that. I'm going to take that forward. Trust, no AI, and the normalization, normalized deviance.

50:41Johann RehbergerNormalization of deviance.

50:42Chris RomeoNormalization, normalization of deviance. Okay, I'm going to take those forward. I'm going to do some more research and read some more of this stuff in your blog.

50:48Johann RehbergerYeah, it's not a term I invented. It was actually a term invented by people that analyzed the Challenger disaster and how NASA's internal organization slowly moved off from the quality assurance controls they used to have in place. And over a long period of time, these little things became like really big prob— bigger problems. And that's where the term originates from, actually.

51:10Chris RomeoIt makes sense in the AI world though, that we could be on that cycle of—

51:17Johann RehbergerVery much, yeah.

51:18Chris RomeoSlowly. And it's not something that'll happen over one evening into the next day. It's something that's over a number of years could catch up with us. So, Johann, this has been brilliant. Thank you so much for sharing the depth of your experience and whatnot with us here. And I know the audience is going to love it. So, thank you very much. I look forward to chatting with you again in the future just to catch up on what you're up to.

51:42Johann RehbergerYeah. Thanks for having me again, and it was great. Have a good one. Bye-bye.

51:47Chris RomeoThanks for listening to the Application Security Podcast. If you enjoyed this episode, subscribe, leave us a review, and share it with a friend or colleague who would enjoy the conversation. We'll see you next time.

8,986 words · transcript by assemblyai

More on AI and LLM Security

View all episodes →

Get Reasonable AppSec: new episodes and useful picks from the archive.