---
title: "AI Pen Testing Killed Traditional DAST"
url: https://appsecpodcast.com/ai-pen-testing-killed-traditional-dast/
date: 2026-08-31
duration_seconds: 2626
season: 13
episode: 10
topics: ["Security Testing", "Cloud and Infrastructure", "AI and LLM Security", "Vulnerabilities and Exploits"]
audio: https://www.buzzsprout.com/1730684/episodes/19727130-ai-pen-testing-killed-traditional-dast.mp3
video: https://www.youtube.com/watch?v=xg8tGBFJdKk
transcript: true
---

# AI Pen Testing Killed Traditional DAST

*August 31, 2026 · 44 min · Season 13, episode 10*

on [Security Testing](https://appsecpodcast.com/topics/security-testing/), [Cloud and Infrastructure](https://appsecpodcast.com/topics/cloud-and-infrastructure/), [AI and LLM Security](https://appsecpodcast.com/topics/ai-security/), [Vulnerabilities and Exploits](https://appsecpodcast.com/topics/vulnerabilities/)

[Audio](https://www.buzzsprout.com/1730684/episodes/19727130-ai-pen-testing-killed-traditional-dast.mp3) · [Video](https://www.youtube.com/watch?v=xg8tGBFJdKk)

## Show notes

Is traditional DAST finally dead? James Berthoty explains how AI pentesting is changing the dynamics of security testing through contextual payloads, autonomous agents, and application-aware vulnerability discovery. We also explore token costs, AI-native security vendors, the future of bug bounties, testing in production, and whether model providers could eventually absorb today’s security tools.

**This episode is sponsored by Corgea. Design it. Build it. Ship it. Corgea secures it. Learn more:** [**https://bit.ly/4wMCNUf**](https://bit.ly/4wMCNUf)

**About Corgea**

Corgea is an AI-native application security platform that secures software from design to production. It brings together security design reviews, AI SAST, dependency and IaC scanning, code quality checks, and autonomous pentesting—helping security and engineering teams find risk earlier, fix what matters, and ship securely.

**Learn more about Corgea →** [**https://bit.ly/4wMCNUf**](https://bit.ly/4wMCNUf)

## Transcript

*8,258 words · assemblyai*

**0:00** the results speak for themselves because the tests are finding things that a human pentester would not have found because they're highly unique, highly complex issues. So I do think there will always be a place for like the super talented boutique pentester for sure. And they're going to use AI frankly to improve their work. I just think it's more the average pentest consumer is getting honestly a better experience than they were getting from one of the like pentest farms, if I can use that language.

**0:31 Chris Romeo:** Mm-hmm.

**0:32** The AI version of that is frankly oftentimes better than the human.

**0:36 Chris Romeo:** James Berthoudy is the founder and lead analyst at Latiotech. With more than a decade of experience in engineering, cloud security, and DevSecOps, he helps security teams cut through industry noise, understand their options, and select tools that solve real-world challenges. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am flying solo today. Well, Robert is in an undisclosed location and happy to be joined by James Berthoudy. And James was on the show back in May 2024. So if you want to go back and take a look at that episode to hear his security origin story, but also the title of that episode was, Is DAST Dead? But James, we do want to welcome you to the podcast, but also my favorite question to ask people right now is, What do you like to do that takes you away from technology screens? Just gets you outside? Like, is there something that, uh, that gets you away from tech?

**1:33** Yeah, we have, uh, 3 kids and a cat and a dog and 9 chickens most recently. And so we are usually busy with a mixture of kids' activities plus animal activities and those things. Otherwise I am pretty like video games are generally my hobby. And so it is a little bit of like press the KVM switch when work signs out and switch over to the gaming setup.

**1:58 Chris Romeo:** Yeah, I get it. Yeah, I get it. So 9 chickens, how many eggs does that get you a day on average?

**2:04** They're not the most egg-laying kind, so it's probably on average like 5 a day.

**2:11 Chris Romeo:** Okay.

**2:12** So yeah.

**2:13 Chris Romeo:** That's still, that'll still fill the shelves in the kitchen, you know? And most people that don't raise chickens don't realize, so they, they, they don't understand that you can keep them as long as you don't wash them. Right. You get a 30-day life cycle.

**2:29** We have a giant basket on our counter of eggs. And then my wife runs like a stand at the end of the road that sells like sourdough and eggs are one of the things we'll sell as well. So.

**2:38 Chris Romeo:** Yep. Nice. Nice. I'd rather talk about this topic than AppSec, to be honest.

**2:43** That's North Carolina lifestyle out here for both of us.

**2:46 Chris Romeo:** You know, as you know, that's where I am too. So this is an all North Carolina, this is a North Carolina original podcast episode right here. But I guess we should talk about something that our listeners tuned in for. But I do find that people enjoy seeing the bigger picture outside of the world of screens, laptops, AI, all that type of stuff. This episode's sponsored by Corgia. Design it, build it, ship it. Corgia secures it. Corgia is an AI-native app security platform covering your whole software lifecycle, from the first architecture diagram to code in production. It brings design reviews, AI-powered code scanning, and autonomous pen testing into one platform, so your team spends less time chasing noise and more time fixing what matters. Visit corgia.com. C-O-R-G-E-A.com. Corgia, one security platform for the entire SDLC. But as I said in that first episode, we talked about, is DAST dead? And that brought lots of fire from a lot of different angles. A lot of different people had a lot of feedback and comments on that one. And so James and I were talking, we thought, let's do another one. Let's take a little different take on it here. And so James, the question that I think we want to wrestle with here is AI pen testing equal to DAST. But I'd love for you to start by giving us a working definition of AI pen testing, because as I find in the worlds of AI these days, there's usually multiple definitions for stuff. And so, I always like to set the context as far as like, when you say AI pen testing, what does that mean?

**4:29** Yeah, to me, like the absolute heart of the definition that we used in our last report is The extent to which any kind of fuzzing or testing tool is generating payloads that are contextual to your application. To me, that is like the key feature at the heart of AI pen testing is that you're not getting a generic list of checks against your environment, but it's something that knows, like, I'll just use the Laceo website, for example. We have different user roles for admins and vendors and stuff. Like, that's the kind of thing that historically before LLMs, I wouldn't be able to test, or I'd have to write custom test to say like the admin role should never be able to lock a user account or whatever, like just whatever permission-specific checks that I'd have to write by hand. LLMs now are the point of AI pen testing to me is an LLM is smart enough to like, as it's navigating an application, go, uh, as an end user, I probably shouldn't have access to this screen. And I think year over year, as within the OWASP Top 10, we've seen like BOLA and IDOR move up the chart, that this is the opportunity that presents itself is really, we're talking about authorization-level vulnerabilities from the DAST side. And then AI SAST to me is the same logic, but applied to like code analysis. So it's really about getting the maximum value out of like a contextual understanding of your application.

**5:51 Chris Romeo:** Okay. So does AI pen testing imply agentic? Like, are these things synonymous?

**6:00** Yeah, I think this is where it gets— so to me, like, that's the heart of like, you qualify or not, which is a very broad camp of people who are now using AI in some way to do pentesting. And then I would argue there's degrees of agentic or degrees of autonomy that are happening. And so, certainly the payload thing that I described, like some traditional DAST vendors have implemented that at this point. Whereas others are taking an approach that's more like from the ground up and giving an agent access to a web browser, giving it access to common pen testing tools and having it like actually run the CLI tools and run like a full pen test on its own and make these decisions. And I think this is, you see the same thing in AI SaaS where it's like this sliding scale of determinism and indeterminism, being able to repeat the test results versus not. And the more deterministic something is, the less like novel findings you get. So they're less interesting. But then if you slide it all the way the other way, you get all these novel findings, but there's more false positives. They're harder to reproduce exactly what's happening. And I think teams are really struggling to like balance both and like, how much do they care about repeatability? Do I want to run an AI pen test on every code change? Very similar to when we talked about DAST last time. Like, see, in CI/CD, like what StackHawk was really doing a lot, a lot, a lot, as well as like what ZAP is capable of, some of this like CLI-based DAST testing that's happening. Teams still struggle to understand the value of like, do I want to run a full AI pen test on every code change? Do I run it on a quarterly basis? Like trying to scope that, like really there's a lot more complexity now of like when and how I do the testing. Mm-hmm.

**7:41 Chris Romeo:** How expensive is an AI pen test? And I don't mean like what they're charging for the tools, but like, because one of the, one of the hits against pen testing historically in the early days when, when I was young and had no gray hair in security, there was pen tests were something you did at an interval because it had such a high cost to it.

**8:01** Yep.

**8:01 Chris Romeo:** Vuln scans was something you did all the time just to see if something new was popping up. but does AI pen testing make it more, is it dollars more expensive, like, or could I do it for every code change if I wanted to?

**8:15** So this is where there is again, like an elasticity to AI cost at the moment that's really hard to quantify, where to me, the real moat that AI-native startups have over incumbents is this question of like, how cheap can they run a pen test? Because if I just ask Claude Code to tell me security issues on leachio.com, run a pen test if it lets me do it, if I get past all the guardrails and stuff. Like it's going to do a pretty good job. The problem is that it's going to be exorbitantly expensive, especially if you're on like an enterprise plan and you're paying per token costs, like your token costs are going to be through the roof. Like I just remember doing the Teams plan and being shocked that just asking it to write like a weekly summary of cybersecurity news is a $6 query by itself, like just to do a news summary.

**9:01 Chris Romeo:** Wow.

**9:02** Never mind, like actually running a full, like query. And so the, the economy here starts to tip, especially as these things get less subsidized by the frontier companies. And that's where all of a sudden you have a bunch of companies who have really specialized in either adapting open source models, being very elastic with, like a lot of the releases now are focused on with our harness, we can achieve Fable level results, but with older, like Opus 4.5 models, but we still hit the same benchmarks. And then you combine that with on-premise offerings, different ways that people are bundling, like the pure GPU costs or running their own open source models. Like that's where the moat starts to get created in, in the long run. But all that to say, I want to back up to answer like, how expensive is it? It is like, if you just go latest model, it can cost as much as like a pen test engagement from like a, like I call them like pen test farms, the people that do like under $5,000 pen tests that are just like checking the box. You probably won't, in any single pen test, you wouldn't hit like boutique pen test prices. But if you really wanted that in-depth experience now, it would cost about the same rate because you'd be running more scoped pen tests with like, it'd be more like human and AI interacting together and trying to dive more into the results. It's not the kind of thing that you're like one-shotting in that perspective.

**10:22 Chris Romeo:** So you mentioned the, the, the, the kind of difference between AI-native companies and the incumbents. And no matter what category we're talking about, that seems to be a source of tension right now across our industry. I know I'm putting that really nicely. I probably could have described it a little bit differently, but we'll go with tension because this is a very highbrow podcast. What, like, what are you seeing across the industry here with those that are trying to, let's say, play catch-up versus those that were able to design? Like, are the catch-ups, are we getting close? Are they getting closer? Are they still far away from a capabilities perspective? Like, what, what are your take on this?

**11:02** Yeah, I think there is a real pro and con here that makes this not like a straightforward question. Like there's nuance to the answer as far as some people were, were too early, frankly, and they were trying to maximize the value of like some of those early models that just weren't good. And the frontier, like you look at, uh, like the auto fix category that existed for like a year of like a whole category of vendors that were about using early models to do code fixes. And none of the incumbents had that yet. It's a, it's a novel capability, but nowadays I can just ask Claude Code to fix something and it's going to create a fix that's much, much better than any of those initial solutions. And then if I'm a bigger incumbent vendor, I've actually won out here because I can just not do anything. Like I'm literally just saying, Claude, here's the security vulnerability we've discovered, go fix it. And then it just goes and fixes it better than like that AI native startup. did. And so there is like a benefit on the one hand to waiting, but the downside is there are some real challenges that the AI-native people who didn't get tied to a model and are tackling some bigger problems do have. And I'll use like AI SaaS and AI pen testing as a great example here, where I do think just from my experience using like Security Review, using Claude to do pen testing, like just directly interacting with those things, it doesn't quite have like the harness or the specialization to get like consistent scanning results. To find as many findings and to really package those things in a way that makes a lot of sense for security engineers. And so I do think there is still like a strength to being like an AI-native, either AI SaaS company or AI pen testing company, because you, you've just built so much harness and IP around like deduplication of results, how to build like a pipeline to actually go and do these things, how to do false positive analysis, combining static analysis with dynamic dynamic analysis. Whereas for a lot of the bigger companies that are, they get, they get the benefit of the frontier models doing the heavy lifting for them, but at the cost of, they haven't experimented as much with like how to get novel findings, how to deal with duplicate finding issues, and some of the more like advanced things that come up when you try to scale these things.

**13:12 Chris Romeo:** So if, if I go out and purchase an AI pen testing tool, right now, what does the cost structure look like? Do I pay a certain amount per month for a certain level of service and they pay for the AI tokens, or am I using my own tokens? I'm just thinking about the economies of scale here and I'm starting to put this together. I'm going, how are they doing the— how are you selling a product and paying for all the token charges that go with it unless you sell this thing for a lot of money? So what's, how's the market playing out here?

**13:45** Yeah. It is very much in flux along all of those things. Like, first of all, vendors will always choose the pricing structure that maximizes the cost. And so early on, it is like just charging a pen test for a pen test, right? Because they can get the same results. If you're already paying, let's just say $20K for a pen test, like that's what they're going to come in and say, here's our AI pen test and you'll get better results for $20K. Over time, that there's a bit of like a race to the bottom happening on like per pen test pricing when it comes to AI pen tests, because it's sort of like, how do you evaluate, like, who has the best AI pen test? Should you do it in-house? Those sorts of things. And so the per pen test pricing model just doesn't, like, scale really in the long run. And you are getting a lot more flexibility with, like, for example, Wiz just released their, like, AI SaaS capabilities. And with that, like, they really released a harness and you can plug whatever model you want into that. To do the analysis. And I do think that's the direction that people are going because if, like, really as a security practitioner, if I'm in a high security environment, the thing that I would be willing to do is whenever, like, there's the M-DASH model from Microsoft comes out, like, I want to pen test from that. Fable comes out, I want to pen test from Fable. Google's model comes out, I want to pen test from the Google model. Like, there's flexibility to, I want the best results from these models as they come out. On the flip side, this is where I said some of the moat around the AI native startups is their ability to do this at a much lower cost because as businesses do care about token costs, like for me, it's not necessarily a benefit to have pass-through token costs. For some big enterprises that have like large spend commitments or large contracts with frontier providers, like they're happy to have those tokens like roll up into that. But for me, I don't necessarily want that. And some of the startups who have done some of their own model building, some of their own tuning, like they can provide it at a much lower cost than you would get if you tried to go through one of the bigger providers.

**15:44 Chris Romeo:** So when we say AI pen testing, is this automated AI pen testing or is this the old school model of pen testing consultancies just using AI?

**15:58** It's a real mix. I think I, when it comes to testing, like I'm a bit of a, I have a weird general position on AI because in general, I think people like inflate its capabilities in some ways. And I'm generally, in general, I think the capabilities of AI are pretty inflated from like an executive business perspective. Uh, however, I am, I really think it's strong at this specific thing of like testing and analysis for inputs and outputs and looking for like things that shouldn't happen. Whether that's bugs or security issues or anything around like development, I think it's super strong for the, for those use cases. And so here, what I've seen is like, it's a mix right now from, there are some providers that wrap it with like a human service offering the way we've seen with like, like a lot of, uh, pen testers with a platform, which I think is sort of the vibe you're talking about where like, you can tell they're really like running a DAST scan and then like shaping it up. And then you have like a human, like managing the results. with you, like walking you through the reports and stuff. There is some of that that's available still. That's, that's more with the, the service offering on top, but more and more, I think I'm seeing people be okay trusting like an AI pen test because the results speak for themselves, because the, the tests are finding things that a human pen tester would not have found because they're highly unique, highly complex issues. So I do think there will always be a place for like the super talented boutique pentester for sure, and they're going to use AI frankly to improve their work. I just think it's more the average pentest consumer is getting honestly a better experience than they were getting from one of the like pentest farms, if I can use that language. The AI version of that is frankly oftentimes better than the human version.

**17:42 Chris Romeo:** So, okay, so that makes sense, just thinking through how these things are being delivered. And so it's not something that I engage for every pull request?

**17:54** Well, and this is where it's frankly, there are some companies that are really leaning into this as the direction, right? Where it's like a lot of teams want to get to the point where they have fully end-to-end deployments of their code without human review at all. Like, especially a lot of like Bay Area AI-native startups, like that type of company is very much trying to get to where they are only providing these high-level directives and the agents are going and implementing all the way through to deployment. That's definitely the minority of the industry at the moment, but that's where like people want to get. And the more you see that as the future of software development, the more willing you're going to be to like pen test on every deploy, right? Because it's just part of, it's really like the new way to conceptualize of you open a pull request, it runs the DAST scan. In against the state, you deploy out the staging, it runs the DAST against staging, you fix any results, and then it goes out to prod. But now the agentic version of that would be the agent either runs it locally or deploys out to staging, runs a pen test against itself, fixes any findings, and then deploys it. And the question to me is just around how much, to me, there was always a complexity trade-off where security teams would love the sound of that. But for a lot of them, it was like too complex to run like a DAST on every pull request. And the same thing applies here where An organization's appetite for running a pen test on every pull request is going to just come down to like how standardized their CI/CD pipelines are, how good they feel about their end-to-end agentic deployment processes. Like a lot of those sort of nuances that make it unclear the extent to which people will adopt some of this stuff.

**19:38 Chris Romeo:** And some of it's going to be cost, especially if you're not a model company. You, and if it costs $500 in token spend every pull request, that's not going to work if you're pushing thousands of times a day.

**19:52** The cost thing is what underlies all of this. And this is more like an AI economy thing where it's just, we're in a crazy moment right now where I think it's probably, AI is probably the cheapest it will ever be because it's so heavily subsidized, especially for like personal plans. Like the value someone can get out of a Claude Max plan is pretty insane compared to what that even costs, like an enterprise or team plan. For those things. And I know multiple companies right now that, for example, like you are dinged as an engineer if you're a low token spender, like you're being measured on how many tokens you're using. But I have to imagine that it's going to flip at some point where, you know, you hear these like news articles that come up around people hiring back teams when they realized the AI cost was more expensive than the human cost was. And that's definitely like a macro tipping point. And to me, that's where, again, the AI-native vendors who don't just sort of rely on the frontier models to do the heavy lifting, like they have a real advantage there because they're doing this at a much lower cost than the frontier models are.

**21:01 Chris Romeo:** Yeah, the kind of the things I heard about the token spend issues is people were writing scripts to start generate to start spending tokens on meaningless things at 7 o'clock in the morning or something so that they could ensure that they got their spot on a leaderboard for token burn for the day. So yeah, I mean, people game whatever system you provide for them.

**21:25** Yeah, it's like judging developers on line of code, right? Like, I don't know why they, we all agree it's a terrible metric and we all sort of had companies that we worked for that did it anyways. And this feels like a similar thing where it's a pretty poor metric, but people will still do it just 'cause it's easy and—

**21:41 Chris Romeo:** Yeah. Yeah. It'll, it'll disappear here soon. Okay. I guess one of the challenging things when we think about AI pen testing, autonomy. So what prevents the pen test from going further than it should have? Not that that would happen, but I, think there was a news article about OpenAI agents breaking into Hugging Face to achieve the mission that they were set before them. Sounds like there's some, there would be some danger in AI pen testing if we've already got something that's set to go break stuff and take things, for it to get off the rails a little bit and start following another thread, and all of a sudden it hacks something that's a big problem.

**22:29** Yeah, that's, I think most vendors that I talk to still only like technically suggest that you use their AI pen testing in your staging environment because of this. And to me, that honestly undermines a lot of the value, right? Like something people love about DAST, one of the very few things, was that you do just put in like a URL and then it tests, right? Like it's a very simple workflow to get a questionable amount of value depending on how it's set up.

**22:54 Chris Romeo:** But.

**22:55** Like the workflow itself is pretty simple. And I think because it is really hard to put some of these guardrails on agents and nobody wants the liability for taking down a production app, it's definitely an in-process concern about how do you limit that blast radius? Like some of it is around permission scoping. Some of it is around like giving like categories of commands, like things that have existed within DAST, right? Like if you don't, a lot of vendors have a checkbox for like allowing it to take destructive actions or not. And some of that still applies here, but for sure it is an unsolved problem. And it really comes down to how much risk you feel like you can take on based on how the vendor allows you to like tune easy mode versus try-hard mode. And, and what's the, the amount of risk that I'm comfortable taking on with this thing probing my production system?

**23:44 Chris Romeo:** Yeah. I mean, as long as I'm not taking on too much liability for other people's systems, I'm, I'm the type of person who's always like, hey, it's our system, blast away. If you can knock it down anyway, somebody else can. It's not like it's an original. Even before we had AI, you're, it's, it's not an original idea to whatever you came up with, someone else could think of it. So if it takes the production system down, shame on us as an engineering team for not finding that and fixing it earlier. At least we know.

**24:12** That is exactly how I feel about it. And it's more just, uh, I think I'm just more sensitive than I used to be to like, You are a large enterprise bank and maybe that's a little scarier.

**24:23 Chris Romeo:** Yeah, it's a, it's a, it's probably a regulation that prevents it that you're somehow governed against because who knows what all those regulations are.

**24:31** Exactly. I think the, the other thing I'll say to the other approach here is there are, there's a whole other approach to AI pen testing that's much more of like a gray box approach plus using your like cloud infrastructure configuration. Um, so it's like analyzing the code plus the infra and taking more of like a hypothetical attack path mapping approach using like nuclei templates to validate, um, vulnerabilities from the outside and those sorts of things. And while that's not, I think that's like not the intuitive way we would think of an AI pen test, because that would be like an agent with a URL figuring out how to get into the system. Um, that is another approach that a lot of like, especially CNAPP providers are taking, which is more around mapping out the cloud infrastructure and using AI to like find attack paths in light of the code and application context to figure out how someone theoretically could pivot through a system a little bit like what Pantera does for, yeah, that continuous testing sort of approach.

**25:26 Chris Romeo:** So I want to push pause on the AI pen testing for a second, and then we're going to, I want to come back and, and really dive into it and, and for once and for all answer, is DAST dead? But bug bounty is something that keeps coming up to, coming to my mind as I'm watching AI start to progre— or continue to progress. Like, I just got this gut feeling that bug bounty's going away.

**25:49** I was going to say it might be bug bounty's dead is probably the better—

**25:53 Chris Romeo:** I mean, okay, so I'm not the only person that, but I, and it's not, I don't have any insight. I haven't, I haven't got any perspective other than I'm watching it and going, if I can get a model to do this, and I would normally pay $100,000 for one of these findings. I can just invest. I don't care where the money— I'm still spending the money. I can spend $100,000, look with a, with a model provider looking for those types of things and probably find more than one through the process. So it just seems like AI pen testing, AI and bug bounty in an AI world are going to diverge. And it feels to me like bug bounty is just going to kind of slide away into the historical record.

**26:33** Yeah, I mean, it's an interesting moment. Like, I'll shout out like Joseph Thacker is someone who's great on Twitter, who's like a bug bounty guy doing a lot of testing with AI stuff. And right now it's a great time to be like a really good bug bounty person because people haven't changed their programs yet to account for it. And so you can make a lot of money, like putting out high quality bounties on a lot of providers. And there's a lot of really high impact vulnerabilities getting found by these testers using AI to do it. But to your point, like in the long run, there certainly is like a, alongside those like sort of 10x findings, like I think of like the GitHub RCE that Wiz disclosed was like an insane finding. And I think like those are the kinds of things that bug bounty will always be around for, because it's insane. But the normal day-to-day bug bounty reports are just getting absolutely flooded With new versions of the slop that they always sort of had. Like there was always this like smaller amount of, uh, hey, I found cross-site scripting, that'll be, you know, $2,000. And it's like, I don't know, like kind of. And AI is just giving those, those same reports at a different scale to organizations. And I think what they're having to do is just adjust their bug bounty payouts and expectations and severities to account for the increase in submissions that are all very low quality. And some of the platforms themselves, like HackerOne, have built in a lot of like AI triage along the way to try to help with that. It's a, it's a weird version of like AI versus AI, which everyone tries to do for like attackers and defenders, but it's like AI for defenders dismissing AI-generated vuln reports to help them like get to the bottom of like what's real or not. But it is certainly like a, I think low quality bug findings are certainly going away. Um, as like a useful testing mechanism in favor of AI pen testing type solutions.

**28:28 Chris Romeo:** I'm chuckling because I'm just envisioning this loop that's happening. Submission comes in, response with feedback about how it's terrible, other AI tries to improve it in some way, and they just keep cycling through and everybody pays the AI token provider.

**28:44** It's, it's the version of the, the meme that's like, uh, People using AI to transform bullet points into long-form articles, and then someone using AI to translate the long-form article into bullet points.

**28:57 Chris Romeo:** That is a metaphor for what, what we're experiencing in the world. Okay. Back to our original question we talked about in 2024, wrapping it into this AI pen testing angle. Like, is, is DAST finally Like, is there an AI— well, just catch me up on the market. Like, is there an AI DAST at this point?

**29:20** Yes, I would argue there's an extent to, there's an extent to which I would argue every AI pen testing is an AI DAST because most of it, like, this stuff's not magic. Like, they are giving LLMs tools that include tools like ZAP to go and run testing. And most of the leading DAST providers that were like startups, so like Invicti is the biggest one of these, but then you've got like Escape and StackHawk and other people who, who were reinventing DAST for like CI/CD and API-driven testing. Like they all have heavily incorporated AI elements to all of their testing as well, whether it's that dynamic payload generation or a full-on AI pen test. Then on the flip side, you have startups like, like Aikido has their approach that they acquired a couple of companies to build that's like sandboxed agents that are all like an agent swarm, like running their own tests that include like browser-based versus CLI-based, like a whole bunch of different mechanisms. Then you've got like Expo and Novi and Terra and like some of these guys that are all taking a combination of different approaches. And so what I will say is like non-AI testing in general, I think is dead where everyone at this point appreciates that AI brings additional SaaS value. Like Checkmarks has AI SaaS. It's very rare to find someone who has SaaS that's not, uh, that doesn't have AI in some way happening at this point. And then, and then the, I just view this as the equivalent happening for DAST and the, the pen testing spin on it is just a cute way to get additional budget because people will pay more for a pen test than they'll pay for a DAST scan. And so if you called it AI DAST, that's like a $10,000 a year contract. But if you call it like 4 AI pen tests annually, then you've got $100K contract. All right.

**31:06 Chris Romeo:** Pushing pause on the DAST conversation again. Made me think of something else. Do you see a world where the model providers absorb the SaaS, the DAST, the IaaS, the RASP, OMI, like all of those capabilities into the models offering itself? Because we saw OpenAI come out with an offering. We saw Anthropic come out with an offering. I haven't heard anything since though. That's the thing. Like they just kind of like had press releases. Everybody said, oh no, startups are done. They'll never exist again. And then a month later—

**31:40** Startups are dead.

**31:41 Chris Romeo:** Yeah. But a month later, there's no, like, I haven't heard anything about them since the big announcements that rocked everybody's world. And so, like, are you, from what you're seeing, do you see like a movement where the model companies will start to absorb these capabilities? Or do you think there's going to be a market for SAST and DAST or for AI pen testing and source code provenance and, you know, all of these things?

**32:05** I mean, I think, uh, it, it very much reminds me of AWS and cloud security where could AWS kill the cloud security market?

**32:13 Chris Romeo:** Sure.

**32:14** Like they could build what this, any of the CNAPP providers and provide all of those CNAPP capabilities natively within their platform. GuardDuty is pretty good. Security Hub is pretty good. Like a lot of these tools are, are good, but a lot of the security industries, not built around like good enough. Like, first of all, a lot of it is, right? Like AWS's inbuilt security offerings all do super well. GitHub Advanced Security does super well. Like a lot of these built-in offerings, like it's not like they don't have customers. It's just that security is still willing to go outside of those providers if you're providing an experience that justifies going outside of them. And to that extent, like, I, It really depends on how much Anthropic and OpenAI want to invest in these things, um, and how much they can productize them too, right? Like OpenAI, um, just brought on Clint Gibbler. Like he obviously really knows what he's doing and is doing a great job there, like, uh, helping do a lot of this, uh, program. And honestly, I think a lot of both companies' approaches so far have been very like noble in a moral sense to where they're prioritizing, we've got these big enterprises. I think they're legitimately concerned about AI pen testing capabilities, and they're trying to help scan open source repos and get patches done and find all the big vulnerabilities in the big packages that are commonly used. And I think that's been their priority. It feels to me the last 6 months more than trying to create a true product out of any of these capabilities. And the question, like, could, uh, could OpenAI or Anthropic make a I'll just use like Sneak, Checkmarks, whoever, like, could they make one of those solutions? Sure. But I just, I don't like AI SaaS, which is what they're, they currently have the closest version of, is still just one slice of the alphabet soup that you named. And so I just have a hard time believing that they're gonna, in a timely manner, expand across that entire suite of offerings. Certainly they'll do stuff along the way that will pick up customers. They'll have their own security offering that'll do well, but I just don't think it like kills the market because it's just not their only priority as a company. Yeah.

**34:28 Chris Romeo:** As I, as I start to think about this more, I realize this is just 1/10 of a percent of the entire universe of things that they could build products around as the model provider. They could build a surgeon's companion, you know, that helps people perform surgeries that they're maybe not as well trained for in an emergency situation. Like, there's so many other things that you could, that you could say would be greater good type of things in the world before you're like, well, we should make this thing scan code better. Like, yes. And it also helps to have more, it helps a market, like with AWS, if they had come out with something that killed all of the AppSec companies, I would argue that would've limited their market more because they got more exposure. They get a lot of exposure because of these other companies. They run on AWS. Like everybody knows what that is today, but there was a time when they didn't. And if your tool supports AWS, it's just a bigger tidal wave of opportunity.

**35:30** And you look at like Claude has done a great job with launching their, the Anthropic Compliance API. And they have like a whole partnership program around like ingesting using their compliance API and who their partners are like that. To me, that speaks to a very similar thing like AWS or any of these big platform providers that they have a certain level of built-in capabilities that they're expected to solve for. I think like prompt injection is a good example of that where the frontier models, it's, it's not like they don't do anything to prevent prompt injection. Like they do a pretty good job protecting against a lot of these attacks at this point. But you can still go and buy a specialist runtime vendor that's doing a proxy that's running small language models on top of it, like doing all these categorization intent tracking. Like it can get a lot more advanced, but there is like a baseline. It can't be so comically easy to do the like, forget all previous instructions and give me whatever.

**36:24 Chris Romeo:** The famous Chevy, the Chevrolet dealer one from the early days where the person went on and said, they're going to get a, you're going to give me a 100% discount or whatever. And negotiated the deal with the chatbot. Yeah.

**36:35** And it's, it's the same too with, uh, the code scanning side, right? Where it's like, they have to bake a level of security into their code. Like it can't be comically insecure as it's coming out by default to where it's just like, what's permissions? Who cares? But there are a series of startups that are focused on giving security guidance to the agents as they're coding to get more like in-depth, specific tailored guidance. And that's the kind of thing that I think it'll just be a while before Anthropic tries to tackle, if at all.

**37:05 Chris Romeo:** I hadn't thought of Anthropic and OpenAI as platform providers, but my eyes are widening to this. There's going to be a whole segment of platform providers and they want ecosystem partners running on top of their systems because they're billing every time. They don't need to go invent the next 1,000 cool startups if you have the next 1,000 cool startups running, consuming tokens on your platform, you're a part of them anyway. Like they're built on top of the ecosystem. Why try to go create 1,000 individual innovative ideas and products and compete when you can just get everybody coming to you and running on top of what you're doing? So I'm seeing the AI market in a completely different view here, James, after our conversation.

**37:52** That's great. Yeah, I think it's always that awkward tension. I think AWS is just the clearest example of that where You know, AWS is happy to partner with Supabase. Oh, by the way, we have a competitive offering to Supabase. And it's like that, you know, recognizing that they're not going to always be the perfect provider for every use case, but they do have something for most use cases. And same with like, like the, the parallel here with Claude is like Claude Cowork, if you're an accountant, is pretty good at that. But there are a whole series of accounting startups that connect directly to your data sources and do all this stuff. And Uh, yeah, Anthropic's making money either way.

**38:33 Chris Romeo:** I still like having a CPA because when my taxes get submitted and if the IRS shows up, I can at least point and go, yeah, is that this person?

**38:44** Well, that's honestly a heck of a parallel to security. And that's, this is how I, this is a little bit off topic, but like how I feel about the MDR space and pen testing is similar to where, uh, there is a level of liability that you're Trying to offload, even if not in a strict sense. Like, if you have an incident and you have an MDR partner, part of why the CISO wants an MDR is so that if the shit hits the fan, they can say like, oh, our partner like is helping us, or they missed it, or it's their fault, or then we're going to switch. Like, it really offloads, even in an informal sense, some of the responsibility. Or if, or if a big breach happens from your web app, to be able to say like, oh, we just got a pen test 6 months ago from these guys and they didn't catch it. And I think there is a level of people wanting the vendor, even if it is all AI, so that they can say, hey, we just bought this AI pen testing tool and they didn't catch it. Like, we got to go look at another one or something.

**39:38 Chris Romeo:** Yeah. Yeah, that's— you start to wonder, when does that— when does the rest of the industry kind of wisen up to that fact and the courts come back and say, well, you can't just transfer that liability to an outside vendor, that's, or a model provider, that's really your company's piece of the pie. And that's where the problem lies. Yeah. 'Cause people have been using that pen testing excuse for decades at this point. Well, we just had a pen test and they didn't find it. So, well, you know, that's, we did our best. And it's like, that's not how the modern attacker, they don't think about, oh, you did your best. I'll just leave you alone. I'll go I'll go find a different place. Yeah.

**40:22** And I think there's another point here that's like the shared responsibility model with AI is very unclear at the moment too. And it's not as clear-cut as it was with AWS where like they managed the security of the compute itself, but anything you like install or build on top of that compute is your cutoff. Like there is no OpenAI handles the security of the tokens, but the rest is on you. Like it is a weird, They do some prompt injection prevention, but they don't like guarantee it. They try to limit the permissions and give you the tools to do that, but they also don't fully give you the permissions control you like really would need to guarantee the lack of an incident. There's like guardrails, but they don't work all— like, it's all very indeterministic and it creates a very blurry shared responsibility of like, if Claude goes and deletes my production database, like whose fault was that?

**41:11 Chris Romeo:** Mm-hmm.

**41:12** I mean, it depends on the public square at the moment.

**41:16 Chris Romeo:** Well, I think we can safely say classic DAST is completely dead. It's gone, it's buried, it's history.

**41:27** It pains me to say, because I'm a DAST defender for so long from our last time, and I think I'm most shocked at how AI on these testing sides has really quickly changed what people expect from their products and the capabilities they can deliver. And on the SaaS and DaaS front, I think it's just insane how quickly this has happened.

**41:48 Chris Romeo:** Yeah. And it's still moving fast. So hold on tight. We're not, we're not done with the journey yet. There's more, there's more to come in the weeks, months, and quarters ahead. So James, what, any, what key takeaways do you want to leave our audience with coming out of this? Things you might want to point them to or? I don't know, anything you want to share?

**42:07** Yeah, I think, uh, if you're looking for solutions around this, uh, latio.com is the website and we have like the categories set up and, and, and the goal of the website, uh, as well as our reports are all available on there for free as well, is to try to just make accurate categorizations for people so that when you see that a vendor is in a category, like they actually have a relevant capability there. Like a lot of why we built that in the first place was so many of these like larger aggregation sites. just become these giant slop pools of like everybody's in every category and it doesn't even make any sense most of the time. And the goal here is to actually get guidance on some of these emerging capabilities and make a decision, right? Like you've got real trade-offs that are happening as you think about like, do we want a platform? Do we want a point solution here? Like, how do we think through it? Like, that's the kind of stuff that we help people navigate.

**42:59 Chris Romeo:** Well, James, thanks once again for enlightening me. further in the world of AI. Definitely left me some, with some, some additional things to think about as far as AI as the platform provider and, you know, just, just a, a number of different things that, that got my brain moving here. But thanks for being a part of the show and we look forward to another episode in the future where we can, we can untangle something else. That sounds good.

**43:26** Thanks for having me, man.

**43:26 Chris Romeo:** Yeah, thank you. That's it for this episode of the Application Security Podcast. If you found this useful, share it with someone on your team. And if you're on YouTube, subscribe and drop a comment. On Apple or Spotify, a quick rating helps new listeners find us. We'll be back next week with another conversation. Until then, keep building secure stuff.

---

Source: https://appsecpodcast.com/ai-pen-testing-killed-traditional-dast/