---
title: "Sarah-Jane Madden -- What AI means for AppSec"
url: https://appsecpodcast.com/sarah-jane-madden-what-ai-means-for-appsec/
date: 2025-07-09
duration_seconds: 2279
season: 12
episode: 10
guests: ["Sarah-jane Madden"]
topics: ["Secure Development"]
audio: https://www.buzzsprout.com/1730684/episodes/17474602-sarah-jane-madden-what-ai-means-for-appsec.mp3
video: https://www.youtube.com/watch?v=GsJ8tbcjUuk
transcript: true
---

# Sarah-Jane Madden -- What AI means for AppSec

*July 9, 2025 · 38 min · Season 12, episode 10*

with [Sarah-jane Madden](https://appsecpodcast.com/guests/sarah-jane-madden/)

on [Secure Development](https://appsecpodcast.com/topics/secure-development/)

[Audio](https://www.buzzsprout.com/1730684/episodes/17474602-sarah-jane-madden-what-ai-means-for-appsec.mp3) · [Video](https://www.youtube.com/watch?v=GsJ8tbcjUuk)

## Show notes

Sarah-Jane Madden joins Chris and Robert to ask what AI actually changes in software development—and what foundational practices still matter. Drawing on her OWASP Global AppSec EU keynote, she challenges the idea that AI makes the SDLC obsolete or turns every prompt into production-ready software. The conversation examines vibe coding, the difficulty of operationalizing AI-generated prototypes, skill atrophy, hallucinations, and the quality problems hidden by code that merely passes generated tests. Sarah-Jane argues for treating AI as a useful assistant rather than an unquestioned authority: engineers must preserve critical thinking, understand their systems, and verify the output. The episode closes with practical guidance for engineering leaders who want teams to gain efficiency from AI without surrendering judgment, accountability, or software quality.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
We provide diverse training content and easy-to-digest lessons to meet individual learner needs.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Sarah-Jane Madden:  
→ [Sarah-Jane Madden on LinkedIn](https://www.linkedin.com/in/sarahjanemadden/)  
→ [Nemo Resideo keynote at OWASP Global AppSec EU](https://www.youtube.com/watch?v=CZdSGKoi90Y)

Mentioned in this episode:  
→ [Nemo Resideo keynote at OWASP Global AppSec EU](https://www.youtube.com/watch?v=CZdSGKoi90Y)  
→ [Sarah-Jane Madden — Threat Modeling to Established Teams](https://youtu.be/jFSc8LBiW-A)  
→ [GitHub Copilot](https://github.com/features/copilot)

Chapters:  
00:00 Meet Sarah-Jane Madden: What AI Means for AppSec  
02:35 From an OWASP keynote to AI in software development  
04:59 The biggest misconception about AI-assisted development  
07:13 Why AI will not eliminate software development  
11:58 Vibe coding and the production-readiness gap  
15:46 Operationalizing AI-generated prototypes  
17:53 Where AI helps inside the SDLC  
23:58 Over-reliance, skill atrophy, and engineering judgment  
27:00 Using AI for grunt work without losing core skills  
29:38 Can developers trust large language model output?  
33:36 Practical guardrails for responsible AI use

## Transcript

*6,463 words · assemblyai*

**0:00 Chris Romeo:** We're diving deep into AI's impact on the software development lifecycle, unpacking common misconceptions, real risks, and the surprising ways AI acts more like a secretary than a savior. We'll explore how AI is reshaping developer skills, security practices, and team dynamics, and what leaders can do to navigate this shift responsibly. Stick around as we imagine what a healthy, AI-supported SDLC might look like in just a few years.

**0:25 Sarah-jane Madden:** The Application Security Podcast is brought to you by Security Journey. We provide diverse training content and easy-to-digest lessons to meet individual learner needs. Learners report improving their knowledge as much as 85% on AppSec topics. Learn more at securityjourney.com.

**0:40 Chris Romeo:** Hey folks, welcome to another episode of the Application Security Podcast. I am Chris Romeo, the CEO of DaVinci. Well, also DaVinci was recently sold, so now I'm the AVP of Security Compass. Also joined by my good friend, Robert Hurlbut. Hey, Robert.

**1:08 Robert Hurlbut:** Hey, Chris. Yeah, Robert Hurlbut, I'm Principal Product Security Architect and Threat Modeling Lead at Torion, and great to be here.

**1:17 Chris Romeo:** That's a lot of words too.

**1:18 Robert Hurlbut:** It is. It is. And I'm, I'm still practicing.

**1:22 Chris Romeo:** get AI to summarize that for you.

**1:24 Robert Hurlbut:** I know, right? In words.

**1:26 Chris Romeo:** And maybe it'll hallucinate and put fun word in the middle of it. So, uh, excited to be joined by our friend Sarah-Jane Madden, who's been on the podcast before. I wanted to call this out. It was March 2023. The name of the episode is Threat Modeling to Established Teams. I remember sitting at the OWASP event in Dublin, of all places. And I'd never met Sarah-Jane before, and she did this talk about threat modeling, which always catches my attention at any event. And I was like, wow, how have I never met this person or know who she is? But that led to the interview on the podcast, and she was an advisor for DaVinci. And so, great to have Sarah-Jane here. You delivered a keynote, I understand.

**2:14 Sarah-jane Madden:** I did at OWASP Barcelona there, um, 2 weeks back now, I think. So, yeah, I was, of course, threat modeling was part of it, but that wasn't the, the, the key to it. Um, it's about all the change that we're surrounded by at the moment. Um, AI, the good, the bad, and the indifferent.

**2:35 Chris Romeo:** Yeah. And that's what we want to pick your brain on today. So, I'll, I'll start us off. What, I guess, how'd you get to that topic? Like, what drew you to that topic, and how'd that get to the keynote stage?

**2:47 Sarah-jane Madden:** Yeah, so it's, um, I don't ever like to talk about anything that I don't, you know, experience, that I'm not finding interesting. So, uh, I actually, the talk was Nemo Residuo, which is, you know, Latin for no man left behind, or no person left behind. Um, and it was just about, you know, we hear so much about unprecedented change, blah, blah, blah, blah, blah. And if you stop and think about it, every change is the biggest change that you've had. You know, if you've never been on a high-speed train, the first one you get on is the fastest, you know, and it's the moment that you're in and it might be the slowest one where you are, but you know, it's all relative. And I see, you know, again, real teams, scrambling to not get left behind, but also, you know, not get caught up in the chaos too much. And the way I see it is, it is just another change, you know, and it won't be the last. Quantum is probably around the corner and gonna, you know, pop its head in. And who knows what else we don't know about. Because if you put yourself back 20 years, all the changes that we've seen since have been, oh wow, you know what I mean? We went from kind of programming and PL/1 to— I even remember when we started talking about object-oriented coding, it was like, oh, that's the next big step up and everybody has to educate themselves. So that seems silly to even talk about now, you know, but put yourself in the time, virtual machines, you know, Kubernetes, microservices, APIs were a whole thing, you know. It's, it's all big change at the time. Now, I concede that AI is rather a big one at that, but I don't make many predictions, but I think we'll look back on it and go, oh, that was just that, you know, and it'll just be the new reality. So it's about not getting left behind, but not falling foul of it either.

**4:49 Chris Romeo:** Yeah.

**4:49 Sarah-jane Madden:** You know, you want people to be excited about technology, but let's not get overexcited. Okay. This is not the first time we've had a major change, we'll be fine.

**4:58 Robert Hurlbut:** So, Sarojini, in your view, what's the biggest misconception about using AI in software development that you see or had experiences with?

**5:12 Sarah-jane Madden:** I think the biggest misconception and, well, first of all, anyone who thinks they know it all at the moment. doesn't or is seriously deluded. Okay. Because this is rapidly evolving. So right now, today, as I see it, the biggest misconception is that that's it. It's all super easy. We plug the AI in and we go and that we don't have to consider, you know, the good practices that we, we've, we've had all along. Like the SDLC is not dead. If I see another post on LinkedIn with that, I just, I don't know whether I laugh or cry or, you know, it's just. Yeah, right. It's, it's, it's clickbait at this stage. It's not dead. Um, we still need the same assurances. I did a lot of research before doing the keynote, obviously, and, uh, I, I was looking into kind of, um, some studies that were made more around social impact in that. And there was, um, one done for the World Economic Forum, and it was about, you know, those who want to embrace it, embrace AI, and those who are, you know, Again, don't want to see it in their product journey or their customer experience. But both sides, the one thing where they kind of dovetailed in together was at some point they wanted a human element in it. So I think there's a natural thing there. You know, we call this human supervision or whatever, but even those who are not technical realize natively that, you know, it's not that easy. I think it can just, in a corporate or an enterprise environment, you very quickly go, great, I only need 30% of the developers I ever needed before. I don't need any of this SDLC nonsense. Just tell the thing to write me the bot or whatever and go. And I've cut all the fat and I'm now a bazillion dollars a thing. And it's just, life is not that simple.

**7:13 Chris Romeo:** If only it was that simple, right? Where we could just bottle this up and, you know, all companies are making money because they've optimized for AI. Yes. Yes.

**7:24 Sarah-jane Madden:** It's all perfect. And we'd be having this conversation on a beach somewhere.

**7:29 Chris Romeo:** I do think AI is going to impact different industries in different ways. And so, when we've been specifically talking about software development perspective. I think about something like delivery, like truck driving. I think AI is going to cause a big change. It's probably going to take 25 years to happen. We'll probably all be on those beaches somewhere at that point in our golden year retirement packages or whatever. But like when you think about autonomous operation of vehicles and the number of people who deliver goods across Europe, across the United States. There's a lot of humans that spend 14 hours a day, which is the legal limit inside that truck, driving something from point A to point B. And when you think about like a, it's just an, it's not a technology thing, but I think AI is gonna, there's gonna be a disruption. But I agree with what you're saying here. Like it's not gonna, It's not going to eliminate software development. I don't even think it's going to eliminate software development jobs.

**8:38 Sarah-jane Madden:** I think it's going to be— It's just going to change them. And that was, that was kind of where I was going with it. It's like the baby's not going out with the bathwater here. Um, now, as a developer, if you're a software engineer now, I heard somebody maybe a year and a half back say, oh, all software engineers are going to become prompt engineers. That's the next new thing. And all the universities need to change. Not wholly sure about that either. Um, but if you're a software engineer and you're either in the industry or coming into the industry or in it and tired, and you think now AI is the solution, it's going to write everything for me. And I absolutely did say, you know, AI is your, your secretary, your intern that you never had, the budget was not there for. Um, if you just rely wholly on it.

**9:24 Chris Romeo:** Yeah.

**9:26 Sarah-jane Madden:** Uh, that's not going to work out well for you. Your brain is like a muscle that, you know, if you don't use it, will atrophy. Okay. And that will catch up on you. And the other kind of effect that I spoke about is people coming in and, you know, my dad always says a little knowledge is a dangerous thing, you know, getting an overinflated idea of what they can do because You know, when I first started as a developer, I remember sitting, now, maybe I wasn't the top developer in the world, but spending the whole day staring at one page of code going, um, I don't know what's wrong with this. You know, we didn't even actually have internet access where I first worked. That was only for the privileged seniors when something needed to be looked up. You know, we were in books looking at things. So all that is, all that's gone away. Basically, if you have an AI assistant, but you have to be careful not to lose that relationship with the code.

**10:26 Chris Romeo:** Okay.

**10:26 Sarah-jane Madden:** That natural understanding, that engineering ability, you know, if you're just taking it from AI and you're putting together blocks, are you an engineer? You know, you answer me that question, you know, are you truly an engineer? It's a bit like the difference between, you know, a top-end chef who goes farm to fork and sources their stuff and all the rest, as against somebody who takes a microwave dinner and a juice box and goes, there you go, you know, I cooked this. Like, they've both fed a person, you know, but if that microwave dinner was like, do you really know how to make the curry? Do you know what goes in it? No, you don't, you know. And I think that's the danger in terms of people's careers as much as industries and things like that, because there will be some phone call at 3 o'clock in the morning saying it's doing the wrong thing or whatever. And if you've no real relationship with the code, that's going to be a difficult and a bit of a scary place to be. And I, I know that, you know, we've all sat, um, with somebody else's code that's broken. Okay. And I'm sure somebody has sat with us with our code broken as well. And you could argue and say, well, you don't know that code, but you've been, your brain has been there. You've worked it out. You've gone through the processes, et cetera. So it's a pain, but you can step through and see what's What's what's wrong? If you're never quite getting in at that level, I don't fancy that three a.m. phone call.

**11:57 Chris Romeo:** So I'm going to ask kind of what I think is a loaded question now, and then what's your take on this whole vibe coding thing based on how you just explained your current stance on in software development and AI? And.

**12:15 Sarah-jane Madden:** Yeah, it's, it's, can I just get out of that by saying it's not for me? I—

**12:20 Chris Romeo:** You certainly can.

**12:21 Sarah-jane Madden:** I, I don't think it'll, I think it'll be a thing. It'll be a good question for a, uh, software development table quiz at some point. You know, what was vibe coding? Um, I don't think it might mutate into something. I don't think it's— I think it's dumbing it down a little right now. But I think there's a difference between— you kind of hit on something you said earlier, software development. And then in my head I went, or the development of software, where you spoke about the truck drivers. Another thing AI has done is it's seriously democratized. So the development of software, they're not, which isn't necessarily, notice I'm not saying software engineering.

**13:11 Chris Romeo:** Okay.

**13:11 Sarah-jane Madden:** And that's not necessarily a bad thing. And we don't want to be, you know, the gals in the corner saying, oh, you shall not touch our thing. You know, we are developers. You shan't touch. Another one of those studies that I was looking at was saying that when they broke up the general functions within kind of the corporate world and how they embraced AI, that marketing actually embraced it first. Okay.

**13:34 Chris Romeo:** Mm-hmm.

**13:35 Sarah-jane Madden:** And this is similar. It's kind of a tag on to the low-code, no-code movement. It led a lot of people who were fed up doing intake processes, waiting for the development team. I want to try an idea. I'm like, I can do this on my own laptop using this. Now we have AI. That's just gone, you know, and I don't think we should be there trying to take back their toys.

**13:59 Chris Romeo:** Mm-hmm.

**14:00 Sarah-jane Madden:** And I think, I don't think we should be overly proud. I do think there's still a role for computer scientists, software engineers. I don't think anybody who's midway through university should drop out now, you know, and change direction. And we need those people, but we need to be able to influence those other groups and say, look, You're doing it a different way. It has, it fits certain purposes. It's probably a better lightweight approach to get your use case out, which might be a short-lived use case typically, you know, that, that was often the challenge where, again, I'm going to pick on marketing. They wanted something, they had a good idea, but it needed to get out there and be done and be done in, it was only a 3-month thing, right? But the development team had like a queue of 12 months before you, you know.

**14:48 Robert Hurlbut:** Yeah.

**14:48 Sarah-jane Madden:** the idea's gone. You've missed the boat. So, I think we need to be more agile. We probably need to be more agile than when agile came out, you know, and not be so prissy about our positions and roles as technologists. We need to help them and say, look, go you, but here's a few things that you need to consider, you know, and we're terrible for jargon, but, you know, maybe you put those security layers around it without throwing them at the, you know, SDLC. Oh, what do you mean you don't know what the SDLC is? Well, therefore you can't publish, 'cause they're just gonna do it anyway. It's human nature. So, let's partner. But that's where I think we need the people to continue to understand engineering, continue to understand computer science, continue to understand security.

**15:36 Robert Hurlbut:** Mm-hmm.

**15:37 Sarah-jane Madden:** So that you can say, okay, for your purpose, let me help you with that and influence how that's done.

**15:46 Chris Romeo:** But, um, yeah, it just hit me. The, the thing that nobody's talking about with vibe coding is how do you operationalize what comes out of this and run it in an enterprise? There's no way you, you can't take something that was, it's great for a proof of concept. Like it's like if you have a startup idea, you can vibe code your way into a minimum viable product. Now, it's not something you could run in production with 10,000 users because it will just die because it hasn't been architected in a certain way and the AI can't take you there.

**16:20 Sarah-jane Madden:** Yeah. Well, you see, that's the risk is that, you see, you say naturally you can't take it to production, but there's an awful lot out there of it works, why not? And it goes from MVP to STP, straight to production, you know, and what could possibly go wrong? You know, and I think there's enough stories there. It seems naturally wrong to us, but if, again, you're from a different background and you go, well, hang on, we've done this vibe code. And I'm going to be honest, vibe coding. isn't necessarily developers doing it. You know, people kind of say that that's what vibe coding is, but vibe coding has spread, you know, it's because this, you know, and, and when you don't have that background, it's a natural thing to go, why do I now need to inject another X number of days, weeks, months before I can go to production on this?

**17:26 Chris Romeo:** Yeah.

**17:27 Sarah-jane Madden:** Looks like it works to me, you know, and all you need is one or two stories where it works out and that would be precedent for, yeah, all this security stuff and quality stuff. You're just making it up. Or ask the AI, is this software good? Yeah, this software, which I just wrote, is good.

**17:47 Chris Romeo:** Trust but verify.

**17:51 Robert Hurlbut:** Go forth and use it.

**17:53 Sarah-jane Madden:** Yeah, for sure.

**17:53 Chris Romeo:** Where we always want to go. So, when we think SDLC, what, what, I mean, what are we going to get? Like, what is AI going to do for us in the SDLC? And, and, you know, this is something Robert and I have both spent a lot of time kind of in the SDL world from various perspectives, but I'm curious from, from your view, like, what are we, what's going to be the return here? What are we going to get out of this?

**18:21 Sarah-jane Madden:** I think instead of even blocks in the SDLC, and although I don't ever think they were even, you know, some, some, some stages take longer than others, you know, and I think certain things are going to get squashed because they can be executed by AI or with the assistance of AI. And that's going to, I'm going to say this, I know it's weird, but shift us right, shift us, the humans, a bit more right. You know what I mean? Some of the boring stuff, the requirements, you know, getting that in order and all the rest, that can be processed through this digital secretary. You know, you put in prompts, you know, describe what it has to do. Well, it needs to hit these standards. You know, we're subject to, I don't know, GDPR or whatever, and get that out. You'll also get over, you know, Boilerplate code, that type of thing that can be produced by AI. And then have the bits where it gets really sticky, you know, so it's, you know, give an actual developer the time to look at that. Now, I'm reading a lot about people saying, you know, QA or software engineering and testing, that's gone, that can all be done by AI. I think that's underestimating what goes in there, you know. Um, I did have a story that I, I shared in Barcelona about, um, a good friend of mine. He's a senior developer on one of our teams, and he was just playing around with, um, AI, and he had some old legacy code basically that he'd, you know, had been orphaned and he just acquired, and it had no unit tests written for it. And they have really time to go back to these things, et cetera. But he said for the last that he'd get AI to write the unit tests for it. Okay. It passed, it wrote all the unit tests, ran all the unit tests, everything ran fine, et cetera. And he knew there were problems with the code because that's why he was looking at it in the first place. But it had literally just inverted everything in the unit test, which I know is kind of the definition, but it was like, so we just tested that it did what it was meant, what it was written to do rather than what it was meant to do. And that's where the human aspect was in. Because I got my first ever first round of unit tests completely clean out of old botchy code. But if he, if Geoff wasn't wise enough to go, this is a bit of nonsense. You know, you could just say, yeah, there you go. It's full test coverage. It's all perfect. Let's deploy it in a nuclear power station. What could possibly go wrong?

**21:04 Chris Romeo:** I mean, this is just the garbage in, garbage out thing they taught us in university like 100 years ago. Like that, it's, it's, that is the principle though. Like if you put garbage in, you get garbage out. And in this case, it just created the unit tests that made its crappy code pass.

**21:23 Sarah-jane Madden:** Yeah.

**21:23 Chris Romeo:** It didn't, it wasn't, there was no quality metric or quality behind it. It was like, I just want you to go away. It was almost like the AI saying, I'm just gonna give you what you want so you'll go away.

**21:33 Sarah-jane Madden:** Yep. And there was no context to it or anything like that. But, you know, you've hit on the crux of this matter there in, you know, that there are principles that will continue to stand and they will stand when we've quantum computing in mainstream and everything else. And that's one I'm quite anxious that people don't throw away, okay? It's easy, you know, they always say, you know, don't judge people in the past by your present standards and things like that, but it's easy to be with the cool kids and go, oh, you're so boring with wanting your, you know, quality checks and your threat modeling, your this, that, and the other. Why do I need to know the code? I'm like, you will one day, you will. There's certain things That will evolve, but the principles will, the principles are solid and the principles of the SDLC are solid. I've heard a good few people refer to it now kind of changing the S more to a P in product development lifecycle in, you know, and I can sort of see where they're coming from that. Because it's, it's, it's, you know, it's trying to step away from the pure software engineering thing and recognize the AI components to rapid development now, but it doesn't really change. There's lots of people debating it out there at the moment. There's nobody really taking any steps out of the SDLC. Nobody with kind of with a valid argument. They're just kind of moving the blocks around a bit. And I think that kind of is a testament to the fact that SDLC has evolved to a point at this stage. I'm never saying that it's never ever going to change. Never say never. You know, you'll need to put an egg on your face. But it's evolved to such an extent at this stage that the stages are fairly solid and everything around it will change. And that's, you know, some people are getting confused by my message saying, well, hang on, you're saying things are going to change, but things are going to stay the same. Yeah, things are going to change and things are going to stay the same. And that's what I mean to say.

**23:38 Chris Romeo:** Right.

**23:38 Sarah-jane Madden:** We have done this with all sorts of different types of coding languages, different, you know, as I said, the APIs and the microservices and, you know, waterfall to agile and all the rest. Some of the core bits of development have just stayed the same because they're solid. You've got to change the technology around us, and that's what we're doing now.

**23:57 Robert Hurlbut:** So, we sort of hinted on this, but just maybe be a little bit focused on risks that could happen regarding developers over-relying on AI tools. And also maybe another part of that is their skills, you know, potentially atrophying and, you know, no longer being current or relevant. But what are your thoughts on those risks of those developers over-relying on AI and their skills? in general as they continue to use AI as their primary coding assistant?

**24:34 Sarah-jane Madden:** Yeah, I'd ask developers for a bit of introspection. If you're suddenly on easy street or, you know, you're the hero in the office because you're making a mint on AI, it's like a get-rich-quick scheme. And it, you know, it's gonna run out. Probably is a bit too good to be true. Like you say, even if you spend a time, I'm a number of years kind of since I sat in a solid development role, you know, it would take me an awful lot longer to write a quality class at this stage than it would then, you know, and that's just because I'm not in it every day. So why do we think it would be any different? You know, listen, listen to the gray hairs, the voice of experience. If you spend time out of it, you will start to forget. It's like a language, you know. If you spoke a language when you were younger, you know, we learned languages in secondary school and whatever, and we were really quite proficient doing the Leaving Cert, which is the Irish kind of exit exam for high school entry for college. You don't use it. A couple of years later, you go try and order your dinner in France and you sound like a toddler mumbling, you know. The same thing will happen with your ability to interpret code. And I say that deliberately as well, because it's not just about remembering a language, you know, Java, I've had to switch languages. It's not that. You can switch languages. It's the constructs that you start to forget. What exactly is that? Having a rough idea when something's broken in particular, having a rough idea isn't good enough. You know, it's like, ooh, is that public? Is it private? Is it, what is it? You need to know exactly. And that's where I think it's just gonna put that distance between us. Like people using AI to write love letters and messages to their significant others. And whether they're true or not, there's some great stories out there, you know, about deciding this was going very well, let's have the AI continue the conversation, and then they meet back up in person and it's all gone a bit awry because the conversation has been with the AI and not with the other partner. So.

**27:00 Chris Romeo:** It's funny because that sounds like the plot of a movie from like 10 years ago. That would've been the plot of a movie, you know? They had the person's accent. They thought they were talking to their significant other when in fact they were talking to a computer.

**27:13 Sarah-jane Madden:** Computer.

**27:15 Chris Romeo:** I could hear the, the, in the, at least in the United States, the same person does the voice of all the movie trailers. And I can hear that voice in a world where computers are having a romantic relationship or something.

**27:26 Sarah-jane Madden:** I don't know. We have, The Simpsons have told us all about this complacency. Do you remember when Homer had a little bit, he ended up wearing a smock, let's just say, and he was working from home. And then he got the bird that could dip and do his job. And then he was like, I don't really need to be here. He had this nodding bird thing and ended up nearly blowing up the plant until he got himself stuck in the chimney of the cooler. And it's the same thing. You could see the progression. They kind of predicted this. You know, he was like, I don't need it. I could just do this, easy street. That's what I'm afraid of within our development community. So I think you really need to look at AI as something that's taken the grunt work out of development. Yeah, I think really good for documentation, you know, because you say, hey, I've done this now, you can write it up in that nice format thing to, you know, for whatever purposes, be it customer documentation or audit purposes, you know, that's all very time-consuming. And you've already done the engineering bit, use it for that. Free yourself up for that. I think it's a really good opportunity to get back to doing what we love. The transition from being a bedroom coder or, you know, leaving university where, you know, you did your final year project and it's all fun and all the rest. into a commercial environment where there's meetings and there's planning sessions and there's documentation and there's this, that, and the other. And you will hear, you know, you guys have met plenty of people, I'm sure, who said, I actually only code for X percent of my week now. And, you know, this isn't what I got into it for. You know, I liked coding. This is giving you a chance to get some of that back. I'm not saying you're going to be able to be at your desk you know, headphones in for 40 hours of the week. You know, there's, there's still some trimmings go with it, but it's an opportunity should we choose to seize it.

**29:28 Chris Romeo:** Sounds like you just told me that I can have the AI go to all my meetings so that I can just code.

**29:35 Sarah-jane Madden:** I think that's the idea. You get home where it's bird, maybe.

**29:38 Chris Romeo:** Well, that's a whole other podcast about what's wrong with meetings in the world, but Yeah. So let's, let's kind of, we talked about the, kind of the impact for developers, how the world is changing, but it's not changing. When you think about the technology behind this, like, how good are the results that are coming out of these things? Or how bad are the results? Like, can we really trust this stuff that's coming out of these large language models? No.

**30:11 Sarah-jane Madden:** One thing I'll give you a definite answer on. At the moment, no. I was working on another presentation and I asked a given AI tool to give me some quotes to support what I did, because we all know in the world, I can say something, I can say my shirt is blue, but it sounds much better if we say the esteemed, my esteemed colleague, blah, blah, blah, says it's blue. And it's like, ooh, it must be blue. So. And I got some great quotes back and I was only playing around with it, but I was like, hey, they're true, then, you know, onto something. I went down some rabbit hole with, none of the quotes are accurate. And this was, this was a fairly basic thing. They were all summaries of articles written about something somebody had said and complete hallucinations because I went to the point of Saying, okay, in this version of this book, ISBN this number, what page are you finding that direct quote on? And it was telling me page 22, second paragraph, whatever. And I was like, book is open on my desk, not there. Oh, you're right. If I heard the read one more time, oh, you're right. And it actually, most of what it had given me just didn't exist at all. They were not. And I'd very specifically asked it and then re-asked, are these direct quotes? Are these, you know, just wasn't, you know, but how many people are doing that as against taking the first answer? Now you start to think about that in, fine, that was for a PowerPoint. Okay. But what if people are doing this for medical diagnosis, accounting, you know, critical decision-making? It starts to get a little bit scary. And then we bring it back to code again, probably give you a piece of code that runs. What's it doing? I think we're giving everybody nightmares, but—

**32:19 Robert Hurlbut:** Yeah, right. But no, we've all seen that. Yeah, the hallucinations and I've seen that in Uh, just some things that I've been working on where people are putting in, and very authoritatively, this is what I found. And I go look it up, like, that doesn't even exist.

**32:40 Chris Romeo:** No, you didn't.

**32:41 Sarah-jane Madden:** You end up in a childish conversation with the AI. Oh no, you didn't. Oh yes, I did.

**32:46 Robert Hurlbut:** Exactly.

**32:47 Sarah-jane Madden:** But again, talking about principles, Stan and Tessa, trust but verify. How long have we been saying that for? This applies in this situation as well. You know, I'd say maybe use or leverage, but verify.

**33:01 Chris Romeo:** You know, now I'm concerned that I think the source of that quote is Ronald Reagan, famous United States president. But now you're making me like, don't ask AI. I gotta go research it some more. Yeah, I gotta go confirm that like in a book in a library somewhere that he actually said that.

**33:17 Sarah-jane Madden:** Yeah, 'cause if you ask AI, it'll probably say, yes, he did. And then if you ask them the next sentence, go, did Neil Armstrong say it? And go, yes, you are in fact correct, Chris. Yes.

**33:26 Robert Hurlbut:** Actually, yeah, you're correct.

**33:27 Chris Romeo:** He did say that. He probably did say those 3 words at some different time in his life, but not necessarily in that order.

**33:34 Sarah-jane Madden:** Yeah.

**33:35 Robert Hurlbut:** Bringing it back to teams and developers, what could, or maybe some practical steps that engineering leaders could take to ensure their teams are using AI responsibly?

**33:51 Sarah-jane Madden:** Right. So I think, and security teams will have a lot to say about this, but, you know, you need to, it's like the whole open source challenge of a couple of years ago. You need to be fairly strict on what tools can and can't be used. You should have a process for them to propose new tools and evaluate them, but not all AIs are equal. You know, it depends on your geography as well as to what ones are acceptable. It depends on your industry. Okay. You also need to look at your industry and see, look, is this acceptable at all, at all, at all? Okay. To be using this. And, you know, devices should be managed, web traffic should be monitored, so things should be blocked if they're not allowed. Okay. You'd like to say to people, please only use Copilot or whatever your company has approved. But I would still recommend that there are, you know, people will get stuck, people will Sometimes people think they know better, you know, and they'll go off the rails. And it's not to curb their enthusiasm or kill innovation. That's why I say you should have a process for them to propose something new and say, look, I've heard of this new thing. I know we've paid a fortune in licenses for whatever, but, you know, at home I've had great results with this. Can we have a look into it? And have your security team or maybe procurement team or something that has a look and go, can we use this? You know, it might be the next best thing and you don't want to throw that out. As a development manager, I would say, you know, partner with your team. Don't hang down diktats and say, okay, where are we going to use this, folks? You know, you know, and look at your SDLC and say, which bits are taking us time? Which bits do we not actually see a huge amount of value in us individually? Like, if I gave each of you an intern in the morning, which bits would you fob off on them? Right now, can we get the AI to do that? And then agree. So try to still have something of a structure, maybe not a structure, a framework that everybody's working to and say, you know, I know lots of people are using it to create their DFDs for threat modeling now and things like that. Say, okay, I spent forever dropping boxes and things. I can now describe it and have it drawn. for me and then review it. But decide that the whole team is going to do that. You know, a bit like you're— just think of it like coding standards and whatever. It doesn't mean somebody can't innovate and we circle back around and change the standards. But try and get them working on the same areas that make sense so that you know where AI is in your development process, whether that's assisting you in writing the code. or is actually in the code itself, in going into the product. And that's about the transparency. You know, make sure you know where it is, make sure you know what you're using, and have that process that they can introduce new ideas as well.

**36:46 Chris Romeo:** How about a key takeaway for the audience here based on all the research you've done in AI and development and security team integration? Like, how is it— could you sum it up? As a key takeaway.

**37:02 Sarah-jane Madden:** Um, use it, but don't lose it. And I mean lose it as in to go nuts, uh, as you know, keep your feet on the ground. Don't, don't get left behind. You know, it is, it is the new way. Um, AI is not evil. Um, but we've, we really do need to use it responsibly and thoughtfully. Um, so I wouldn't hang up your keyboard just Yes. Um, I, I think embrace it, but just always a little bit of cynicism, a little bit of that trust but verify, you know, and be a bit of a control freak. Besides, you do need to understand what your code is doing.

**37:47 Chris Romeo:** Sarah-Jane, thanks for joining us again on the Application Security Podcast, and we look forward to another conversation next year sometime.

**37:57 Sarah-jane Madden:** Thank you for having me. Thank you both.

---

Source: https://appsecpodcast.com/sarah-jane-madden-what-ai-means-for-appsec/