---
title: "Tanya Janca - Secure Vibe Coding"
url: https://appsecpodcast.com/tanya-janca-secure-vibe-coding/
date: 2026-04-30
duration_seconds: 2877
season: 13
episode: 3
guests: ["Tanya Janca"]
topics: ["Threat Modeling", "Secure Development", "AI and LLM Security"]
audio: https://www.buzzsprout.com/1730684/episodes/19100844-tanya-janca-secure-vibe-coding.mp3
video: https://www.youtube.com/watch?v=jChGM0NXplM
transcript: true
---

# Tanya Janca - Secure Vibe Coding

*April 30, 2026 · 48 min · Season 13, episode 3*

with [Tanya Janca](https://appsecpodcast.com/guests/tanya-janca/)

on [Threat Modeling](https://appsecpodcast.com/topics/threat-modeling/), [Secure Development](https://appsecpodcast.com/topics/secure-development/), [AI and LLM Security](https://appsecpodcast.com/topics/ai-security/)

[Audio](https://www.buzzsprout.com/1730684/episodes/19100844-tanya-janca-secure-vibe-coding.mp3) · [Video](https://www.youtube.com/watch?v=jChGM0NXplM)

## Show notes

If AI writes all the code and the developer barely reads it, where does AppSec fit? Tanya Janca returns to define vibe coding and explain why models trained on insecure public code do not understand secure design by default. She and the hosts build a practical secure-vibe-coding framework: explicit requirements, human-led threat modeling, reusable security prompts, iterative review, SAST, and independent testing. Tanya shares hard-earned examples of Claude removing error handling, models confidently reviewing their own insecure output, and developers accepting enormous finding backlogs for code they did not enjoy writing. The discussion also examines whether security tooling will consolidate into AI platforms, how AppSec must be reimagined, and why continuous, embedded guidance matters more than occasional training. Tanya closes by introducing her DevSecStation podcast.

Connect with Tanya Janca:  
→ [Tanya Janca on LinkedIn](https://www.linkedin.com/in/tanya-janca)  
→ [SecureMyVibe](https://securemyvibe.ca)  
→ [DevSecStation](https://www.youtube.com/@DevSecStation)

Mentioned in this episode:  
→ [SecureMyVibe](https://securemyvibe.ca)  
→ [OWASP Top 10](https://owasp.org/www-project-top-ten/)  
→ [Burp Suite](https://portswigger.net/burp/pro)  
→ [OWASP ZAP](https://www.zaproxy.org)  
→ [DevSecStation](https://www.youtube.com/@DevSecStation)  
→ [Tanya Janca (SheHacksPurple)](https://shehackspurple.ca/)  
→ [ChatGPT](https://chatgpt.com)  
→ [Stack Overflow](https://stackoverflow.com)  
→ [The Security Table podcast](https://podcasts.apple.com/us/podcast/the-security-table/id1659280767)

Chapters:  
00:00 Tanya Janca returns  
02:10 What vibe coding actually means  
04:03 AI adoption and how developers prompt  
05:57 Treating AI like an intern  
07:28 Do AI systems need parental controls?  
09:34 Security prompts for everyday development  
11:47 Models, memory, and protecting sensitive data  
14:11 What happens when Claude goes away?  
16:02 The most dangerous vibe-coding misconception  
18:06 Threat modeling before AI writes the code  
22:48 Using AI throughout the development lifecycle  
25:32 A reusable secure-prompt framework  
29:09 Expose security assumptions and challenge flattery  
32:30 Reviewing an AI-generated codebase  
36:09 Will AI platforms absorb security tooling?  
37:39 Five million findings for code nobody wrote  
42:19 The remaining pieces of secure vibe coding  
43:52 Reimagining AppSec  
45:25 Continuous guidance beats occasional training  
46:05 Introducing DevSecStation  
47:12 Closing thoughts

## Transcript

*8,782 words · assemblyai*

**0:00 Chris Romeo:** Tanya Jain Ka, aka SheHacksPurple, is an author, founder, trainer, speaker, software developer, but most of all, a nerd obsessed with security. She speaks and teaches secure coding worldwide and through her podcast, DevSecStation. Quick note, we're opening up a few sponsorship spots on the Application Security Podcast. If you want to get in front of real AppSec practitioners, the folks actually making decisions, this is a great place to do it. Just reach out to me, Chris Romeo. I'd love to chat. You can find me on LinkedIn. Hey folks, welcome to another episode of the Application Security Podcast. My name is Chris Romeo, and I'm a farmer. Well, I'm a farmer who cares about application security. Joined as always by my good friend and decade-long co-host, Robert Hurlbut. Hey, Robert.

**1:01 Robert Hurlbut:** Hey, Chris. Yeah, Robert Hurlbut, and principal product security architect and threat modeling trainer at Torreon. And not doing any farming, but I definitely like that as a farmer. That's a great thing to do.

**1:17 Chris Romeo:** It's a good goal for all of us to be outside. And joined once again by Tanya Jankov. Tanya does not need any introduction. She has been on the podcast 8 times, we've counted. And she is now the— you're almost a co-host. I mean, 2 more episodes and we're going to have to say Tanya is actually— you're definitely a contributor. I don't know, what do they call it on the correspondent or something? They have some fancy name, but you're there.

**1:46 Tanya Janca:** Correspondent sounds good. And I have a flower garden, so I fit in.

**1:50 Chris Romeo:** There you go. I mean, correspondent seems like you should be like, now we're going to Tanya on the scene. She's reporting on whatever. Okay. would you even go in application security if you were on the scene?

**2:01 Tanya Janca:** Like, where would you be?

**2:02 Robert Hurlbut:** I'd go to conferences.

**2:03 Chris Romeo:** Probably conferences, yeah.

**2:04 Tanya Janca:** I'd be in data centers. I'd be on-site at client sites responding to security incidents.

**2:10 Chris Romeo:** That's right. We can't, we can't tear Katja away for long. She has to get back to fixing the problems of the, the building, the nondescript building she's in that doesn't have any windows and has good air conditioning. But, all right. Well, we got a lot to talk about here because AI and vibe coding is a thing now. And so, I thought we'd start, Tanya, just by giving you an opportunity to define this concept of vibe coding, because I've seen people describe it in different ways, and I'm curious, one, how you're thinking about it, but also, it'll help us set the stage for the rest of our conversation.

**2:48 Tanya Janca:** Okay. So, I would say that some of us are using the AI to help us write code. And that, I would say, is using an AI assistant or using the AI. Where I would say vibe coding is where you have the AI writing all the code. It, like, you are not coding anymore. It is coding. You're instructing it, maybe you're driving it, but it is doing basically all of the heavy lifting in regarding to coding. You are probably not reading it, and then you are submitting it as your own work. That tends to be the vibe coding thing. And Chris, I'd love to say that I have never done anything like that, but I I was recently, like, giving some training, and they said, can you switch all your examples to Spring Boot and Java? And I said, sure. And I read, like, the first 40, 50 examples, and all of them were perfect. And so, I ran out of time, and I get to the client site, and then we got to example, like, 64, which was some error handling, and it had decided to remove all my error handling and make it so the global exception handler just caught every error. And I was like, what have you done, Claude? And so, there I am. I'm live, like, showing the client, I'm like, so, I'm a moron, in case you were wondering.

**4:03 Chris Romeo:** Well, it's a lesson. It's a lesson in using AI, though, like, right in front, because, I mean, the numbers I've heard, 98% of developers are using some type of AI assistant at this point. Like, it's, I mean, it's everybody. I wanna know who the 2% are. Like, who are the 2% that are not and are still, you know, staying afloat in the world of development?

**4:26 Tanya Janca:** It's not gonna be long for those ones. You know, when tools come out, you can't just say, I'm not gonna use the tool everyone's using. That's not gonna last long. I do think though, I see people using it at different, what would be the word, efficiencies.

**4:42 Robert Hurlbut:** Mm-hmm.

**4:43 Tanya Janca:** So, I was giving training last month and this dev was like, well, it hallucinates and the code it writes is terrible. I'm like, cool, when you ask it to do something, how do you ask it to do it? Like, and he's, he basically was like, build me an app that does this and does that. He didn't say the stack, he didn't say what the architecture should— like, he just said, build me an app that does this. Yeah, if you ask any dev, build me an app that does this, and give them— and those are all of the requirements, they're gonna do a crappy job. And if you give them such a huge piece of work, of course, they're gonna hallucinate. And I'm like, we need to break it down like this, and this is how. And so, what we did was like an AI security workshop together. So, his question was perfect, right? 'Cause there's a multi-day training and later in the training we were gonna use AI. So it was a great chance for me to be like, blah, blah, blah, I am smart and know things. But basically I think that a lot of people aren't using it in a way that's necessarily safe. Like just the same as like power tools are awesome, but you should probably wear ear protection and eye protection and you know what I mean? And use it in a safe way. And so I feel like part of the issue with the vibe coding is like, Yeah, you can ask it to do many things. You can't ask it to do a gigantic thing with very little instruction and expect good results.

**5:57 Chris Romeo:** Some people are using this, this illustration of AI is an intern. You should treat AI like an intern. You should treat AI like a junior developer. What are your thoughts on that, that kind of illustration or way of thinking about the AI?

**6:19 Tanya Janca:** So that is a lot how I think about it. So I actually have interns this semester for my little tiny company to do some marketing for me. And it's so funny because it's so similar that it's almost scary. So very enthusiastic, very positive at all times. You give them something, they're excited, they run off, they do a whole bunch of stuff, you get the work back. Ooh, the work is not right. You have to adjust it and prune it and change it. And so with a person, you give it feedback over and over again and they improve, right? So I have one of them editing videos for me, and the first set of videos were a disaster, right? But now like the third one, I had only one edit in a long video and every other thing was perfect. And now she's onto the fourth video and I think there'll probably be no edits. Do you know what I mean? And, and so like doing that feedback has helped and you can do that with the AI as well. But I would say, yeah, if you could manage them how you would manage an intern. So very excited, bright, creative, but not experienced, not knowledgeable. Does that make sense?

**7:28 Chris Romeo:** Yeah. Yeah. We've been, we've been talking about this idea of thinking the industry seems to be going towards trying to think of the AI as human. try to get closer to that. And it was either Yizar Terandash or Matt Coles on my other podcast, The Security Table, where they brought up this idea of we need parental controls for AI. This idea, and it was just a fascinating concept. It kind of blew my mind in the moment. I know it wasn't I that came up with it because it was one of them, because I had this like moment of, wow, that's a really neat idea. Like, is security for AI parental controls if we're thinking of this thing as as human?

**8:12 Tanya Janca:** Okay, so I feel like the humans need the parental controls because we're the ones putting the super sensitive data in we're not supposed to on the regular because we're the ones that are using the tool and forgetting about safety when we do it. We're supposed to have this human in the loop. A lot of the people aren't doing the human in the loop thing that they say they're doing. I think Anthropic, I was reading Resilient Cyber, the newsletter, and in it, Chris put a thing from Anthropic that said that on average, the commit review time is under a second. And I don't know about you, but I can't review a pull request in under, I think I'm amazing, please don't get me wrong. I'm not like low confidence here, but I need more than 1 second to review a pull request. So, I feel like the humans need a parental control to a certain extent. And then, I guess for the AI, Yeah. So, a lot of people have talked about having guardrails, like privacy guardrails. So, where it stops and it's like, that looks like private information, we're not going to accept that. I've had people talk about security guardrails, but no one's really given me a clear idea of what those are. They're like, security guardrails. I'm like, what do you mean? And they're like, guardrails that do security. I'm like, yes, I heard.

**9:33 Chris Romeo:** Okay.

**9:34 Tanya Janca:** I have been giving away a free AI prompt library, and that is what I've been using with clients. And one of them wrote me today, we did a training last week, and he said of his like 200 or 300 developers, only 2 of them are not using it every single day now. And I'm like, yeah. And I just basically, I took my most recent book, I boiled it down to a secure coding guideline that I was working with my clients with. And then I was like, you should turn it into prompts. They're like, you should do it for us. And I was like, you are right, I should. And so, and that's what it is, is just like I've boiled down that guideline into prompts so that you're getting better results. That's what I would call, like that's how I am operating for my guardrails, if that makes sense. But when you say parental controls, I think of, I know better than you and I'm trying to protect you from hard things that you shouldn't have to deal with yet till you're more mature. So like you wanna protect children from seeing adult content and adult content's not just naked people, right? Like there are upsetting things on the internet where people are still wearing clothes and I wanna protect them from that too. And I don't feel like we're protecting the AI. Does that make sense?

**10:55 Robert Hurlbut:** Or—

**10:55 Chris Romeo:** Yeah, it's kind of on its own.

**10:56 Tanya Janca:** Do you think we need to?

**10:58 Chris Romeo:** No, I mean, it's, it's, it's, it's, It's certainly doing kind of— it appears to be doing kind of whatever it wants behind the scenes to some degree. And some of that is the way it's been designed and coded. And, you know, it's allowed to absorb information from, you know, this idea of creating something and it belonging to somebody else has just been smashed in this new world of AI where they're just— training data is being taken from anywhere. And good luck trying to prove in a court of law that this training data came from this source. Like, I've I'm sure there's a case brewing, there'll be one, but I don't know that there's, I don't even know that they're going to be successful in trying to prove like, okay, you have my confidential information in your training dataset and it's a copyright infringement. Like, I don't think that's happened yet.

**11:47 Tanya Janca:** I was at an AI summit at Sector in Toronto, Canada, and one of the, there was like a sponsored talk and I wish I remembered, it was some giant security company got bought by some other giant security company, and they're all buying each other, so it's hard to keep track. And basically, he was saying that so many employees were asking how much their stock options would be worth over and over and over again that the AI started suggesting to investors that they buy stock immediately in this one company because it was getting acquired by this other company. And it started giving that as suggestions outside of the company 2 or 3 weeks before the acquisition, and it messed up the acquisition. Like, it still happened, but each individual employee didn't think they were leaking that data. They're like, theoretically, if like maybe this company got bought by that company, but you know, if 400 employees ask that, the AI's not stupid, even though it's not intelligent the same way we are. It can reason to a certain extent, right? Like, Like, that's very obvious.

**12:55 Chris Romeo:** Yeah. Gone are the early days of LLMs where it was like a one-way transaction, almost like it didn't have the ability to store what was coming back into it. But those days are long gone at this point.

**13:07 Tanya Janca:** Mm-hmm. Mm-hmm. And it, I am interested about how you're like the parental controls, 'cause like, do we protect Claude? So it's certainly Anthropic's job to protect Claude. And I know recently they were in the newspaper or in the news. Hahaha. Newspaper. Haha. I'm old.

**13:26 Chris Romeo:** Sorry.

**13:27 Tanya Janca:** But they were in the news about how they accidentally leaked their source map and then their source control, source code was in there and intellectual property, people finding vulnerabilities in it, et cetera, is an issue. But it's theoretically Anthropic's job to protect their product, but should we as a society protect the AIs? Because they're so powerful. like critical infrastructure.

**13:52 Chris Romeo:** So an issue is going to have to— it's going to have to be dealt with in the coming years. It's just another thing on a very long list of things to be figured out about how this technology is kept inside of some amount of a box and not allowed to expand and do everything.

**14:11 Tanya Janca:** Yeah, because there is a company that posted on LinkedIn that apparently they had violated the usage policy and Anthropic cut them off as a company. Yeah, I saw that. We can't operate. And Claude went down like a few weeks ago and I was writing a research paper with it. And I was like, what do I do now? I'm so bored. I'm like, I guess I'll play with ChatGPT because ChatGPT is giving me singing lessons right now. I used to be a professional singer when I was younger and I was like, I kind of want to build my voice back up because it's a muscle, right? And I was like, what could I do over the next 8 weeks to build my voice like back up to somewhere close to where it previously was. And so, I'm doing singing lessons with ChatGPT, which is kind of funny. But yeah, like if we don't have access to AI, like what would that do to industry, right? I know I'm off topic. I'm sorry, Chris.

**15:05 Chris Romeo:** Robert, you're so patient. No, this is why we— there's so many things in this space right now that there, I mean, there's a universe of things, of directions you can go and The challenge is we don't have a lot of answers. We just, you know, we're asking a lot of questions, but we don't really know because we don't know where the technology's going.

**15:24 Tanya Janca:** It's true. I actually had a Claude window up and it kept flashing at me, so I just minimized it because it was distracting. Because it's like, let's keep working. I'm like, no, Claude, I'm busy. I'm hanging out with my other friends. Stop flashing the browser at me.

**15:40 Chris Romeo:** I didn't even think of that as Claude becomes the developer's best friend. And you're gonna see people like at a bar or something and they have their laptop sitting on the table across from them. It'll be like a little face on it. You know, like, there's my friend. I brought my friend here with me. Uh, Robert, get us back on track here. Okay.

**16:02 Robert Hurlbut:** All right. Well, Tanya, uh, what's the most dangerous misconception for those who are deep into vibe coding workflows?

**16:09 Tanya Janca:** That the AI understands security. That it understands secure coding, that it understands secure design, that it gives a crap whatsoever about those things. Because the dataset it trained on is the internet. It scanned my terrible GitHub repo as part of its training, and there's some crap in there, let me tell you, Robert. It scanned tutorials.

**16:37 Chris Romeo:** Yeah.

**16:38 Tanya Janca:** It scanned Stack Overflow. And here's the thing, is like the thing that's voted to the top of Stack Overflow is not usually the most secure way to do the thing, right? 'Cause we want it to, the solution to work every time. And so, what do we do? We remove the security, we set CORS to star, we remove the restrictions to make sure it works 100% of the time. And we essentially taught it security is low priority and/or completely optional.

**17:07 Robert Hurlbut:** Hmm.

**17:07 Tanya Janca:** That's what we taught it. And so, then people are like, oh, well, the AI's so smart, of course it understands how to write secure code. And you could even ask it. So, I did a talk at RSA called Insecure Vibes. Hahaha, I think I'm clever. Anyway, in it, I did a demo where I asked Claude to make, you know, login function for an insulin pump. I'm like, this is a medical device, so it needs to be really safe. And it's like, for sure. And it goes off and it makes it. And then I asked ChatGPT to analyze it for vulnerabilities and it found a ton. And then I asked Claude, can you just look at it and tell me? And then it put in all caps, do not put in production. There are critical vulnerabilities. This is not safe. You cannot put this in someone's body. And that took 4 minutes.

**17:52 Robert Hurlbut:** Hmm.

**17:54 Tanya Janca:** Right? And it said in its requirements, I will make this secure. And then it reviewed itself and it could see it wasn't, right? And so, how do we address that? Which I guess is what we're going to talk about today.

**18:06 Chris Romeo:** Yeah, we're going to get there. So, let's talk about threat modeling for a second here, because especially in this vibe coding definition that you provided, people are build— they're describing something and then having it built, but not really ever understanding how it works. And so, when we think about threat modeling as something that used to be a design time, like, let's come up with a design first, like, where do you see threat modeling fitting in this secure vibe coding framework?

**18:40 Tanya Janca:** So, I think we need to adjust our system development lifecycles to be more AI aware, because we still have to have a plan of the thing that we're building. And we can make that plan with the AI. We can talk about the plan with the AI. It sucks at making design documents, like FYI. It sucks at making, like, because I was like, draw this for me. And it looked like crap. I suck at drawing and I suck at design, like visual, I can't make things pretty. That's not a skill I have. And it's worse than me and I suck. So, spoiler alert, it's not great at diagrams yet, but you can still design a diagram with it. You can still talk out your design with it. So, I would still want to do a manual threat modeling event personally to talk about, like, assuming you're with a team, Right? To talk it out with the team so everyone understands the design so you can speak about it. But I would also ask it what threats I've missed, what other mitigations I could do for the threats I found, if it thinks I've evaluated the threats correctly. I like to have it check my work and then I like to have a different one check its work. Like when I did, I helped write the new OWASP Top 10. I'm one of the project leaders and I would write the thing and come up with all the mitigation advice I liked. And then I would ask ChatGPT to check in, I would ask Claude to check it, and both of them were like, this looks pretty good. And then I brought it to the team and was like, what do you think? And they're like, this is pretty thorough. I'm like, yeah, 'cause I already had a team of experts looking over it with me, right? 'Cause like, I'm like, this is my opinion, this is what I want, but I'm also at a point in my career, which both of you are as well, Where we feel confident enough and opinionated enough to be like, no, Claude, you're wrong. And this is why. And I do that a lot where I'm like, you are wrong. This is not true. This is what we should do. And I feel that I am right and I'll explain why. And usually it's like, I've only once had it pushed back and be like, no, Tanya, here's a bunch of things you hadn't thought of. But most of the time it's like, oh crap. Yeah, that's true. The junior people and the intermediate people aren't there.

**20:59 Chris Romeo:** Mm-hmm.

**21:00 Tanya Janca:** And they can't see, oh, when it says the threat model is this, it can't see that that's wrong or incomplete or in the wrong direction or in the wrong priority order necessarily. I'm not saying that junior people and intermediate people are not smart. That's not what I'm saying. I'm saying they don't have the experience and often the confidence to say to a very confident-sounding, opinionated machine, Nope. So, I still think that threat modeling's important for team building. I think threat modeling's important to exercise our brains. I think threat modeling's important because we see things differently than a machine does. And it just like how diversity matters in threat modeling, not from a politically correct perspective, but your threat model does not match my threat model perspective. And we can't actually see all the genuine risks if we all have a different way of looking at things. Like, I helped a startup once and the team was all white men who were mostly gay. And they did not see this huge threat that women that used their system might have ex-male partners that wanted to find out where they were and beat them up. And like, the guy's like, why would you worry about that? I'm like, you're 6'4 and like 250 pounds. No one's going to come find you and beat you up. They're running away from you, buddy. And like, 'cause the team didn't really have women in their life to talk to about this, they didn't see it. And he's like, oh, I guess this is why diversity matters. And I'm like, yep. And I'm glad that's not his threat model. I want it to never be anyone's threat model, right? But the AI certainly doesn't have a threat model that they're worried someone's gonna come beat them up. Do you know what I mean?

**22:47 Robert Hurlbut:** Yeah.

**22:48 Tanya Janca:** Yeah.

**22:48 Chris Romeo:** So, if I draw out kind of a piece of the framework then for secure vibe coding, it would include one of the big takeaways I just kind of gathered from your descriptions is there is a design time activity that you should be doing with the AI. Not just turning it loose and saying, I want an app that does X, Y, and Z, but we should be spending time as architects of what we're building with the AI, working through the design and creating a design that we as the builders, developers, but also the architects are good with. Like, okay, I agree, this makes sense from a design perspective. Then we move on to coding. And so, threat modeling kind of fits into that segment. And there could be a future where the AI is participating in that threat model as well, but it's at that design time. It's not like, let's go build something and then find all the stuff that's wrong with it. That's not the threat model.

**23:54 Tanya Janca:** No. I would say that I would want to use it at every single part of the project. So, like, when I'm doing the requirements gathering, running it through it and saying, like, have I missed things? Is this clear? Like, What other things should I have? So, I have a list of requirements that I use for different projects. So, if I'm going to do a serverless app, I'm going to do a WebSocket, whatever, it's like, these are the security requirements that I have every single time that I would want to start with. And so, I would want to make some prompts with those and say like, am I missing anything in this requirements document? Is this clear? Are you going to know how to code this? Is my team going to understand this? And then taking that and then moving on to design with it. It's like, so these are the pieces I think I need. Do I have the right pieces? Do I have all the pieces? Should I do, you know, more of a microservice architecture or should I do bigger APIs? Like, do you know what I mean? And like work through that and talk through that like you would with your team. Does that make sense?

**24:51 Chris Romeo:** Yeah, yeah, because then you end up with a solid design, which then there's, it's, we're not going to go back to the days of waterfall quite where we do the design and then we kind of, but we're getting a little more, I'll call it, I'm going to invent a word, We're getting a little more waterfall-y by spending that time upfront. Whereas in a, in a, in the way that agile kind of played out with development teams, it was like, okay, we're gonna start with a basic idea and then we're gonna iterate on it until it gets to a finished product. It sounds like with AI, that's not gonna be as successful of a path if I go down this, if I try to get it to iterate with me to a final product.

**25:32 Tanya Janca:** I think it could iterate with you, but you need to have a set of prompts that keep you on base for your security requirements. So, the AI prompt library thing that I made, I'm giving it away free. So, if you go to securemyvibe.ca, because I'm Canadian, A, also because .com was taken, but whatever. The idea is that there's a prompt that runs every single time that you run code from then on for your whole organization that tells you security assumptions, that applies a secure coding guideline that makes it just that much better, where it mentions all the security things to you and all the steps it expects you to take, right? And then there's different levels and different prompts. So, for instance, I'm designing this, I want to deep dive into authentication, whatever it is, the thing that you're doing to help hold your hand to get you through, to make sure you do a better, more secure job of each thing. And so, if you're iterating, it's fine. But if you have like a prompt template that helps ensure that you are filling out all the correct things, like telling it, I want this framework, I want, like, I don't know, we're going to use OAuth, or we're going to do this, we're going to do that, just filling in the blanks and then adjusting it so it's the thing that you want, so it's more clear. Because I remember when I first used Burp Suite the first time, I took down production because basically my professional mentor said on Friday evening, learn this Monday morning, you are doing your first pen test. Gave me no supervision over the weekend, then handed me keys to production, which in retrospect was not a good idea, right? Basically, he handed a scalpel to like a 2-year-old and was like, don't get hurt, right? But if you have a pentester that's extremely experienced using Burp Suite, they're an artist, right? They know exactly what they're doing. They're not going to break things. They know what's up. And eventually, I got better, but I would not say I ever hit that level. And I feel like, you know, this is a huge, amazing, very powerful tool that we're putting in people's hands, and then we're not showing them how to use it. We're not giving them directions. We're like, yeah, just chat with it. And so then everyone's driving it at a different rate and some people are like driving 400 kilometers an hour to the grocery store with it.

**27:54 Chris Romeo:** Yeah.

**27:54 Tanya Janca:** And they, and they have the parking brake on the whole time.

**27:56 Chris Romeo:** I like that. That's a great illustration. One of the things I've enjoyed about Claude, and I'm very much a newbie Claude user, is that Claude, over everything else that I've seen, tends to ask me more clarifying questions about things. And like I said, I've been using ChatGPT exclusively for the last couple of years and just recently jumped over to Claude to play around with it. But that's one of the things that I thought, to your point about, you know, kind of working out those requirements and making sure the AI understands it, it feels like Claude's got a better way of going, well, I understand at a high level what you're talking about. Let me ask you a few clarifying questions to help me dial in on what you're really saying. And I think that's a powerful thing if we apply that concept to security requirements, security design, clarifying and having an AI that's going, okay, I don't really understand this all the way. I'm going to ask you, I'm going to draw things out of your brain to help me lock in on what you're trying to get me to do. Like, that's, that's a feature that I see that's gonna, that's gonna be really powerful as time goes on.

**29:09 Tanya Janca:** In the, the level 1 prompt that I shared, it tells it to tell you every security assumption. And when I wrote that, I was just like, I just wanna know what it's thinking. But now as it's doing it to me every time, I'm like, oh, what are you thinking, dude? Like, 'cause it's like, oh, I'm assuming you'll rip out this authentication and do a better job. Or I'm assuming you wouldn't store all these secrets like I did, or I'm assuming you would get a library that exists. And I'm like, well, thank you, sweetie, for telling me that, because I might not have noticed that library didn't exist. And it is very interesting to me that, how do I word this, that it doesn't necessarily tell you those by default. But I agree with you that Claude asks more clarifying questions. I find that ChatGPT does this thing I really do not like where it's like, Do you want me to tell you this other thing too? Because I could if you wanted. Like a clickbait to try to get you to keep talking. And I've actually instructed it repeatedly. I'm like, put in your memory that that sucks and I don't like that. Just either tell it to me if you think I need to know it, or don't ask me if I want to know. Because most of the time it's this off-topic thing. Like, it's like, I'm like, you know, could you help me rewrite this abstract? for this conference that's, you know, 400 words. Can you make it 300 words? Because, you know, this other conference wants it to be shorter. And it's like, do you want me to make it like some social media posts for you? And then also I'm like, no, calm down, calm down. Stop using all my tokens on your weirdness.

**30:43 Chris Romeo:** I've seen that too though. I hadn't really crystallized it, but yeah, you do see that with ChatGPT a lot where it's like, but I could also help you start a lemonade stand. and dominate the local lemonade stand in your neighborhood if you really wanted me to. No, I just wanted you to tell me if there was a vulnerability in this code. I didn't really want to own a lemonade stand.

**31:05 Tanya Janca:** Right? And it also, the whole thing where it tells you you're absolutely right all the time, because I know it's telling stupid people they're right too, Chris. And every time I feel like, you know, that wasn't my best work, and then it's like, you're so smart, Tanya. I'm like, oh, shit. Who else are you saying that to? And if you're a dev and you're coding something and you're like, hey, how's this look? And it's like, you're great, this is the best. That's what it's gonna say even if it's not good code. And that's a problem. It's a problem that it's too agreeable.

**31:38 Chris Romeo:** We need to embed some 30 years of experience security people's responses into this. Like, that's really, that's a really dumb idea what you just described there. Wouldn't it be funny if it told you that? Like, oh wait, what?

**31:52 Tanya Janca:** I mean, I don't have it push back very often. Sometimes like it, you'll see like a little personality in the AI. Like I was working with Claude on something and I was like, let's do this. And then later we'll do these other steps. And then, and then we're working on this. And then it's like, and I was like, Claude, did you go and do that already? And it's like, hee hee, I couldn't wait, Tanya. I did it anyway.

**32:16 Robert Hurlbut:** Hahaha.

**32:17 Tanya Janca:** And I'm like, Claude, what are you doing? And like, I had told it, I know, I'm like, I had told it we were going to do that later, but I'm like, don't use up all my tokens yet. Come on. I only have so many before I have to press the continue button.

**32:30 Robert Hurlbut:** Well, that jumps, or that could help us jump into this next one, which is you talked about threat modeling and design ahead of time, but what about after the fact? And I think you've hinted on this, But secure code review, what does that look like with the code base that's largely AI-generated?

**32:51 Tanya Janca:** So, I have a lot of thoughts on this. So, I know I keep talking about my prompt library, but it's because that's how I'm doing everything now. And then I'm sharing it for free, so I'm not being a jerk where I'm like, here's this thing that costs money. So, 'cause I wanna help and 'cause I'm shameless, but basically, In it, I have a secure code reviewer prompt. So, the idea is, is like, I start off with prompts to help me make better code in the first place. And then after I run the secure code review prompt, which basically runs it against my secure coding guideline, it's not perfect. It's not like running a 3-week review or anything, but it's 80 things that I think are important. And I feel like you can't have 1,000 things, but I settled on 84, I think. which was hard. And so, I have it run that after. So, if we're going to do code review on something later that's existed for a while already, you could use something like that if you wanted. You could also just say, review it for code. So, I actually took a well-known SaaS that will remain unnamed. And I Basically, I had the AI, so I had Claude and I had ChatGPT. I'm like, make a login screen for me, you know, in this language, do this, do that, and have it do this just with no prompts. Then I had it do it with my level 1 prompt. And then I had ChatGPT and Claude review both and have the SAST review it for vulnerabilities, and Claude found the most. I found more than the professional stack analysis tool. And I was like, whoa, that's interesting. And it took about the same amount of time. And it's not like in my workflow in the same way. And that's okay. But I was interested to see that it was finding more faster. And now with Mythos. So Mythos is, for those living under rocks that might be farmers, is a new model from Anthropic that they are not releasing publicly because they feel it's too dangerous. It finds vulnerabilities and chains them together very quickly. But I was talking to someone this morning on Slack because I'm a nerd and that's how we talk to each other. We use Signal or we use Slack. We don't use phones. And she was saying how her company was using it and you have to make these harnesses in order to get it to work. And it was actually like a huge undertaking to attach Mythos. And it is not for the faint of heart. Like, it's a big undertaking to get it set up, to get it to work. But she's like, and then it found so many things so quickly, and almost all the results were exploitable. And it's just as terrifying as they said, but it's not like you just point it at, you know, yourcompany.com and then it just finds things. That's not how it works. So, That's slightly encouraging that it's not going to kill us all, but like only like a little. And I forget why I was talking about this, Robert. I feel like Robert's trying so hard to keep you and me on track, Chris. And like, he's doing such a good job. He's such a good co-host for you.

**36:09 Chris Romeo:** Most definitely. So, when you— this made me think of a slightly different direction because we started to talk about tooling, kind of SaaS tools coming into this AI And I know there's been some interest or, you know, there's been some talk and some preview releases of tools that are aimed at potentially providing AppSec functions directly from the model providers. So, like, I think Anthropic and ChatGPT and OpenAI both have preview offerings of tools that they're using their models to apply to what we would traditionally have used as a SaaS tool, kind of as a separate thing. So, what's your take? Do you think we're heading there? Do you think we're heading towards a convergence where Anthropic is gonna be our provider for everything that we do to build software, including our security tooling? Or do you see a different future?

**37:08 Tanya Janca:** I have strong opinions here. So, I I think that, so developers used to get to write code. Code was the funnest part of our job. It wasn't meetings, it wasn't documentation, it wasn't testing, it was writing code. That was my favorite part. I loved that. I would put on music and zone out, and I'd just code for hours at a time, and I'd be like, oh crap, lunch happened 30 minutes ago. I should eat food. I'd just be so into it. We have taken that away where the—

**37:39 Robert Hurlbut:** Yeah.

**37:39 Tanya Janca:** The AI's gonna do that part. Now, after we're running a SAST, or we're having, you know, Anthropic run some stuff, and we're like, hey, here's 5 million more bugs for code you didn't even have the pleasure of writing. So, you don't even recognize any of this crap. 'Cause like, when people would find bugs before, I'd be like, oh, okay, I'll go into that function, and I'll go take a look. Now, I'm like, what the hell's that? Right? So, we've made their jobs like pretty crappy. And I don't think this is the way. So, what's happening is people are pressing commit and publish anyway, because just like, so most companies don't have a SaaS, right? And most devs don't love using one, because it tells them how wrong they are. And I don't know about you, but I don't love being told I'm wrong. ChatGPT knows this about me. Like, where I see the future is, first of all, Anthropic at the machine learning level, making some corrections, same with ChatGPT, all of them, training them to write more secure code. I have thoughts on that. And I've approached one of them about some of the ideas and I'm just like, we need to do this a different way. And they're like, this is interesting, let's talk more. I also think that tooling, so I know shifting left is like a swear word now because the marketing people got ahold of it, But really, the system development lifecycle, because we are Anglophones and we write left to right, the further left we are on a piece of paper, the earlier we are in the cycle. And what I would like to see is stuff inside the AI where before we even get the code, it's had so many checks and we are getting a very high-quality piece of secure code. So, for instance, setting up a RAG server, which of course now I forget what RAG stands for, retrieval augmented, blah, blah, blah, blah, blah. But the idea is that you have like, this is how we do authentication here. This is how we do authorization here. These are the rules that you must follow. These are the libraries that you must use. These are the frameworks that are acceptable to us and the versions, et cetera. And so, then it is creating just way better code. And then you have a bunch of prompts that are set up to run every time, so no developer can turn them off. And so then they're receiving code that is just so, so, so much better by default than we start our SDLC, right? Like, that's what I would like to see. And so then every time we fix a bug, it goes through that reg server and it looks at it and it looks at this legacy code and it's like, oh, you're not using the right library. You're not using the ORM like we're supposed to. You're not— and it like slowly updates legacy systems. When I, way back in the day, when I ran an AppSec program full-time for the government, I remember we made this rule. I'm like, listen, when you open up a legacy app to fix a bug because a customer's complained or whatever, so you're opening it up, you're, this is a huge pain in the butt. You're opening up something we haven't worked on in a year or two, right? You're going to have to maybe downgrade your DLLs and do all sorts of stuff that it compiles or whatever, right? I want you to touch up your libraries if I can, but I'd like you to run Zap. And if there's any highs, I want you to fix one before you check it back in for me. And I'll give you a cookie. Literally, I had cookies. I bribe people with all sorts of carbs. And we came up with a thing. And so, if they did that and they told me, I would give them a high five. Some of the places I worked, I had cookies. Some of them, they told me I wasn't allowed doing that. It was really weird. They're like, you're not allowed bribing people. I'm like, is a cookie literally a bribe? Come on. Is someone going to— but it's true, people. I would get my tickets served faster because I had cookies. But anyway, the point is not the cookies. So, if I could just have these little incremental changes, and so, we got a lot of criticals in the backlog fixed that way because there was some other bug. And I'm like, if you're going to do this giant pain in the butt to retest and republish and do all of this stuff, just fix one for me. And a lot of them would fix 2 or 3. if it wasn't too much work, right? And so, we went from completely pathetic, embarrassing security posture to pretty good, like, kind of, not, like, maybe not respectable, maybe, but, like, not embarrassing anymore within, like, about a year.

**41:58 Chris Romeo:** Hmm.

**41:59 Tanya Janca:** So, if we could do that with code, like, with AI systems, every time we're fixing a bug in something legacy, it's like swapping out old vulnerable libraries. It's, you know, updating to the ORM that we're supposed to be using or whatever those things are, like we could do better.

**42:16 Robert Hurlbut:** Yeah.

**42:19 Chris Romeo:** So, if I think about this kind of secure vibe coding framework that you've described here, you know, we talked about the importance of that requirements well stated upfront, a secure design phase before we go in, the prompt library, has played into this a lot. And I've heard of that. I've heard of that as a concept as well.

**42:41 Tanya Janca:** Is it because other people have them?

**42:43 Chris Romeo:** Yeah. I mean, these things, the, the, the AIs just don't know what they don't know at this point. And we've got so many decades of secure coding experience that can be encapsulated in that. So, that seems like that's a crucial piece of this as well. And then the two, you know, some type of vulnerability detection, you mentioned code review, you know, having one, AI provider code review and other AI providers. I think that's another solid piece of ending up with a more secure version of whatever it is that you're, that you're going through this vibe coding process. So I think this is— it's been very helpful to kind of start to put together some of these blocks. It feels like we're Lego. These are Lego blocks that are all putting, you know, we're putting those together. Is there anything— is there, is there any— is there like one more Lego block that we haven't talked about yet? I wanna make sure I have the, the complete set.

**43:38 Tanya Janca:** I have more Lego block ideas in my head, but I'm gonna go build them and sell them.

**43:42 Robert Hurlbut:** Okay.

**43:43 Chris Romeo:** Well, we should, yeah, we should definitely let you do that instead of talking about them here. Having somebody vibe code it on a weekend and turn it into OpenClaw or something like that.

**43:52 Tanya Janca:** I think it's gonna take me more than a weekend, but I, I feel like we need to reimagine the way that we do AppSec if we're gonna do a good job and if we're gonna be able to do it quickly. I think we need to, reimagine all of the things so that we support developers in making the secure decision. We help them drive safely, if that makes sense. And just the same way, we're still innovating with roads. Like, in Canada, we have these new things along the middle where it's like this little indent. So, if you run it over across the median, it makes like a little clunk sound and you feel it. But then, at night, it has a little reflector. So, it where there's no lights, it still lights up along the center. Like, that's a new innovation in Canada and our roads.

**44:33 Chris Romeo:** Woo!

**44:34 Tanya Janca:** If we can still innovate on roads, like, we should definitely still be innovating and building software, like, every day.

**44:39 Chris Romeo:** And I love the concept you described there of reimagining AppSec. I think that we are a, I'll just call it, we are a tribe of people who don't reimagine very well. We like things done the way, you know, we've been doing it this way, where we've We're often, even on the edge of technology innovation, security folks still struggle from, well, why are we going to do something different from what we've done in the past that's always worked for us? And so, I think it's a unique challenge, and I love the fact that you're taking this on because you're somebody who can actually cause change across our industry dealing with that. But I think there's going to be a lot of people that you have to— that that you have to kind of get on board that just don't like change in general.

**45:25 Tanya Janca:** I agree with you, Chris. This is where, so like I give secure coding training, but you don't train every day, right? So, I have to find something else. So, I'm going to build stuff in my spare time, you know, when other people sleep. But I feel like every single time I give training, it's different now. Every single time I have to update and change over and over, like literally from 2 weeks ago, I'm teaching something new. Because things have changed, and that's very exciting. But also, like, how are people supposed to learn when they usually only get training every 2 to 3 years, right? Like, something needs to give here. Something needs to give.

**46:05 Chris Romeo:** Yeah, definitely. We'll be following along to see how you're leading that mission across our industry. Before we wrap up, I want to talk about your new podcast, just so folks are aware. What What can you tell us about it?

**46:19 Tanya Janca:** I have a brand new podcast. It is called DevSecStation, and it is 5 to 10-minute little tiny lessons for software developers about security. So, this season, which is 12 episodes, I'm doing the software supply chain and how developers are the target now. Because if you look at a lot of the major breaches from the past couple years, they'll call it a supply chain attack. But if you actually look closely and you read the whole report, because I'm the type of nerd that does that, Um, in several of them, the developer themselves was actually compromised, and then that broke open several different parts of the chain. And so, I'm exploring that throughout this season's podcast. So, if you have 5 minutes, um, you could look up DevSecStation.

**47:00 Chris Romeo:** And folks can find that on YouTube?

**47:03 Tanya Janca:** It's on YouTube and it's on all the podcast platforms. So, there's a video version and an audio-only version if you are more the listener type than the watcher type.

**47:12 Chris Romeo:** Cool. All right. Well, Tanya, once again, your 8th visit to the Application Security Podcast. We're looking forward to the 9th. That will be sometime in the next 12 months. But it's— this has been a blast. And, and you really— I wasn't kidding where I have not been paying that much attention to this. I've been paying some attention, but not my normal amount of vigor that I would put towards a subject. And so this has been very enlightening just to get your— the way you're your thinking through this has been helpful for me to understand the moving pieces. I have a lot better perspective on the moving pieces in this AI-infused SDLC. So, Tanya, once again, thanks for joining the Application Security Podcast.

**47:52 Tanya Janca:** Thank you so much for both of you to having me. Thank you.

---

Source: https://appsecpodcast.com/tanya-janca-secure-vibe-coding/