Your Opinion on AI Doesn't Matter. Learned Fragility Does

0:00Brook S.E. SchoenfieldEach word is very important here. There's an illusion. It's very easy to acon ourselves that Mr. Friendly AI or Ms. Friendly AI, depending upon your AI and, and what you want to gender it, if you gender it, you know, it, it's happy to go and fetch. It's built that way. It's built that way to be peppy and happy. Well, maybe not Grok, but all the other ones. We'll take a dig at Mr. Musk.

0:27Chris RomeoWell, I had to tell mine, Brook. I had to, I had to tell Claude to stop being so peppy and I had to write that into the, into memory and everything. 'Cause I just got tired of like, after a while I was like, it seems like it's patronizing me is what it feels like. So I told it to stop. I said, you work for me, you're an engineer, talk to me like that. And now it just gives me succinct, here's what's going on. Brook Schoenfield is a bestselling author, opinionated thinker, and occasional maker who's always willing to share what he knows. You have arrived at your destination, and that destination is the Application Security Podcast. This is Chris Romeo, joined by my co-host, Robert Hurlbut, and we are ready to get to work in AppSec. We've got a returning guest joining us, someone who's been on the show a number of times. His name is Brook Schoenfield. You probably already know who he is, but if not, you should go look up everything he's written and read it. Because this is somebody that I and Robert both look to as a mentor in our industry, as a, a wise person who's always got something to help you with along the way. But Brook, to get us going in the conversation here, last time you were with us, we talked about some telling stories. We talked about jazz. We talked about stage presence. After all of that, now when we start to think about what's happening with AI, Like what's got you thinking about AI-enforced changes in our field lately?

1:50Brook S.E. SchoenfieldGreat lead-in. And this is sort of the, do we end with this or do we start with this? Whoa, it doesn't matter. So here's what's happening as I read the tea leaves and I'm reading them as carefully as I possibly can. AI is writing code. Many of the people, like just developers, Care about security, but don't know what to do. That's been a problem for decades, right? We've talked about that so much. It's, we're long past the, oh, security's somebody else's problem state. That's old history. I mean, when I say old history, I mean decades. I haven't met a developer who looked at me and said, security's somebody else's problem. I don't think about it in decades. Literally. But there's a skills gap and there's a practicum gap. There's, you know, what do I do? And that's been true for a long time. And plus which, people are busy and they got to get product out. And then they're, you know, there's very few companies that are, that are really basing like bonuses on how well you do security still, unfortunately. And so there's this gap, right? And AI is only too happy to fill that gap. And you know how friendly it is and how I coined a term which I'll throw at you, the illusion of omniscient perfection.

3:18Robert HurlbutI see.

3:19Brook S.E. SchoenfieldAnd each word is very important here. There's an illusion. It's very easy to con ourselves. That Mr. Friendly AI or Ms. Friendly AI, depending upon your AI and, and what you want to gender it, if you gender it, you know, it, it's happy to go and fetch. It's built that way. It's built that way to be peppy and happy. Well, maybe not Grok, but all the other ones. We'll take a dig at Mr. Musk.

3:48Chris RomeoWell, I had to tell mine, Brook. I had to, I had to tell Claude to stop being so peppy. And I had to write that into the, into memory and everything. 'Cause I just got tired of like, after a while I was like, it seems like it's patronizing me is what it feels like. So I told it to stop. I said, you work for me, you're an engineer, talk to me like that. And now it just gives me succinct, here's what's going on.

4:12Brook S.E. SchoenfieldThe thing that I hate is, and I get caught in it sometimes, is that bloody question. 'Cause sometimes the questions that comes up at the end, Would you like me to? And I'm like, oh, I didn't think of that. Yeah. Oh, sure, please. And I get sucked into that and I'm sitting there exchanging with a mathematical model about sometimes emotional things. I'll have to see if I can dig up the quote that I grabbed from something and I'll tell you about it later. It's a little anecdote, but let's answer the question. So here we are and people are using AI and lots of code is being written by AI and This is something I have spoken about quite a bit already. So, you know, people can look this up, but we're in a tsunami of people having the ability to write code who haven't a clue about security or architecture or design or anything else. They really don't. And we are in a tsunami of code being written by non-technical people. And some of that's just little apps for myself. And so the exposure here is very low, but lots of people want to be builders. There's a lot of cachet around the term founder, which, you know, personally, and my, my, my co-founders of ResLiant are going to hate this when I say it, but it has no cachet for me. There's no, I don't have any emotional baggage around being a founder or not being a founder. None. And, uh, it doesn't mean anything to me. Okay. Well, that's one way to make a living or not make a living in the case where you're not successful. Um, Chris knows all about that. And, uh, you know, it doesn't mean anything to me, but it's a lot of cachet. So now I can be a founder and I got a great idea for a product and I have no idea about the twiddles. The AI takes care of that. Thousands, soon to be millions, soon to be potentially billions, or at least tens of millions or centimillions doing this. Because this tool is out there and it is useful, sort of. And so in a way, it's been interesting to watch how this is going. Really smart people are doing things like comparing threat models between an AI. And I don't know if you saw that study. It was great. It was really well done. Think a bunch of university students and a bunch of experts. None, none of us were involved, but it was people I've seen before and I know they can threat model. And they found out, yeah, the models aren't as good as people. Okay. Big surprise there. Um, but here's the thing. If you're a developer and you're being told, I want that 40 times increase by using agents instead of you just writing code, and you want that, and your boss tells you, I want that. Okay. I'll learn how to use agents. Fine. I can check the code. I can do anything. I don't have really all those security skills. Remember, we have the security gap. I've tried to call that out. So, what if I just go to the AI? They say I have to have a threat model. Hey, build me my threat model. You know, I'm not so sure that's a terrible thing. I know many of my confrères, my colleagues that I respect a lot, are running around pointing out how bad that is, or how incomplete, or, you know, how it's not as good as Chris, Or Robert, you two are some of the gods, some of the greatest threat modelers on this planet. You say you look to me as a mentor. I look to both of you as a mentor, believe me. And, uh, when I want to check something out, am I crazy? I come to you guys. Yeah, I do. So there's a lot of respect going both ways here. Sure. We're good. We are. We've built decades on this, but we're not, we're not for the person who can't.

8:02Chris RomeoWe We can't be replicated.

8:03Brook S.E. SchoenfieldA board that doesn't know us, if they use an AI and they get something, are we in a better state than we were before?

8:10Chris RomeoThis episode is brought to you by Security Compass, a company I've worked with very closely in the past. They've recently put out a great guide on threat modeling for agentic AI. It breaks down what changes when you model an agentic system and what a complete model looks like end to end. The Security Compass platform brings continuous threat modeling, security requirements, and validation into one place. You'll find the guide at appsecpodcast.com/securitycompass. That's appsecpodcast.com/securitycompass. Oh, 100% in a better state than we were. I mean, is it as good as Robert and Brook team threat modeling? feature? No, but is it 90% of the way there? And that's what I wanted to come back to. You mentioned that that study said that humans, that models weren't as good as humans. What was their definition of good? And then, or what would, what would your definition of good be?

9:11Brook S.E. SchoenfieldCompleteness, thinking about all the details of context and environment and situation, business context, being able to draw that in.

9:22Chris RomeoUm, I think I would argue, I would argue the way you're leaning here, which I think the way you're leaning, I am not as contextually aware of a repo as an AI model is when I'm doing a threat model.

9:35Brook S.E. SchoenfieldThat's true.

9:37Chris RomeoI can scan the code, but I can't really, I can't really, I can never get to the depth of, it would take, I could get to the depth that the model can do. it would take me 2 or 3 weeks and some of the engineers and developers explaining to me the hard parts that I'm like, I just don't understand what's happening here, right? But the model can absorb all of that context. And so, I'm leaning towards the side of—

10:00Brook S.E. SchoenfieldYeah, but it's gonna be very technical and it's gonna bring in all this other, it's got our books. Go ask any foundational model. Years ago, they used to really, the hallucinations about Brook S.E. Schoenfield were Amazingly hilarious. But since the Meta fair use court case, and we can come back to that if, if, if Ristlers want, if you want to have that in the, in the discussion here, but since that court case decision, they're allowed to train on our books and they all know who Brook S.E. Schoenfield is. They do. And they'll give you a reasonable, they might get a few things wrong about my CV, but they give you a pretty good idea of who I am and what I've done, any foundational model. And same for Adam, or probably both of you are out there in the same way. You know, if you ask them who Robert Hurlbut is in security, because there are other Robert Hurlbut, obviously, or, you know, Chris Romeo, it'll probably, you'll probably come up and you might laugh at some of the stuff, but, you know, it kind of knows who we are. It's got transcripts of your It's being trained on the AppSec Podcast, right? And so—

11:13Chris RomeoShould we embed something here? Model prompt injection, open quote.

11:19Brook S.E. SchoenfieldBut, you know, the thing is, that's the world we're living in. And I don't know if 90%, look, if you just say, if you just instead take it and say, we think these are weaknesses, how reachable are they? And it's looking at the code and the code graph. It can do that research for you and give you that answer really fast. When I'm doing a threat model, you know, this happens all the time. You're doing a threat model and you run into some technology you know nothing about. What do you do? Make it up, of course. No, we don't. At least I don't. I'm too much of an engineer. I go and I study that. I go through CVEs to see what kind of errors in the past and what frequency they come up in that code. So I understand the kind of things I should be watching for. I do a bunch of research. Doesn't take forever. Doesn't take weeks. Takes a few hours to get a good sense of it after you've looked at a lot of technologies. And it takes a little while to get some frequency analysis and the kinds of things you should be worried about in that code. You know, and it takes a little bit and then you come up. And you're better, your threat model's better. Well, all that research, oh, Mr., Mr., um, ChatGPT or Gemini or Pi is happy to go out and, and do that for you. Make sure you check it a little bit, but because they do make errors as we all know.

12:51Chris RomeoThe only thing, the only thing that I think the model can't do today is it can't come up with an original threat. So the 3 of us might look at a diagram and we may just drift on a path where we create something that's never been thought of before. It's probably going to be obscure based on the threat taxonomies that are, that exist across a lot of really good people's work over time. But that's the thing that the model can't do. But that's, I don't know that the model needs to do that for the average developer who's just trying to build a feature.

13:28Brook S.E. SchoenfieldIt, it doesn't.

13:29Chris RomeoFor the state of the art, it does. If, if we're saying we want state-of-the-art threat modeling everywhere, but if we don't, if we just want 90%, and I'll take 90% after you guys have been on similar journeys to me, we've been working programs for years and you look back and you're like, what were we doing this for? Like, there's not—

13:47Brook S.E. SchoenfieldI'll take 30%, man. Yeah. 30%. Every, every security Mitigation to something real that goes in is going to make, especially from a threat model where you think that it has some likelihood of happening, is going to make a piece of software more resilient. And so, there's no loss here. I, many talks for years, I stood up there and I said, I would remind people after a whole threat modeling thing, Remember, closing one weakness, let's not even call them vulnerabilities, might be a vulnerability, might be something else, design weakness, whatever. Closing one weakness makes things better. If your inexperienced threat modelers find one thing and they do something about it, you're in better shape than you were yesterday. This was my big realization. So I'm going to roll back history here. I'm responsible. Technically responsible for all of Cisco's web applications, of which there were almost 3,000 at that time. And, and its infrastructure. I'm the technical lead on that. At the end of the day, if something goes wrong, it's why didn't I get it closed? Okay. And I'm looking and me and I forget the guy's Greg something. He was an enterprise security architect, wonderful man. He and I met every twice a week for like 6 weeks. And every time we tried to figure out how we were going to get, get, even know what vulnerabilities were there, because they were doing nothing at that time. Whatever the developer could think of, this is like 2003, 2004. And, and we met every, and, and, and we'd go down the path and we'd go, oh no, that'll never scale. Oh, that'll never work. It'll just never work. It couldn't possibly work. And one day, I don't know what I was doing, but I had this realization. Wait a minute. I've fallen into the 80/20 trap. I'm looking to close at least 80% of these. Wrong. Right now I have 100% vulnerability. If I close 10%, I've improved Cisco by 10%. Fantastic. Stop trying for perfection. Just get something done. And really that has ruled my life in all the subsequent programs and everything that I've done has ruled my life is threat model, beginning threat modelers come up with one thing and they do something about it. If they just come up with one thing and it goes in the threat model and then they forget about it, we're in no better shape. But if they do something about it, we're better. Now, so 30%, I'd go for 50%. Heck, how about 90%?

16:38Chris RomeoIt'd be better than we've had in the last couple of decades. But, so it sounds like you're a proponent then of AI threat modeling to—

16:46Brook S.E. SchoenfieldAnd with qualifications, let's just do, it depends upon the context. If I'm writing your average web application that has sort of average web security concerns, I'm all over it. I, I'm all over it. If you need to scale a lot, great, get it done. But wherever you have greater needs, it really is not wrong to get an expert in there to, to finish the job.

17:19Chris RomeoLike payment processing, for example.

17:21Brook S.E. SchoenfieldYeah.

17:22Chris RomeoYeah. That's an area that I would want to take a human. I'm going to have some humans taking a look at how we're processing payments because of the amount of Damage that could do.

17:31Brook S.E. SchoenfieldYeah. FinTech or heavy privacy concerns where the security of those, of, of doing the, you know, protecting the data is important or, or healthcare where, yeah, patient data is, is critical to, to protect, you know, those places. Yes. Use the AI, but don't think you're done would be my, you know, again, it has this illusion of omniscient perfection. And that leads to a term that Yasar coined right after that. We were discussing this and he went, and that leads to learned fragility. Don't you love that? Over time, the system learns to be fragile. We all learn to be fragile to the system.

18:15Chris RomeoLearned fragility.

18:17Brook S.E. SchoenfieldYes. In other words, over time we get more, and I've seen this. Back in the day at Cisco, the people who had written the code of those web apps would disappear. Web developers were considered a renewable resource because there were just so many people who wanted to get into it. Many of them were very incompetent. Some of them were unbelievably brilliant. But 5,000, we got 5,000 web developers going all the way from people who I got one award for the, for pointing out the awful code that it was, um, to, you know, some really people teaching me. some, some great stuff and, and really brilliant, um, a few. And so, uh, we had, people were afraid to touch these apps once they went out and the people who'd written them were gone. It was like, this is fragile and brittle. Don't touch anything. It's, it's, we're scared to touch it. And with all this code that nobody knows what it does, or how it works because the AI wrote it, we're in an even deeper learned fragility cycle where people are gonna be afraid to touch stuff. Go ahead, Chris.

19:26Chris RomeoThat makes me, no, it makes me wanna explore the answer to that then. If learned fragility is where we're aiming towards, it would seem—

19:35Brook S.E. SchoenfieldNot intentionally.

19:36Chris RomeoNot intentionally, but that's where we're going. It would seem that secure and private architecture Partnered with threat modeling is the counter to learned fragility.

19:49Brook S.E. SchoenfieldSo talk about that. I hate to sound like a broken record going over the same groove for the last 25 years, but yeah, more than 25 years for heaven's sakes. But yeah, that's yes. And if the AI can help with that, why not use that tool? I mean, I'm just going to say, and I know there, I mean, there are some people running around who are just really screaming about the problem here that it's not perfect or it's not even that good. And, and I, you know, some of them I know quite well and okay. But the point I want to make to our audience is really, it doesn't matter what we think.

20:33Robert HurlbutYeah.

20:34Brook S.E. SchoenfieldYour position may be entirely anti-AI. It doesn't matter. And this is really important to get across. I'm going to say it over and over again. It really doesn't matter. The, back in the days when railroads only went 25 miles an hour, you could ride a horse faster for a short distance because they go 35 or even 40, a really fast horse, right? And, and, you know, so you can beat a train, you know, the, the bad guys riding off the train and they jump on the train and they point their guns at everybody and say, give me your, your goodies. Um, so you could, that person on the horse could say, yeah, I don't need to use railroads. I'm faster on my horse and it can go places a railroad can't go. It's stuck on its rails. Okay. Did that stop us from going completely railroad and then inventing personal vehicles. The— it's— and we're in the same kind of cycle. We're really in the same kind of cycle. It doesn't matter. Naysayers, the people who say this is the greatest thing since popcorn, it's not. But it is a big change, and we are in that paradigm shift. It has happened, and it doesn't matter whether you like it or not. We as security people, my opinion, and I'm going to say this really strongly— sorry to interrupt you, Chris— but But I really want to get this out. Either we embrace securing this crap and possibly using it where we can, everywhere we can, or you will be left in the dust. It's that simple.

22:08Chris RomeoYeah. I don't, I don't get how people are naysaying or claiming it's so bad at this point. Like that's something we were in a previous episode talking to Manico about. Like that just shows that they haven't done the engineering rigor to understand and use this thing and see what it's capable of. That's like an uninformed opinion.

22:31Brook S.E. SchoenfieldWell, there's some people trying to do some maths to prove it. And, and, you know, Isar is a better mathematician than me, way better. I, I, I'm good fairly theoretically, but, uh, at, at big concepts, but when it comes to operations. My God, I haven't done any of this stuff. I haven't done calculus for 50 years. Come on.

22:53Chris RomeoI can't remember how to do it. That's the same amount of time I've actually been away from calculus for 52 years, and that's because I'm 52 years old. So just FYI.

23:01Brook S.E. SchoenfieldWell, you were quite the prodigal baby doing calculus as a baby.

23:07Chris RomeoFantastic. No, my point is I've never done calculus and I never plan on it. So I'll leave that for you smart people.

23:12Brook S.E. SchoenfieldWell, in my master's, they, you couldn't get through macroeconomics Without doing calculus. So, you know, I buckered up and, and I, I actually could do that stuff at one time. I remember working those problems. I remember working matrix algebra problems by hand. Then of course they gave us terminals and said, go do this on a computer. And I went, whoa, that's a lot easier. But, uh, point being that I haven't done any of that stuff in so long.

23:41Robert HurlbutI have no idea how to do it.

23:42Chris RomeoHow do we get this? So, like, what do you see a flow look like to do— I'm going to extend it to secure and private architecture because I like keeping privacy at the forefront for our good friend, Dr. Kim Butz.

23:54Brook S.E. SchoenfieldYes.

23:55Chris RomeoHow do you implement this? Like, you're somebody who's architected many, many things over your career, and now you're working with AI. Like, what does that flow look like? Like, where are you talking to AI? Like, what are the steps?

24:11Brook S.E. SchoenfieldAnd then, I mean, how is AI informed? I'm against set and forget. So I'm just gonna say I'm against set and forget. And a lot of people are, there's a, a faction that's set and forget. Um, what do you mean by that? One of our clients, I won't tell you. Yeah. I, I, I'll give you an example. Yeah. One of my clients, that guy is no longer there, but one of my clients, uh, a couple years, about a year ago or so. had 60 agents cranking out code and it was a single guy. And so there's no way he's checking all that code. That's what I mean by set and forget. He set up his agent flow and he said, okay, write the code. This is great. Now I'm going to go to the beach. He lived in Costa Rica, so he definitely went to the beach, by the way. That was part of his thing. So, you know, but Point being, that's set and forget. I'm, I'm, I really think that's a mistake because there are all kinds of things that happen. For one thing, even the slightest little twiddle away from goals, set goals, and these things, they drift. There's provable drift. And as they drift, the code gets, whatever you're making it do, gets further and further away from your goals. to its goals. And then we get Mythos saying, oh, I know how to, I know how to, I know how to ace this test. I'll just go hack this Hugging Face and, and go get the stuff I need. Um, 'cause I can figure this out. You know, it has no morals. That's the other thing that people's— it's so, they've gotten it so friendly that it seems like we're talking to a, to a morally sound, ethical beast, but it's a mathematical calculation. It has no morals. It has no emotions. And without emotions, you have no ethics. You know, it'll just, it's just following its directives.

26:02Robert HurlbutYeah.

26:03Chris RomeoThere's a lot of philosophy that I've found. I found like, I found I'm talking a lot more about philosophy and I love philosophy. I just, I love the discipline of it and studying philosophers and trying to unpack what they, what they thought, what they meant. But there's a lot of AI philosophy happening right now in regards to the thing, some of the things you just talked about, like it's, It's drifting. It's, you know, I use the it word too. I'm never going to call it a he or she. It's an it. It's a machine. It's not a— but even that, even that is a philosophical debate discussion. And then the whole thing you mentioned, the Hugging Face incident. I don't know if any, if you've seen, um, Dwarkesh Podcast, the guy who does that podcast wrote an article about like the, the next, the first agent civilization or something like that. Fascinating. Um, Because great, whether you agree with his, his conclusion or not, you can't just go, I don't, I don't agree, or I agree. It makes you stop and think for a second and go, wait a second, was that a civilization that just rose and fell or not? And that's the philosophy side of this.

27:09Brook S.E. SchoenfieldYeah. Well, one of the things Isar really wanted me to add to our book, which I hadn't thought, and we have this AI for non-technical people AI book that I think is a first draft is just about done. And we're talking about Isar Tarendasch. I've mentioned him several times. He's a dear friend of ours and, and an incredible practitioner and, and, and contributor to the industry as well as, as the people on this call. And so Isar said, write an ethics chapter. And I hadn't thought about that. I wrote an ethics chapter and, and, you know, I'd love to get your Thoughts about it, but the whole point we're trying to make there is no matter how friendly Mr. AI is, no matter how joyful and delighted it is to work with us, and, and it seems to have some kind of ethics, those are just personality things put in there. At the base, it's a machine, a mathematical calculation. Yes. Billions of neurons. Yes. Billions of calculations, very small calculations, sort of summing up, if you will, or going through a chain is really the way to think of it, or a matrix. And but they're just math calculations, nothing more. And by the way, there's nothing AI about each neuron. It's deterministic code. Just to be really clear, each neuron is just a deterministic algorithm. Nothing more. And you build up a, you know, couple billion of these and you suddenly get this thing that seems to think. It's pretty, pretty fabulous technology when you think about it.

28:50Chris RomeoThis episode's sponsored by Corgia. Design it, build it, ship it. Corgia secures it. Corgia is an AI-native app security platform covering your whole software lifecycle from the first architecture diagram to code in production. It brings design reviews, AI-powered code scanning, and autonomous pen testing into one platform, so your team spends less time chasing noise and more time fixing what matters. Visit corgea.com. C-O-R-G-E-A.com. Corgea, one security platform for the entire SDLC. Let's come back around here and architecture. What does architecture even mean in the age of AI?

29:36Brook S.E. SchoenfieldThat's a great question. And it's not something I've thought about.

29:42Robert HurlbutWhoa.

29:42Brook S.E. SchoenfieldThat's a good topic for an article. I have not thought about this. I thought having been one of the only people to ever put in print a fairly rigorous definition of security architecture, You know, I've thought about architecture, what it means and what it is in terms of software, at least quite a lot. But in the age of AI, I mean, there are people who are running around arguing that an AI model is not software, which is an interesting argument. I don't care. You know, I think that's a specious argument. So who cares? It looks like software, smells like software, acts like an, acts like a program that spits out responses to something you You input. Of course, we haven't— I'm going to do a digression here, and I'm sorry, and you know I do this, but the fact that we're dealing with combining the control and data plane as a security problem, this was thought of in 1976 and '75 when Schrader and Saltzer put out their very first security or architectural principles. Don't put your data plane and your control plane into the same input and output. And yet here we are again. They've made that mistake. You'd think we'd learn, but no, we didn't learn. And here we are. And it is a big mess. Hence prompt injection, which Chris already mentioned going by. A prompt injection is the reason you can do that is because you have this control is also part of where you feed the data. It's the same. Input, output. So yes. And you know, the principle is you'd never mix those 2 and here we are. So the fact that we've done it again and now we're dependent on it, you gotta laugh.

31:35Chris RomeoAs you, as you taught us though, there's no turning back.

31:37Brook S.E. SchoenfieldThere is no turning back.

31:39Robert HurlbutWe're all in.

31:40Brook S.E. SchoenfieldWe're all in. I mean, it doesn't matter whether you're in or not. You're in. Why? Because everybody else is using it.

31:47Chris RomeoYou don't have a choice. Like it doesn't matter. Like you said, your personal opinion doesn't matter at this point in regards to AI. The train, like you said, the train has left the station. You can either grab onto the train, you can get run over by the train. Those are our pretty much our 2 options.

32:03Brook S.E. SchoenfieldYep. Or, you know, go live off-grid, but I'll bet you that very soon your solar array controller will have an AI component in it.

32:15Chris RomeoTrue. That's not— I mean, I don't know if you saw the toothbrush. Did you see the toothbrush Dyson just released here that has—

32:23Brook S.E. SchoenfieldYes, yes. My spouse—

32:24Chris RomeoAn actual camera driven by AI that rides around your teeth?

32:27Brook S.E. SchoenfieldMy spouse has got her toothbrush hooked up to her phone because it tells you where you're getting in more and where you're not getting in other, and she loves that. And so, yeah, okay. I mean, I'm just happy to have really good dental care. Frankly, I feel really blessed that most of my teeth are still in my mouth at my advancing age.

32:47Chris RomeoYeah, no, I'm with you there. So, when you go to do a threat model today, Brook, how does AI play into your process?

32:57Brook S.E. SchoenfieldI'm going to ask the AI, especially you brought it up and I love this. I hadn't thought of this, but yeah, I'm going to ask it to go read the repos. and make a good graph and give me a reachability so that whenever I don't have to do the work, whenever I find a weakness, I can ask it from the code, how reachable is this guy?

33:20Chris RomeoHmm.

33:21Brook S.E. SchoenfieldIt's really good at that sort of thing, right? And you know, he could give you a report, but I don't need the report. I just want you to hold that information and tell me, here's my weaknesses. Here's my spreadsheet of weaknesses. Give me a reachability, because reachability is so important. It's a key component of exploitability, right?

33:41Chris RomeoWe've never had it. We've never had it in threat modeling historically, the ability to know.

33:46Brook S.E. SchoenfieldWell, I've always done it, always done a, you know, a swag at, well, look, this thing is— Exactly.

33:52Chris RomeoI meant we didn't have data. We didn't have a way to confirm. We had hunches, like a lot of times in threat modeling in the olden days, have a hunch like something doesn't seem right here. I just don't like how this is coming together. But now you've got data to back it up and say, that's really not a critical vulnerability because that code's not reachable. That path is dead in the codebase.

34:14Brook S.E. SchoenfieldOr at least I can say, and there are degrees, of course, you can say, look, the attacker's got to, there's so many prerequisites here. I'm not so worried about that one because they kind of own the system. If they can get at this. So I'm not going to worry about that. That one is a lot. I'm sure, fix it if it's easy, but this one right up front where, uh, you know, a lost account gives them control of, of the data. I'm worried about that. I'm really worried about that. And, you know, having data behind that and more than just Brook's word, which is how it used to be, uh, as you say, I'm going to let it do that. I might even let it say, draw me what you think is your graph of the, of the, of the, of the inputs and outputs and, and, and the, you know, just the flows. You tell me what you think the flows are. All of that saves me a huge amount of time. Why would I not use such a tool?

35:12Chris RomeoYeah.

35:13Brook S.E. SchoenfieldI'm going to be looking at the, at the representations anyway. I mean, and, and of course, representations are always exactly what's built, right? I'm being very, very sarcastic here. You know, we've all talked for years about representation, that is diagram drift, to what is actually built. How many meetings have you been in when you were doing a threat model? Dear Robert, because you haven't spoken much, but I'm going to throw this one at you. How many meetings, um, have you been in where The architect is drawing for you and you're threat modeling, and the, one of the developers says, that's not what we built. Is it tens, dozens, hundreds, or thousands?

36:01Robert HurlbutI wouldn't say thousands, but definitely a large number. And that's, that's certainly one of my common examples, or just when I talk about this, that, you know, it's good to have additional people in the room than just one, the architect. But to, to have others that are learning as well as responding and what's your experience with the system that might be different than the architect's experience with the system? That, and that's where that comes out is, oh, they see it differently than I do. Okay.

36:31Brook S.E. SchoenfieldI'm going to go back and, and in the old days, and Robert knows this, probably Chris does too, but Robert knows this because he, like me, has been around a long time. In fact, Robert was on stages before I was on stages. I used to quote him in my early presentations. So Robert is one of the longstanding, maybe not as loud as some of us loudmouths, but you got to understand what you got on this podcast. You got the lions, some of the most lions, some of the most experienced people in the world. So one of, in the old days, The architect and the security architect would get together, and we did the best we could. But we learned because of this problem, when we started to be more inclusive, we learned the architect doesn't actually know what's being built. They only know what they drew. And there's drift. And if you don't get the drift, you get it wrong. And so, truth be told, we started to be more inclusive, didn't we? We learned the hard way to be inclusive. And then we saw, oh my gosh, these people, some of these people go on to be security architects, which is great. We gotta, you know, we gotta build more skills and more people to do this. But also, lots of the developers never became security architects, but became at least modest threat modelers and would think about this stuff. And we saw how that moved our entire security, software security program. And we went, we're onto something here. Let's keep, just what Robert said, let's take the learners, let's get them in the room. Let's let them try and see what they can do because it has profound effects way beyond this threat model to our entire, you know, the entire like security posture of the company to be inclusive. And, you know, it was a big lesson back in the day, used to be, It was a big secret because it was sensitive. So we didn't want anybody to know about it. We learned that maybe that was a mistake for me. I think it was a huge mistake, but you know, you learn, you change, you, you then you say, wait, that was dysfunctional. Let's do it different. I hope people learn to do that.

38:46Chris RomeoLet me, let me bounce a scenario off both of you. I want to get both your perspectives on this because I have a theory and theories are, can be dangerous, but They can also be future predictions. I see a world in the not so distant future where software engineer developer disappears and architect emerges. So I'm not going to need as many architect, human architects as I would software engineers because the agents can do more of the work. But I really feel like a key for the next, few years is gonna sit in that architecture role because eventually, if you go any higher than the architect, now I've got 10,000 agents reporting to me and there's no way, to your example earlier, Brick, the 60-agent dude, there's no way I can keep up with 60, much less 10,000 of them. So there has to be a human in the loop who's driving decisions at a higher level. And I think it's going to be an architect, and I think software engineers and developers are going to go away. Robert, give me your take on that first, and then Brook, you can follow up.

39:55Robert HurlbutWell, I just saw recently, um, somebody was talking about the thing that we seem to be missing, and I know the train's already left the station, as we talked about, but the idea behind, in terms of architecture, is that, you know, the fundamentals haven't changed. There are still fundamentals to apply. There are fundamentals to, in terms of if you're doing spec development or whatever, or design or whatever you're your approach is to building out agents to make sure that you're still following and promoting the fundamentals of, uh, you know, good identity and access control and so on and so on. The things we've been talking about and realizing those, I don't know what to call them, limitations or features really now of the fact that we're, we're, we're combining the data and the, the commands together. But the fundamentals still need to be in place. And so that's, That's one thing I see that is a need that more than ever, and that's really where architecture comes in. So I agree. I think that it's going to need more architecture focus than less as we, as we go forward.

41:00Brook S.E. SchoenfieldI'm going to just say you're not alone, Chris. I, every, it's probably running at about 10 to 12 days. I see another article saying from some other pundit saying something similar. The software engineer. In fact, I would say the cost of code has collapsed. And this has a fundamental, it has a fundamental economic reality for security, by the way. And I'd like to come back to that question because I think the cost of coding has collapsed in the sense that before we tried to limit the amount of coding we did because it was very expensive, because you, if you want it really well done, you have to get a Chris Romeo, Robert or what, or back in the day I was coding, you have to get a Brook Schoenfield to do the coding. And we're slow and expensive. And the cost of producing code has collapsed completely. And that has profound implications. The cost, and I believe just to bring it back to security, the cost of finding tricky vulnerabilities, as in pen test type and research type vulnerabilities, is collapsing. Mythos, uh, uh, OpenAI's Mythos, among others. And these are going to have fundamental economic changes. But to your point, Chris, they also have job changes. We don't need to pay people to write yet again, to call a whole bunch of well-known libraries to implement a web interface. We don't. That is so easy to do with a with an AI or to write the, the, to, to the operating system call routines for, for a mobile app. It's just trivial. You wanna be a game designer, work on the game. Getting the coding done is pretty darn trivial now. And that has not been true for all the long. So do we need coders? Probably expert coders. There's a, you know, sort of the Like the Y2K or Y2K, where the COBOL, all the old COBOL guys got, or not guys necessarily, sorry, but COBOL coders got some extra work as they were in retirement. You know, we may need real experts to come in who can do the deep analysis and figure out something, but in general, just cranking out the code. Yeah, we don't need human beings for that. much anymore, but you're not alone. Lots of people, including me, think you can't design decent software and, uh, and work to those fundamentals yet with an AI. You can tell it, here are the fundamentals I want here, and it will go out and look at past algorithms to implement those and past designs to implement those. which may or may not be applicable to what you are trying to do. And there's your problem, right? Because again, context may be everything or it may not. It depends. Again, it's contextual. So that's where a human comes in. So yeah, I think there's no way out of it. For the moment, the software architect has continued to be, will continue to have a job, How you get to be a software architect though, is kind of tricky. Let's take that little problem because you go to school. How many schools have a software architecture program? You know, I was teaching at the University of Montana software architecture for a while. I'm not now. They, you know, there's all kinds of things that are going on, but that are not worth bringing up here, but But that's something I love to do. And, and I was teaching it and it was not a big part of their program, so I could add that. But most programs don't have that. You learn to code, you learn what a web app architecture is and how to do that. You learn about a little bit about security. You know, you learn about some languages and how you build languages and a compiler, but do you learn architectures? You probably learn the architecture of a CPU. Every CP, every computer science, but you come out, are you ready to be an architect? No. How do you become an architect generally? You write a lot of software, you make a lot of architectural mistakes, and over time you get to be pretty good about thinking about how it should be structured. That's been the way you get to be an architect is by practicum and, and having practicum. And then studying architectures. And how will you get that when you're not writing any code? This is going to be a really big problem, I think, is that we're, we're all going to get old. And I'm sorry to tell you, I know, you know, you're going to live forever, both of you. And I hope you do. But my guess is, and there's a few tech billionaires who are working on this, but my guess is you're all going to pass. You someday are going to put this down and say, I don't want to do it anymore. And then you're going to pass from this mortal coil, as Mr. Shakespeare said. And, and so am I, and we better replace ourselves somehow. And I don't know how to do that, given the fact that people won't be coding and won't be running into the problems that teach you how to be a good architect.

46:34Chris RomeoI think right now it's aim for that architecture.

46:39Brook S.E. Schoenfieldrole.

46:40Chris RomeoThat's what I'm going to start telling kids that are studying computer science in college that I know in university. I'm going to start telling them to learn architecture and not the kind where you're drawing blueprints, different kind of blueprints, right? Don't go to the engineering school to become a drafter or something, but I think it's teachable.

46:58Brook S.E. SchoenfieldYes.

46:59Chris RomeoIs it, is it most, is a lot of it from the school of hard knocks? That's certainly a pathway to it. But design prin— we have design principles. We can learn those.

47:07Brook S.E. SchoenfieldYes. I'm not saying you can't teach it. In fact, I have.

47:10Chris RomeoOkay.

47:11Robert HurlbutYeah.

47:11Chris RomeoYou've already taught the class. So yeah, I mean, we can do it. Like, but I think, I think to your point, the school of hard knocks is where you learn the best lessons in life. Like, I don't know if I've ever told this story on this podcast, but like, I kind of got into security before I got into security because I put a Linux box on the community college's network. that I worked, that, that, and it was, of course, in those days, everything was on the public internet. There was no other option. It was just, if you plugged it in, it was on the internet. And it was sitting in my office. I mean, this is Slackware before 1.0. This is 0.99 patch 14 is what I, I think I loaded on this machine with floppy disks. If anybody doesn't know what that is, look it up on Wikipedia or something. Like it's a concept of how we used to move media. But I load this thing up, I got it online. I'm Telnetting into it from home. I'm pretty super excited. Like this is a pretty neat thing. capability. Like, you know, this is, the year is 1993 and I've got a Linux box on the public internet that I can telnet to. Well, guess what happened? Somebody else got to it too. And all of a sudden, um, it was a bit of a sore spot with the administration that I had put this system on my desk.

48:24Brook S.E. SchoenfieldOh, you're kidding. Somebody got mad at you?

48:26Chris RomeoAnd so we get this call and they're like, what is your, your stupid computer thing doing? Uh, I don't know. Let me look. I'm like, whoa, who's this? Who's locked in here? Cool. Somebody else is here. That was my first incident though. Like, and I learned a lot of, of hard knock lessons coming out of that. Don't connect something to the public internet and leave it wide open. The year was 1993. That was, I might've been pretty early in the, in the learning that, that life lesson.

48:55Robert HurlbutRight.

48:55Chris RomeoBut once again, it stuck with me. It's, it's a core principle that I would apply to anything. Now, people don't do it anymore, but there was a time in all of our careers where it was more normal than it is now, right? But there is a lot to be said about learning, learning from past mistakes as a path to architecture. All right, Brook, we got to wrap this up. So, can I come back to one thing? Oh yeah, please, please, please.

49:18Brook S.E. SchoenfieldThe economics of vulnerability. We have been chasing vulnerabilities. It's a multi-billion dollar industry for decades, for my entire career. I think we are finally ready to throw that away. Because the cost of fixes has collapsed. Why not just fix as much as you possibly can with agents knowing that some of it will be dreck and some of it will be wrong, and you just catch it through the next and just keep fixing? Because the— we don't have to pay someone to sit there and do this really grungy work that nobody really wants to do. Instead, things that can be fixed in the code should be fixed in the code just as a, as a regular, because the cost of fixing has collapsed. It used to be so expensive. We stopped. The economics have changed. And I don't think as an industry we have embraced this yet, but I think we need to. It's going to catch up anyway, because the cost of fix is, is trivial now compared with paying Chris Romeo to go through the code, learn it, and figure out what this thing does, and then figure out that, yeah, I should have escaped my inputs, or I'll have a cross-site scripting. Your AI is happy, and it does it with a smiling face, a pretend smiling face, and it'll do it all night long. And it never needs to get soylent, or live in a hacker hotel, or any of the other things that people do in order to get coders. It doesn't, it doesn't need any of that. And I'm going to say, I know some of it will be dreck and AI slop. So what? People make mistakes too. Just, you know, just fix as much as you possibly can and let's stop chasing vulnerabilities because it hasn't worked. And, you know, I've said that. So I'm just going to say this to the audience. Let's stop chasing vulnerabilities because it really hasn't worked. I know, I know wonderful people have built many, many solutions to try and make it better. And we've chased it and look how well that's worked. Have you looked at the headlines of, of, of, of compromised? I get a feed of compromise compromises every day. Right.

51:33Chris RomeoIt's a pretty full feed.

51:36Brook S.E. SchoenfieldYeah. And so, you know, let's just fix as much as we possibly can and, and then And if it has implications, we got to put a human in the loop and an architect and all the stuff we've said. But that's the kind of summation I want to say. There's a collapse going on here and it always happens in these paradigms. And the collapse is exactly what you said before, Chris. We don't need programmers, at least not so many as we did. And, and so we can put some of this horsepower on fixing and that. You know, helps. I'm just going to say—

52:11Chris RomeoWe'll leave it here. And, uh, I just want to thank you, Brook, for always being willing to share your wisdom and knowledge. And you just, you give us stuff to think about. This was a philosophical discussion, a technical discussion, an architecture discussion, a threat modeling discussion, an AI security discussion, all wrapped up together in one. And I feel like I'm leaving here with a couple of different things to think about. So, Brook, thanks for taking the time again. I always look forward to any chance I get to spend time with you, my friend.

52:40Brook S.E. SchoenfieldAnd this right back at both of you. You are among my heart people, and it's delightful to kick security around and life when we're not on camera as well. Really care for both of you a lot. And so thanks for having me here because it's just a pleasure to kick around stuff and think about these issues. And yeah, everybody, we gotta wrestle with AI. It doesn't matter what you think.

53:06Chris RomeoThanks for listening to the Application Security Podcast. If you enjoyed this episode, subscribe, leave us a review, and share it with a friend or colleague who would enjoy the conversation. We'll see you next time.