Jeff Williams -- Application Detection & Response (ADR)

0:00Chris RomeoJeff Williams is a veteran application security expert who founded and led OWASP, Aspect Security, and Contrast Security. Jeff also created several highly successful open source projects, including JBomb, JOT, OWASP Top 10, WebGoat, ESAPI, ASVS, and more. Jeff serves as an advisor to NIST, CISA, PCI Council, Oasis Seraph, OWASP CycloneDX, OWASP Foundation, Eclipse Foundation, and advises many companies and agencies on AppSec. Jeff has a BA from Virginia, an MA from George Mason, and a JD from Georgetown. He's also a 2-time Masters basketball national champion who would love to connect with you on LinkedIn. The focus of this interview is ADR, or application detection and response, but we go in a lot of different directions, including security assurance, basketball, life, and a healthy dose of AppSec. The Application Security Podcast is brought to you by Security Journey.

1:02Jeff WilliamsWe help enterprises reduce vulnerabilities through application security education for developers and everyone in the SDLC.

1:08Chris RomeoLearn more at securityjourney.com. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am the CEO of DaVinci and I am joined by my good friend, Robert Hurlbut. Hey, Robert.

1:33Robert HurlbutHey, Chris. Yeah, Robert Hurlbut and threat modeling lead, as well as principal application security architect at Acquia. And always a great time to be here and talk about application security.

1:45Chris RomeoI mean, what's to talk about? I thought we've already covered everything. No, of course we haven't because We have Jeff Williams with us who is making his 3rd appearance on the Application Security Podcast. Previous episodes, he gave us a history of OWASP, which I found fascinating because there was some things I didn't know about the early days of OWASP that Jeff taught us in that, uh, in that conversation. And then most recently, back in 2023, we talked about the tech of runtime security, which, uh, was also a good deep dive into how those things work. And one of the things I really appreciate about Jeff is he works for a company and started companies that provide technology in these spaces, but he's always open to talk about them generically and generally and help us understand how they work. And so that's, I mean, Jeff is a true educator in our space in that you can just, you can talk about this stuff and talk about concepts. And I always appreciate that about you. So. Um, first question, since we've already heard your security origin story before, what do you like to do that takes you away from computers and technology?

2:53Jeff WilliamsOh, great question. So, uh, I, I was a competitive athlete in college and, uh, so I've stayed, try to stay pretty active. And, uh, and if you meet me, you'll know why I play basketball. Uh, so, uh, about. 8 years ago or so, a buddy of mine saw that there's, uh, these masters basketball tournaments and we picked one in Florida and he was just like, let's just go and we'll be, uh, free agents. We'll get picked up by some team and we'll just go play and see how it goes. And it was super fun. Uh, we met a bunch of great guys and, uh, we did terrible. We got smacked. Uh, the guys that we joined up with were really not there to win. They were there to drank a lot. They were staying out till 3 in the morning. And by the, by the champion, by the, by the last game, like they couldn't really even run up and down the court. But, uh, it was super fun. Uh, one of them was a guy my size. Uh, they call him Clean. He's got the word Clean tattooed down his arm and he used to play with a Globetrotters-like team and just hilarious bunch of guys. But I started playing in these tournaments and I realized Like this gives a purpose to being in shape. Like, you know, I used to compete a lot and there was always a reason to stay in great shape, but I kind of lost that. Uh, you know, I'm in my, in my 50s, let's just say that for now. And, uh, so like having a reason has really helped. So I started, I started building a team and I started going to, uh, different tournaments, then, uh, put together a pretty good team and we won the national championship for 50 and over basketball. And then the next year we won it again. And, uh, so, you know, now it's been 8 years. I've played in tournaments in Vegas and Utah and Chicago and, uh, Detroit and, uh, where else? I don't know. Florida a bunch of times. Uh, Detroit was interesting. The tournament was held on 8 Mile. Um, but like, I just think it's, it's such a different world cuz, you know, as I was saying before the show, like, Uh, everybody in, in who, who I know in tech knows that I play basketball because I post about it on LinkedIn and stuff and they, they know, but nobody that I play basketball with has any idea what I do. Uh, and so they don't ask about it. They don't talk about cross-site scripting and stuff like that. So it's a really, it's a nice escape.

5:18Chris RomeoSo how does the competitive firework when you're in a, like in a master's basketball tournament? Is there still some people that want to mix it up and kind of, and get super fired up and, or is it pretty chill and that people are like, hey, we're all pretty old and if we do something stupid, someone's going to be in traction for 2 years.

5:43Jeff WilliamsOh no. It's like the, the guys that, that keep playing into their 50s and 60s. And by the way, these tournaments have like a 75 and over bracket.

5:52Robert HurlbutOh, wow.

5:52Jeff WilliamsAnd, but the people that are attracted to doing that are the ones who are hypercompetitive like me.

5:58Chris RomeoOkay.

5:59Jeff WilliamsAnd so they go and it is on, like I've, I've gotten in, uh, a bunch of scraps, like nothing where I got kicked out of any games or anything, but like, it's, it's super competitive. Okay.

6:12Chris RomeoYeah. Cool. Yeah. That's, uh, something that I've been, you know, having played hockey for a long time in my life, like I kind of reached a point where I just wanted to play. And so I don't play anymore right now because I haven't, there is not a league in North Carolina of people who are my age, also in the 50 and older bracket, but want to play competitively, but are, are will all realize, hey, we got to go to work tomorrow. So, because sometimes in hockey you have more opportunities to do stupid stuff than you even do in basketball.

6:42Jeff WilliamsRight.

6:43Chris RomeoBecause you've got a set of boards and around it and And you're used to smashing people into it as a part of the game. Uh, but I just haven't been able to find an environment, a group of people who are like, let's have fun. Let's, let's play hard, but let's leave all the stupid stuff that we would've done in our 20s, which would've resulted in a fight on the ice. Just, that's just not allowed. So that's what I'm searching for, but I think it's cool.

7:05Jeff WilliamsThere's this thing in, in Utah called the Huntsman World Games, and it's a whole Olympics for Masters level competitors. And so when I went there, there were like 33,000 people went to this town in Utah and they had a whole Olympics there. And it was, it was really fun. Really good time. Yep.

7:25Chris RomeoThat's cool. No, I like that. I think the big takeaway there is find a reason to stay in shape because when they give you a reason—

7:33Jeff WilliamsYeah. Now I, now I think about it, I'm like, I'm training for the next national championship. So like, I, it, it affects me.

7:41Chris RomeoYeah, that's really cool. Thanks for, thanks for sharing that story. That's, it's important for people to see that there is more to life than security and what we do at the, at conferences and stuff. Like there, there are things that are just so much more important.

7:55Jeff WilliamsYeah, definitely.

7:57Robert HurlbutSo Jeff, just to get started here, uh, there's a story about threat modeling and pen testing without production security observability. Uh, I wonder if you could Talk to us about that and that situation.

8:10Jeff WilliamsSo I spent many, many years doing threat modeling and pen testing on enterprise applications. Uh, and, you know, the first part of every engagement, I was, I ran a consulting company. So, you know, we, we tried to standardize our practices. So I've, I've focused on like the process of doing these things. How do we optimize it so that we can be more efficient, deliver more value for, you know, with less effort and really try to think about how to do these things efficiently. But every effort, you start out with a new application and you got no idea how it works, right? So they engage you and then you've got to learn like, okay, well, you know, what framework is it built on? How does it do authentication? How does it do access control? How do routes basically just work? Like what databases do it talk to? There's like all this information about how the application works that you need to understand before you can start doing a threat model, before you can start doing a pen test. And so at Contrast, we've been trying to figure out how do we automate that? And ADR, it's kind of a side effect of runtime security, IaaS, RASP, ADR, uh, is that we observe the application as it's running. And, you know, shame on us for not thinking of this earlier, but we realized like, hey, you know, we could also just generate a security blueprint that shows how this application works and where are all the security controls and where are all the dangerous functions that are used, where are all the backend connections, all that stuff that you need to know. We could, all the, all the endpoints, uh, you know, the attack surface, like you can automatically detect all that stuff. And if you take that blueprint and give it to the pen tester or the threat modeler, then they can start their work, the value add work. And I know you guys have done a lot of threat modeling. You know, a lot of the information that goes into a threat model comes from sources like interviews and old Visio diagrams that nobody updated and Uh, you know, questionnaires that people filled out. And so it's not like the most reliable source of data. And so then you get into the threat modeling exercise and you start digging around and asking more questions and you learn more stuff. And it's like very evolutionary, but I think we can radically accelerate that process by starting with a blueprint that was directly recorded from the running application.

10:45Chris RomeoSo it's, It's real.

10:47Jeff WilliamsIt's truth. And I'll give you an example. Um, imagine you're talking about expression language injection. Okay. Well, if you're pen testing, you got to go through every route and test every header, every cookie, every URL parameter, every form parameter, every piece of the XML payload, every JSON attribute. And see if you can find something that accepts, you know, dollar sign brace or pound brace or whatever, and thinks that it's an expression. Takes forever. But if you have the blueprint, you can filter it and you can say, hey, just show me the routes that actually evaluate expressions. And then when you, you can go to those routes, you can choose the parameter, you can test it, and you're done. So you can easily imagine shaving off 50, 75, 80% of the work of pen testing if you can focus your tests on where they might actually work. And it's the same thing for SQL injection and cross-site scripting and all the other categories. Um, the blueprint can also reveal some things that aren't typically tested, like access control and authentication. So you could, you could filter the list and say, hey, just show me the routes where there's no access control check or no authentication check or some bad combination of those. Uh, again, I don't know if they're right or not. It's not like vulnerability detection. It's not something you'd get from AST tools, but it's an architectural level analysis. And, and instead of having to test every single route. To see if it's doing the right access control checks, which is hard. I mean, I've spent days doing this. This is kind of my specialty when I was doing pen testing. It's like testing access control because nobody does it. But if I could just dump it out and say, here's the access control matrix, here's all the checks that are done on each route. I could say, oh, here's the 3 routes don't have an access control check, or the, the role that they're checking for is wrong. Like it's guest instead of admin. Uh, and you can just, you can look down the list and go like, oh, well, that's, that's what I expected. Awesome. You passed.

12:59Robert HurlbutYeah.

13:00Jeff WilliamsSo that's what, you know, security blueprints, security observability is all about to me is like providing that context so that you can do all the other things in AppSec much more cost effectively.

13:11Chris RomeoSo do you, has your thinking or your approach changed over the decades of time you've spent in AppSec, like to embracing more of a secure by design approach? Or would you say that's what we were doing in the late '90s and we just didn't know, we didn't call it that? Or like, what, how do you, how do you, how do you wrestle with that based on your multi-decades of experience?

13:39Jeff WilliamsOh, that's a great question. I mean, I think security by design is always what I've considered actual security. You know, I'm like an assurance guy. Uh, I would say that none of what anybody does in security matters at all, unless it's contributing to an assurance argument. So that's like my rule is like, does this help me communicate to somebody else that this thing is secure? And that means claims and evidence and like the whole assurance argument piece. And I actually think most of the things we do in security don't contribute very much because what is it? What does a vulnerability scan really tell tell you? If you get a list of vulnerabilities, how much does that really tell you about, how much does that contribute to your assurance argument? Like, is it, does it really prove anything? Uh, so like the world has tipped since the late '90s towards a very much risk management approach to security. It's like, hey, let's try and make sure we don't do any bad things. And if We try to test for a bunch of the bad things. We, you know, we find a few and fix them, then we're good. That is not assurance. That's not what I think of as secure by design. Secure by design to me is positive approach to that. It's like, let's say how we're going to build something with putting the right controls in place to counter the threats that we expect for that thing. And then let's provide some evidence that those things actually work, you know, that those controls are effective. And if you do that, that's the core of your assurance argument. It's just people talk about wanting secure by design and assurance and even threat modeling to a certain extent, but those are really practices that are in orbit around a different planet, planet assurance, where we're stuck in a very tight low Earth or low, low Planet orbit around planet risk and achieving escape velocity to get into orbit around planet assurance is, is not going to happen unless there's intervention in the market. You know, we've talked about this before that the market for security is broken. So, you know, unfortunately we're stuck in a risk planet and you got to play that game. Yeah.

16:05Chris RomeoSo I'm curious now, the comment you made about vulnerability scans and the fact that they don't provide much on the assurance scale today. Did, when we were doing Nessus scans back in the early 2000s, do you feel like it was, that was contributing to the assurance argument greater because we had, we knew less and we didn't have as much viewpoint of the world? Or like, what do you, how do you wrestle with that now historically?

16:32Jeff WilliamsNo, not really. I mean, like, so look, that's negative evidence. If you find something that says, hey, you've got a vulnerability, that's a piece of negative evidence and you can fix the vulnerability and counter it, but it doesn't tell you anything about whether you're doing it right. So like positive evidence works differently. Like if I said, uh, I checked every route that's possible in this application and verified that there's an access control check on each one. that has the right role associated with it. And it's not tamper-proof or non-bypassable. Like now I've got an assurance argument and evidence to support that argument that I actually, I can create a positive assertion about that app. And we just don't have that, like doing scanning for vulnerabilities never gives you that. All you can try to show is the absence of vulnerabilities, which is kind of impossible. There's, there's always could be another vulnerability in something.

17:28Chris RomeoThat your scanner picks up.

17:29Jeff WilliamsIt's only, it's only interesting to me if we're actually proving that the defenses are complete and effective. And that's not what scanning does today.

17:37Robert HurlbutOkay.

17:39Chris RomeoSo now to, to completely overturn the apple cart here, does AI make that assurance, that need for assurance that— I knew you were going to get it. I was going to get a chuckle out of you, but does that But for real, does that, is that, is there hope that we can use generative AI in some way to do more of those that replace, because like having a human being go through that access control example you said where you're checking every endpoint, it's very expensive and it seems like it's something maybe that AI could get to the point where it can go through, but I'm curious about like, how do you, how would you see AI impacting assurance?

18:17Jeff WilliamsSomeone I was reading online described AI as a school bus full of interns. And by that they meant it scales really well, does lots of work, uh, but you can't really use it for something that you have to trust. And so I think that's, that's great. And there's a lot of security work that can be done by interns. Like you can set up the problem in a way that you could make AI do it, but Uh, for the hard stuff, like, you know, actually analyzing code and coming to conclusions, like you can't really trust AI to do that. Uh, it's, it's not so like, I think for, and, and you gotta be careful about AppSec 'cause some things are really hard, like data flow analysis and things like that. Some things are pretty easy. Like if you wanted, to verify clickjacking, for instance, you could potentially get AI to do that. Like, hey, hit all the endpoints on this app. If every one of them has an X-Frame-Options header with the right setup, then you're defended against clickjacking. That's a positive argument. That's actually generating a should. But AI is not going to be able to verify some aspects of AppSec, like know, is this encryption implemented properly? You know, probably, probably not something you want to trust to the AI.

19:44Robert HurlbutYeah.

19:45Jeff WilliamsBut if you can simplify the problem, and the whole reason why I started the Asapi project was to simplify the problem. If you could say, hey, let's make a bunch of strong controls. And then the problem is just verify that you use those controls in all the right places and use them correctly. That's something AI can do, but it can't actually verify the controls themselves because they're, Cupcakes.

20:07Chris RomeoYeah, that's a good, yeah.

20:09Jeff WilliamsSo I don't know, I'm not, you know, I think AI is going to play a role, but actually it feels a little bit like blockchain to me right now. It's like a solution searching for problems that it can be effective at.

20:18Chris RomeoYeah. And it seems like the, the hype cycle seems to be on a downward, it seems to be on the downward trend now though, of people aren't so gung-ho that it's the answer to everything. I'm not seeing that as much as we did in the early days where this is, AI is going to solve that for us.

20:37Jeff WilliamsYeah. Yeah. We'll see. I mean, I, I do think that there are probably some use cases where it's going to end up being transformative, but it's not going to be for everything.

20:44Chris RomeoYeah. I've had the same, I've drawn the same conclusion as well. And for, for a lot of the same reasons. So, uh, we're supposed to be talking about ADR. Yeah. But we had, we had to talk about some other things too and have a little bit of fun there. Um, as I told you before we started recording, I can legitimately say I have looked at ADR almost with no time invested. So this is going to be fun just to, just to get a perspective on it because just, it hasn't come across my desk at this point. So, um, so let's, let's, let's define, because I'm sure there's other folks like me that haven't dove into it yet. Can you, uh, just lay a foundation for what is this ADR thing?

21:23Jeff WilliamsYeah. So it stands for Application Detection and Response. And so immediately you should think ADR is application security and API security in production. So not really anything to do with pre-prod. It's not about testing or finding vulnerabilities or libraries or SBOMs or any of that. All the stuff that we think of is absent. It's really on the other side. And the only thing we've had on the other side of that fence for a long time is Is, wow. ADR is an evolution of protection, application protection and API protection in production. And so if you look at the, the history of operations side defenses in all the other areas of security, what you'll see is moves toward, from perimeter protection to agent-based inside-out kinds of protection. So like we had network firewalls, now we've got MDR, we had, uh, OS hardening, now we've got EDR, uh, to, you know, work on the operating system itself. We got cloud protections has moved into CDR. And so like you kind of see this evolution towards agent-based solutions that run as part of that tech layer of technology. And, uh, ADR is simply filling that gap at the application layer. So I've talked with hundreds of companies about how they use their WAF. And let's just say it's not really effective. Like a lot of them have it disabled. A lot of them use a cloud WAF that has a default set of rules that they're not really tuning or tailoring or anything. And this is probably the most interesting thing is that the telemetry from the WAF, you know, they say they're running it in log mode. Well, where do the logs go? Well, do they feed it into their SIEM? Maybe. Do they use it at all? No, because the signal-to-noise ratio is so high. And it makes sense because you've, you've seen HTTP traffic, um, for modern applications. It's really complicated. And so WAFs can't tell whether something's an attack or not by look, just looking at the HTTP. But ADR works from inside the running application. This is runtime security technology. And as you watch an application actually use the data from an HTTP request from inside a JSON payload or inside an XML payload or, or HTML parameter, HTTP parameters and so on. You can see how that data is actually used by the application. And so you can be much more specific about whether something's a real attack and intervene in real time to block that attack. And so it's, it's just, you know, AppSec gets a lot easier if you're in the right place to observe what's going on. So I'll give you a SQL injection example.

24:30Chris RomeoSo.

24:32Jeff WilliamsA WAF sees, you know, a single tick and, uh, or, and a dash dash or something in a, in an HTTP request, and it has to stop that request. It has to block it, right? Breaks applications. If it doesn't block it, then it's probably under-blocking, like, and, and people can sneak through. So there's like, there's no way to fix that. But ADR, from inside the running application, it sees the data flow through the application. End up in the SQL query and modify the meaning of that query. And when it observes that, it's like, oh, well, that's SQL injection because that's the definition of SQL injection. Like untrusted data modified the meaning of the query. You know, it added an extra expression clause or whatever. So that's how ADR sees these vulnerabilities. So it intervenes only when it sees an actual attack, like something actually getting exploited. Then it says, hey, whoa, let's stop that and keep the application safe. So that's really the difference. And that's why there's another sort of DR in the XDR universe. Like this is just another source of events that feed into, into the SIEM or the XDR environment or their CNAPP or whatever. And that's, it just feeds into the, the operations analysts, uh, the security analysts' current tools. So it's not like a separate dashboard or anything. It's just new data.

26:00Chris RomeoHave they standardized the reporting that comes out of all these DRs in such a way that it's not like the old days where nothing really ever talked to anything else, even though that was always the single pane of glass, which nobody ever achieved. So there's a standard way. So it's not custom for everything.

26:18Jeff WilliamsYeah, generally there's a model called CIM, C-I-M, that has got all the sort of attributes that you'd want. And then, you know, uh, ADR has some, some things that it can report that other layers of detection can't, and that's supported. So you can just feed all that into Splunk or whatever and, uh, and view it that way.

26:39Robert HurlbutYou mentioned about runtime security and And just curious about, you know, ADR and relationship or differences, uh, to IAST and RASP. Yeah.

26:53Jeff WilliamsSo all of those are runtime security technologies. Uh, IAST works pre-production. It just sits in the background and detects vulnerabilities as you do your normal development job. We've talked about that. Uh, RASP was sort of the first iteration. of this in runtime and, uh, in, in production. And so really RASP was designed to sell to AppSec teams. And that's a little weird, right? Because you're trying to sell an operations technology to AppSec and then AppSec has to go to the operations teams and plead with them and say, hey, please put this on those production servers. And they're like, oh, hell no, we're not doing that. Uh, and it, it's just a little kind of a square peg, round hole thing. And so RASP was always designed for AppSec and really kind of development actually, which is a little weird. Um, so ADR is a, uh, uh, evolution of RASP technology. And there's a number of different vendors that are, are, uh, now in the ADR space. And here it's, it's the same kind of analysis happens at runtime. Uh, some of those products use, uh, kernel-level instrumentations, uh, to try to watch what's going on at the app level. Uh, others like Contrast, we actually monitor the application layer directly, but regardless of how it works, uh, the, The evolution here is that it's feeding data directly to operations teams. It's sold to operations teams. It uses the language of operations. So it's much more about events and incidents and runbooks than it is about vulnerabilities and tickets and traces. So it's, it's, you know, different buying centers, I think, is, is a lot of the evolution of this market. And then there's other innovations like the security blueprints that I talked about are part of ADR that wasn't really part of RAST. So it's a little more context because actually the operations folks don't have a lot of insight into applications and what they're doing and APIs and what they're doing. Uh, they're kind of blind to it. And so their first reaction, if there's some problem with the app workload, is go find the development team. But we're trying to give them much more information so that they can They can understand those attacks and deal with incidents, uh, themselves in a lot of cases.

29:30Chris RomeoSo you're describing a world where the, the, what I would think of as the pre-DevOps era, where we had development kind of on one side and operations. And so are you seeing that the reality of kind of the modern enterprise environment Is that there is the kind of the silos are creeping back or they never went away or what's, what's kind of your perception of that? Because I haven't heard people talk about operations as a separate function much. Like I said, I haven't looked at ADR, dove into this at all. So, but I haven't, I felt like we were kind of moving on a trajectory of we're all working together, but now I'm wondering, is, is that really not the reality for what's happening? in, in real companies?

30:19Jeff WilliamsWe've made progress towards DevOps. I'd say we've, in security, we've made more progress towards DevSec than we have to SecOps, at least at the application layer for sure. Um, I feel like DevOps is more of a reality at the infrastructure level where development teams are, you know, they're, they're building containers, they're, they're build pipeline, cranks out a container, it automatically goes into production and gets running and so on. But at the application layer, there's, there's still a pretty significant divide. And it's because application security technologies, you know, in production, there isn't really anything. There's just the WAF and it's, you know, it doesn't, they've kind of been marginalized. That's a 25-year-old technology, 23-year-old technology, something like that. And, you know, it's still, you know, just a lot of regexes at the network perimeter. So it's just, you know, there's, there's just not a lot there. And so I'm, I'm hopeful that ADR will, will help to bring that, that vision of DevSecOps even further down the road.

31:33Chris RomeoWhen we think about the now classic technologies in a company's AppSec stack, so the SASTs and the other one, 4-letter word, I don't say, um, but SAST and SCA and things like that that are, that are typically at the developer workstation level, they're in the pipeline. Does ADR replace some of the need that I have for that? Do I still need to have a full cycle approach? Like, how do these things work together or do they, do they work together?

32:09Jeff WilliamsIt's a fair question. I mean, you might ask the question like, hey, if I've got protection in production, why do I need to fix these vulnerabilities at all? Like, why am I even looking for them? And I think that's a little reckless. Uh, I think there's really good need for ADR in production because I talk with, again, I talk with hundreds of companies about their AppSec programs and Across the board, the size of the vulnerability backlogs for apps and APIs is staggering. Ponemon did a study recently where they found 1.1 million vulnerabilities is the average enterprise AppSec backlog. That's, that's a lot. And so it's not, that's not something that you're going to work off, you know, in the next year or two, like you're going to have vulnerabilities in production, even if your AppSec program in development was totally awesome, completely kicking ass. It's still going to take a long time just to work off that backlog. And then there's new vulnerabilities like zero days coming down the pike and new development and so on. So it's, it's tough. So I think, you know, you need ADR in production to deal with those problems. Like that's going to give you visibility and protection against most of the major AppSec vulnerability classes. You know, imagine if you could just protect yourself against SQL injection, file path traversal, XXE, unsafe deserialization, so on. Like, it at least gives you cover while you go back and fix those vulnerabilities. Um, but I, I also believe in, I, you know, I spent my whole career on, in development side of things. I believe in writing secure code and, uh, you know, having, uh, code hygiene around security. And so I, I don't think it's permission to stop doing those things. Um, I think you really need both. But I do think that the emphasis has been on sort of the, the left side of the equation.

34:11Chris RomeoRight.

34:11Jeff WilliamsAnd that, you know, and now the biggest problem is on the right side of the equation because there's a, there's a huge gap there that's not really doing anything. And You know, if the left side of the equation was, well, again, really kicking ass, maybe we wouldn't need as much on the right. But at this point, your best strategy is to, you got to do some of both. It's like if your house is on fire, like you, you probably want to engage in fire prevention techniques and buy a fire extinguisher. But at the same time, like your house is on fire, like you got to put out some of this fire right now.

34:46Chris RomeoYeah. Yeah. You're happy that there's a fire, a neighborhood fire department who rushes in and puts out the fire for you.

34:54Robert HurlbutRight.

34:57Jeff WilliamsYeah.

34:59Robert HurlbutAre there any, um, in terms of, you know, business problems, uh, that ADR may solve or, or even performance hits or anything like that, that businesses need to consider, uh, in, in incorporating or adding ADR?

35:15Jeff WilliamsYeah. So, uh, you know, there's a few problems that I think make ADR really urgent. And one is zero days, uh, you know, library and, and library security in general, like probably got a ton of libraries with, with known vulnerabilities in them deployed right now. And, uh, Even if you do, you know, analysis of those vulnerabilities to see which ones are actually exploitable, like you do runtime reachability or, or some exploitability testing or whatever, like you still got exposure there. And that's a place where ADR can really provide you some protection. You've probably got the same thing for your custom code, a backlog of vulnerabilities that you haven't been able to get to and haven't fixed. So this is like, to me, that's a a critical use case that you can deploy it against right away and reduce the urgency of that backlog. Maybe it'll allow you to prioritize it a little better and focus on the ones that matter. Because now you've got some air cover. You don't have to be like reacting immediately to every new vulnerability that gets discovered. Um, I do think that, uh, the observability use case is really powerful. where you can get a blueprint of every application. Those things are useful for so much beyond security, like just understanding what your workloads do, uh, and being able to see changes in those workloads over time. Oh wait, a new, we added a new route to that application, or that application just started connecting to our, uh, HR database. Uh, I didn't, I didn't know that. Like those are the observability alerts that ADR can, can, uh, reveal to you. And so it just gives you much more context to have an efficient AppSec program. Now you asked about performance and, uh, always a question. If you're going to install something on servers in production, it's going to be a performance question. And the good news here is, uh, ADR is really, really fast. It's, and, and I should, I should start that by saying like, look, If you want to have visibility into who's attacking you and stop those attacks, there's going to be some performance impact, right? Like it takes a little bit of effort to determine which the real requests are and which the attack requests are. You got to differentiate those. And so there's a little bit of work here, but the good news is that ADR is faster than WAF, probably significantly faster than WAF. It depends a little bit on the app and the environment and so on. But, uh, there's a good reason for it. A WAF has to do a whole extra hop, right? Like it's got to go all the way up the stack, analyze the full HTTP request and response, and send it all the way down the stack into the application. That's expensive. It's probably a few milliseconds on a round trip request. ADR runs inside the running application, so it doesn't have to reparse the application. There's no extra hop. The checks are done inside the functions where the vulnerability exists. So like for SQL injection, for example, the security checks are done inside the method that sends the SQL query to the database. It's like right in the exact right place to see the data the way the app sees it. There's none of this like, you know, decoding and, you know, is it bypassable with URL encoding or Base64 encoding or whatever? It sees the data the way the app sees it and analyzes it right there. So it's much more efficient from an analysis perspective.

38:55Chris RomeoJeff, where do you see ADR going into the future? So if we were to look in our crystal ball and look 5 years into the future, what do you think? What do you think is going to be different about it? Is there, is there, is the focus going to have changed in some way? Like, what are your thoughts on where this is going?

39:17Jeff WilliamsYeah, that's a fair question. Uh, so how long have you been in, in AppSec?

39:21Chris RomeoAbout 100 years or so.

39:23Jeff WilliamsYeah, right. Me too. Have you seen a lot of stuff change?

39:26Chris RomeoActually, no.

39:27Jeff WilliamsThat's what I'm saying. So like, I honestly, like, I think ADR is the right way to solve this problem and that it will become part of the, the standard platform for deploying application workloads will have ADR just like it has EDR and, you know, other DR kinds of technologies as part of the, you know, CDR as part of the normal stack. ADR will be there. It's just, you know, it's like the DR for that layer of the technology stack. And so it'll just, you said firmament before, I think that ADR will become part of that, uh, and that every workload will have that kind of detection response built into it.

40:09Robert Hurlbutit.

40:11Chris RomeoYeah, it's fair. And yes, we have been around a long time.

40:15Jeff WilliamsI use this analogy a lot. I, you know, like industrial factories instrument everything. They instrument all their machinery for vibration, sound, temperature, smoke, fire, like everything. And they instrument that stuff because that's how they can see a problem before it becomes a a massive business issue. Uh, in fact, I study this stuff because I'm kind of interested, but like they can detect problems just because like there's a new vibration in the factory. They're like, oh, there's a bearing out or something. Like they can detect these problems really early. And everything that matters in everything that's complicated, we instrument it, cars, airplanes, Space shuttle, we instrument stuff because that's the only way to see what's going on inside there and detect problems. Well, it's the same thing with software. Software is probably the most complicated thing that man has ever created. And we don't instrument it for security. You have no visibility in what's going on inside there and it's crazy. So like, I believe that that is coming to software is that it's, it, you're going to run with security instrumentation because it's the only way. To handle the complexity of modern systems, which is exploding. You have to, you can't just do it like we used to do it in the old days with a WAF and a, you know, a web proxy. You have to get inside the running application and see it as it's running.

41:44Chris RomeoAnd don't forget a person that knew how to create regexes.

41:47Jeff WilliamsThis, that's right. Uh, there's still a few of us around, I think, but, uh, I don't want to do it. Regexes make my brain hurt.

41:55Chris RomeoThey make everybody's brain hurt. So, uh, so Jeff, since you've already answered the light first set of lightning round questions, we had to create a new set just for this interview. But I'm stealing Robert's thunder. So Robert, please take us to the second edition of the lightning round.

42:12Jeff WilliamsThe second edition.

42:12Robert HurlbutAll right. Yeah. So, uh, first question is, um, shift left, legitimate concept or joke?

42:20Jeff WilliamsUh, that's a weird way to ask that question. Uh, it's, you know what, it's a legitimate concept that didn't really work. So in theory, it sounds great, right? We're going to shift testing early on in the process so that we can prevent these vulnerabilities from ever happening and save all the downstream costs of that. But what happened is the people that were thinking that didn't realize that when you shift left, like into a code repo and you ship like too far left, you lose all the context of the application. And so then you start generating more and more false positives, which eviscerates the benefit of shifting left in the first place. Because once you have, you know, all these findings, you have to verify them to to see whether they're false positive or not, and you burn up all your AppSec resources. And that's exactly what leads to these huge backlog. So I encourage organizations to look at their vulnerability discovery data, look at the rate of how fast you're discovering vulnerabilities, and look at the rate that you're fixing them. And if those things are diverging, guess what? You're going to build up a massive backlog and you got to, you got to change the curve to steal the COVID language. Like you got to bend the curve to fix that problem. You got to, you got to, I believe you've got to move the testing slightly to the right, not all the way to production, but move the, move the testing to where you can get more context and be more accurate. Like while the application is going through the pipeline, running automated tests, that's a great place because the whole application is assembled. It's all put together and you can actually test it. And the results that you get there are real because you observe them. And that's sort of the, the, the essence of the IAST argument is we're going to watch it as you run your normal QA tests and we'll discover security vulnerabilities quietly in the background without any extra work. So I've talked about shifting smart. Because actually there are some vulnerabilities that you can shift all the way left because they don't require much context, like hardcoded credentials and, you know, maybe clickjacking and stuff like that is like easy. So shift it. But other stuff that requires a lot of context, like anything with data flow, all the injections, uh, things that require backend connections and so on, you just see those in the code as easily. So. Shift Smart is my way of communicating that. Now, I do think that actually testing in production is interesting because, and it's, it's something that comes from the quality testing world, uh, because QA environments are never really accurate, right? They're, they have, uh, simulated systems, stubs, mocks, a bunch of stuff. They're not real. They're not connecting to your real production users or your real production data. And so it's not real. And so your test results are biased by that. So like actually doing some security testing in production is the only way to know that you've got some issues. And in fact, a lot of people discovered that their use of Log4j in development was not the same as their use of Log4j in production. And so they went and cleaned it all up in development. And then we came in and looked at their systems in production. We're like, hey, you've still got Log4j. And they're like, no way. Absolutely not. And of course they do. 'Cause we observe it. Like when we report something, it's 'cause we saw it running there. So, you know, I don't really think about shifting right or left as much anymore. I think you should shift to where it's most effective and that should drive it more than whether it's left or right. It doesn't really matter to me. It's like, where is it?

46:18Robert HurlbutYeah.

46:18Jeff Williamscheapest and most accurate to do that test, that's what you do.

46:23Robert HurlbutOkay. Second question is a conference talk, or is there a conference talk that you recommend folks find on YouTube?

46:32Jeff WilliamsOh, great question. Um, there's one AppSec talk that I think stands out from any that I've seen. Um, it's a talk by David Rice in the early 2010s, maybe. I don't know. It's, it's been a while, but this talk stands the test of time. He talks about the whole industry, uh, and the evolution of security. He, he goes through a fantastic analogy comparing AppSec to pollution and talks about, uh, you know, how the, in the, the environmental industry, the first set of regulations were around end of pipe, meaning like where sewage gets dumped into rivers. And then there was like the green movement. And then he talks about like the blue movement, which is like, if you think of green movement as like, uh, sort of bottom line, improve environment because it improves the bottom line. Like the, the, he talks about the blue movement in, uh, in pollution where the idea is, uh, top of line increase because you're doing a good job at environmental protection. And he makes the analogy to AppSec the whole way through and you realize like, holy shit, we're barely on our way to end of pipe regulation. There's, you know, like green movement would be like maybe this, uh, secure by design stuff. And like nobody's really thinking about sort of the blue level of this. So if you want to you want to get a roadmap for like where AppSec could go over the next 30 years. Like, I think he's, he's painted the roadmap there. That's a great one. If you just want a technical one, uh, look for Orange Tsai's talk on, uh, URL, uh, manipulate URL-based attacks and how to manipulate URLs. Uh, it's the guy's name is Orange, last name T-S-A-I. It's a brilliant talk. It's really, really good. I saw it live at DEF CON one year, a few years ago.

48:40Chris RomeoCool.

48:40Robert HurlbutAll right. And our last question is, who is somebody that our listeners should know about in AppSec that they probably haven't heard of before?

48:49Jeff WilliamsI'd like to nominate Naomi Buckwalter for that position. She is an expert in application security and she spends her time building community. She posts entry-level jobs and encourages companies to offer AppSec positions to entry-level folks and not to have an entry-level position that requires 10 years of experience and certifications and ability to use all the tools in AppSec. It's silly. And so she's really thinking big picture about the whole ecosystem. And, uh, she's a fantastic person. She's got a huge following on LinkedIn. So yeah, connect with Naomi and support her efforts because she's really doing the good work of, uh, of building the future.

49:44Chris RomeoVery cool. Thanks for sharing that pointer. Definitely look Naomi up. Um, what about a final takeaway, Jeff, on this whole conversation? We went a lot of different directions from basketball to assurance to ADR.

49:59Jeff WilliamsUh, I'll, I'll close with this. I think AppSec is in a really interesting time. Uh, matter of fact, I think it's the most interesting time in AppSec since I've been working in it. There's interesting stuff going on, like there's The, the government influence, like they're pushing SBOMs and pushing more transparency into AppSec. I think that's going to change things. Even OMB pushing attestations and potential liability around application security. I think that's super interesting. Uh, I also think that, uh, the complexity of applications and APIs is accelerating dramatically. Uh, there's new stuff going on, more ridiculous claims about AI's capabilities, uh, happening. So like, you know, the market is really interesting after being, you know, kind of stagnated with, uh, SaaS and that other one that you don't say and, and WAF for, you know, a number of years. We're finally seeing some real innovation and I'm excited about ADR. I think it's a really neat innovation. And, uh, so that's, if you see me out on the road talking at conferences, that's probably what I'll be talking about.

51:15Chris RomeoVery cool. Well, Jeff, thanks for, uh, coming back to the show to share your wisdom and insights. And, uh, we look forward to your 4th appearance sometime next year.

51:25Jeff WilliamsAwesome. Appreciate it. Thanks, guys.