Josh Grossman -- Building a High-Value AppSec Scanning Program

0:01Chris RomeoJosh Grossman has over 15 years experience in IT risk and application security consulting, and he's also worked as a software developer. He currently works as CTO for Bounce Security, where he focuses on helping organizations build secure products by providing value-driven AppSec support and guidance. In his spare time, he's very involved with OWASP. He's on the OWASP Israel chapter board, is a co-leader of the OWASP Application Security Verification Standard, or ASVS, and he's also contributed to various other projects including the top 10 risks, top 10 proactive controls, and juice shop projects. Josh joins us to talk about high-value AppSec scanning programs. This is a new idea that he and the folks at Bounce Security are developing and are doing some training courses around. And so we, we explain some of the basics of the tools that we use in AppSec, and then Josh takes us through some ideas for What are some of the challenges that developers have with these tools, and what can you do to be more successful in building a high-value AppSec scanning program? We hope you enjoy this conversation with Josh Grossman.

1:10Josh GrossmanYou're about to listen to AppSec Podcast. When you're done with this, be sure to check out our other show, High Five.

1:18Chris RomeoHey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo, CEO of Security Journey. co-host of said podcast. Also joined today by my friend Robert Hurlbut. Hey, Robert.

1:35Robert HurlbutHey, Chris. Yeah, Robert Hurlbut, principal AppSec architect at Acquia, and really glad to be here.

1:41Chris RomeoYeah, it's a new title for you from— we used to be, you know, I always used to say threat modeler to the stars or whatever is what I called, right? You know, Robert's Hollywood people are like checking in with Robert for some threat modeling. You know, let's consider how this thing all comes together.

1:56Robert HurlbutBut yeah, focus on threat modeling. That hasn't gone away.

1:59Chris RomeoIt's hard to imagine you focusing on anything else based on, I think the whole time, I think I met you in a threat modeling talk. That is right. When you were delivering it at, in good old Detroit, Michigan.

2:09Robert HurlbutThat's right.

2:09Chris RomeoWell, so our guest today is Josh Grossman. Josh is, this is his second visit to the show. First time he was with us back in the early days of the AppSec Podcast, back in 2019. And I was looking in the Wayback Machine or trying to remember. So I had a chance to interview Josh at AppSec USA in San Jose. That's how long ago it was. talked about AppSec in Israel and the talk that he was doing at that conference. So it's great to have Josh back with us again. And Josh, what have you been up to since 2019?

2:40Josh GrossmanYeah, so thanks so much for that intro. Really great to see you again. So yeah, so last time we saw each other was AppSec USA. So that was a really great conference, really interesting experience, my first sort of major conference. And I guess what I was talking about at that conference was penetration testing, which is what I was doing at the time. I was doing a lot of application penetration testing, delivering and leading. So I've worked for that company and other companies since then in a more application penetration testing role. But what I guess what I was doing more and more over the last couple of years was more from the internal side, more working with development teams, looking at how to help developers build secure software from the beginning. I'd sort of been doing more and more of that. And then a couple of months ago, Avi Duglin, who I think we actually interviewed at the same time on that previous podcast, reached out to me and asked if I wanted to come work with him. And sounded interesting, so I had a chat with him. And it became clear that the direction his company, Bounce Security, goes in is very much what I was looking at there in terms of working with developers, working with development teams, trying to build security in early on rather than just sort of coming in as a breaker. So I made that move and now I'm very happily working with him at Bounce Security.

4:02Chris RomeoYeah, no, that's cool. Avi's a good friend of ours and co-conspirator in the Threat Modeling Manifesto, so we got to spend a lot of time. Now, I want to dig in a little bit to this transition you've made from kind of breaker to someone who's focused on helping builders because it's a bit of a soapbox for me. But I think as our audience knows, they're like, oh, get ready, here comes a soapbox moment. But as an industry, we put so much focus on red teaming, pen testing, breaking. And every time I speak to a university CS student who's thinking about security, I'm like, hey, what do you think you want to do? Oh, I want to break stuff. I want to, I want to, I want to hack the planet, you know, on my skateboard or whatever. And like, so for you, as somebody who's made that transition from being more of a breaker into a builder? Why make that transition? And then what are some of the things that you've learned about breaking and building through that little journey you've been on?

5:04Josh GrossmanSo I guess the why of it was about the impact. I really enjoyed security as a whole. It was always sort of a bit of a hobby and it became a job. And I think a lot of people in this industry feel the same way that it's sort of Part of this is fun, and you want to, you feel very connected to the industry. And what I saw was that the breaking side, it's fun, it's interesting, it's constantly moving. But I think we see the same thing over and over again. We see the same issues over and over again. And I think a lot of the root cause is that it's not easy to build secure software because developers have exactly the same thing from the other side. Things are moving very fast, issues are coming along, new problems coming along all the time. And it's not something that they are immediately equipped to deal with. I think there's a lot of value in taking that sort of passion and taking that excitement about security, but applying it the other way and saying, look, here's how you can make things more secure. And I think that's where the real impact is. The impact in terms of actually moving the needle of making things more secure and building software that is more secure is only going to come from working with developers. I mean, it's almost sort of an extreme example, but I was at a local conference here, Blue Hat, which is run by Microsoft, and it's very much like a hardcore sort hacking conference. And I was wandering around, I was like, yeah, this is cool. It was really cool, very sort of stylized conference. But I realized that the big people to be talking to are the developers. They're the ones who have to actually build stuff securely in the first place. We can break as much as we want, but I felt like the real connection needs to be with developers themselves.

6:36Chris RomeoYeah, I've summarized that in various conference talks over the years by saying, you can't hack yourself secure. Like everybody's got this big focus on let's break, you know, let's red team, let's pen test. And it's like, yeah, but at the end of the day, that's a, that's a security control or a check that happens after you've developed secure software. If you haven't put the effort in early in the lifecycle, sure, you're going to be able to break— you can break anything. But it comes down to the true place we can change and really have impact is earlier in the process. So kids, Stay in school, study AppSec, don't focus, don't believe the hype about the fact that everybody has to be a breaker. Listen to Josh's story here and join us in the AppSec universe here.

7:25Josh GrossmanNo, I think also, obviously, I think penetration testing is still important, has value, and a lot of that knowledge is important as well. But I guess it's important to see breaking isn't necessarily the goal. Breaking is a great way to learn, especially early on. But yeah, there comes to a point where you want to make real impact that needs to be So in the building side.

7:45Robert HurlbutSo Josh, today's topic, we're going to be focusing on something you brought to us that you've been looking at is high-value AppSec scanning program. And so what do you mean by that?

7:57Josh GrossmanSo I guess if once upon a time, back a few years ago, when you told a company, oh, you need to do an AppSec program, you need to have an application security program, they'd be like, oh yeah, we do application penetration testing. That's our application security program. And I think certainly we've moved on a little bit. And I think one of the directions we've moved in is now saying, well, we need more application security processes, we need to do more of this. And I think a lot of companies have just run to say, okay, let's get tools in, let's get a SaaS tool, let's get static code analysis, let's get software composition analysis, let's get dynamic analysis, let's get all sorts of tools in. Because I think there's a sort of expectation, oh, it's a tool, it's automated, we'll just sort of crank a handle and it'll will become secure. And I think what gets missed a lot of the time is that these tools have to be integrated into an overall plan of, okay, we're going to use this tool and this is how we're going to use this tool and this is what we're going to do with the results from the tool and this is how we're going to prioritize and interact between the different tools that we've got. And I think that's something that often falls between the cracks a little bit. And so the idea is you've got all these tools, but How can you actually build a program around those tools that you know what you're doing, you know exactly how you're going to react to the output of those tools as well? It's not just cranking a handle and something runs. It's actually, what am I going to do about that afterwards?

9:23Robert HurlbutYeah.

9:24Chris RomeoI mean, when I think about AppSec programs, people, process, tools, governance. And Robert, you'll remember our friend Alyssa Miller is the one who introduced us to adding the governance piece to that. And thanks, Alyssa. It's stuck with me since we had that interview a couple years ago. People, process, tools, governance. So you're talking about kind of the tools, obviously, and maybe a little bit of the process as well, I guess, plus the people, plus the governance. There's probably pieces of all of those in what you're thinking about here.

9:53Josh GrossmanYeah, I mean, that's— I think that's one of the interesting things that comes down. I know you added governance recently based on what Alyssa said, and I've seen quite often you've got tools in that, okay, we've got all these different tools that we've now implemented and we're expected to use, And you've got governance because someone somewhere is saying, you need to get these, you need to have zero findings from this tool, or you need to have zero criticals and highs in order to be able to release. But they're missing the process in between, and the people aren't really sure what they're supposed to be doing because that process isn't there. So I think that sort of nicely ties into the overall idea here.

10:26Chris RomeoAnd so before we dive deeper into this, I think it would do our audience well that we don't assume everybody knows all of these 4-letter tools and 3-letter tools. Why in AppSec do we have to have 4-letter tools? Why can't we Why don't we just have, you know, why do we have— we have SAST and DAST and IAST and RASP, and then we have SCA. We couldn't add one more letter onto that to SCAS? I don't know, I'm just trying to make up something new. But Josh, maybe you can just give us just a quick, maybe one or two sentence definition of each of those types of tools that you're thinking about that are part of this AppSec scanning program, just in case we have folks out there that are— maybe they don't know all of the tools, maybe they just know one, so we can educate them before we start talking about how we bring them all together.

11:06Josh GrossmanYeah, so I think that there are a lot of different types. There are lots of different types of scanning tools, especially some of the more modern ones as well that are looking at particular problems. I guess from my perspective, what I've seen most frequently are 4 key types, or maybe 3 plus penetration testing, let's say. I think penetration testing also has a place in this whole program, but it's obviously a little bit of a different thing. But if we look at the 3 other tools, you've got SAST, which is static application security testing, so that's usually a scanner that will go through the code itself or the compiled binaries and try and look for patterns or flows that indicate that there might be a vulnerability at the code level, that's looking at the code that you as the developer or you as the organization have written. There are SCA tools, software composition analysis, so they're looking at the libraries of the third-party code that you're bringing into your product, and generally they're looking to see, okay, What library are you using? What version is it? And do we know of any known issues with this library? And that might be vulnerability— known vulnerabilities in the library that have been reported and have a CVE identifier, or it may be that the library is licensed in a particular way that to comply with the license you'd have to open source your product as well. And the third key type of tool is DAST, or Dynamic Application Security Testing. And that's basically doing testing at runtime while the application is running, sending malicious payloads to the application, seeing how it reacts, and seeing if you can deduce from how the application reacts whilst it's running whether it's got a vulnerability in it or not. It's almost like people call it automated penetration testing, and I get very upset because I'm like, no, this isn't penetration testing. Penetration testing is one thing. This is DAST, and it's effectively a form of pattern matching or firing a pattern and seeing how it responds. But obviously the key thing there is it's happening at runtime, whereas something like SAST or SCA is more at— it's static. The application isn't running, you're just scanning the application code or binaries or libraries.

13:07Chris RomeoSo would you lump IAST then, interactive application security testing, into kind of that DAST bucket where you kind of had the 3 buckets?

13:18Josh GrossmanSo I'd put it as a sort of an advanced form of DAST. Because from what I've seen, in order to make IAST work, you need to have the application actually running, you need to have traffic going to the application, and the difference is that the IAST can detect when you get to certain parts of the application or certain vulnerable functionality and say, oh yeah, you actually hit that, and there's definitely a vulnerability here. So it's sort of DAST+ in some ways because it's— you're scanning dynamically, but you've got a bit more information about what's going on behind the scenes in the code itself.

13:54Chris RomeoAnd then as far as RASP, are you going to, because RASP is in production, are you going to leave that kind of off to the side a little bit from this conversation?

14:03Josh GrossmanYeah. So, yeah, RASP usually gets sort of lumped in with this a little bit, but it's obviously, it's slightly its own thing. It's sort of, yeah, it's more of the production time. It's more at the runtime, runtime detection, runtime response. So, I guess my main focus at the moment is looking at more the development time practices and what's happening while the application's actually being built.

14:25Chris RomeoOkay. All right. So now that we've laid the foundation of what each of these classes of tools are, where did this idea for this topic come from? I mean, was this where you just— did you wake up one morning? I just imagine people have ideas. They wake up and they're like, I had this idea. I got it.

14:43Robert HurlbutI got it.

14:44Chris RomeoWe got to put together a solid AppSec scanning program. Where'd this idea come from?

14:48Josh GrossmanYeah, I think, like I say, over the last couple of years, I've been working a lot more with development teams and I just saw sort of how difficult this was for them. A lot of these teams had these tools in place already, a lot of them already had the tools there, but they were really struggling with them. They weren't sure exactly how to use the tools, they often weren't familiar exactly what the difference between the tools was. They sometimes had very unrealistic expectations of the tools, what they thought they'd get out of the tools very different to what they were actually going to get out of the tool. And then once it came to actually handling the output, they just were completely unprepared for that as well. If you think about penetration tests, you might get sort of 20 findings, hopefully quite well explained and stepped through. You run an untuned static code analysis tool, SAST tool, you might get 1,000 findings. And suddenly someone's left with all these findings thinking, wow, what am I going to do with this? And they had the tools, they didn't have the process. This is even very large organizations where they may have quite regimented development processes, working agile, they've got sprints, they're using Jira, they've got tickets, they've got epics, whatever else, but then suddenly they hadn't really planned out, okay, well, how do the tools fit into this? Where do the security scanning reports fit into this? What sort of process are they? What sort of activity are they? They just didn't have that process in place. That's just a pain point that I saw very often. At the same time, these tools ultimately end up with developers. I don't think there are enough security people in the world to deal with these tools on their own. And most of the advanced vendors now are focusing their tools on, these sort of tools on developers. But when it comes to talking to developers about security, it's usually, well, it's either, okay, here's how you write secure code, or here's the OWASP Top 10, or maybe even here's how you put tools into your CI/CD pipeline, or the automation aspects. But again, I don't think there's much information out there about the actual process aspects themselves. Eventually someone is going to have to look at a report and decide what to do, and someone's going to have to figure out, okay, where does this enter into my process? And that's, I think those sort of processes are crucial for actually getting value from these tools and getting security benefit. And I think a lot of organizations are struggling with that. Yeah.

17:02Chris RomeoAnd one of the, so I can confirm what you're saying about developer knowledge of tools. So one of the things we do here at Security Journey is we have content on SAST, for example. There's no specific— I don't— we don't talk about a specific tool. What I've found is just introducing a developer to the idea of static application security testing and letting them know what is it, what's the, what's the value for you as a developer, what's the workflow look like with it. Not even though I'm not even talking about a specific tool, but just them having that understanding and that knowledge changes the game when they have to go use the tool. So many times as security people, we're like, hey, we just bought all these tools and developers, you have to run them. This is the way that you have to do it. And yes, there's 10,000 entries that came out of it. I don't care. You're going to run it with everything turned on. You're going to eat your vegetables and like it. You're going to deal with all these things as best you can or whatever. And we don't really think developer first. We think security people first. And I mean, we've I mean, I'll admit it, I've been guilty of this in the past. You know, this is how security used to think, and this new world has got to be— to your other point about the number of developers and number of security people out there, like, we don't have enough security people to run SaaS tools for all the stuff that's out there. We have to enable these developers to be successful and understand what they're doing and gain value from the tools, because we also know that developers, if you lose them on the tool, you're not getting them back. If they have a bad experience with the tool, it somehow messes with their, it generates 10,000 bugs in Jira or something for them automatically, they're never going to trust it again for them. So, that's just been kind of my experience in getting these tools in front of developers as well.

18:48Josh GrossmanYeah. No, I think that last point especially is key. I think that these tools can suddenly become security. Suddenly, the vast majority of a developer security day-to-day experience becomes, oh, I've got more results from this tool I need to deal with. Oh no, not again. I've had organizations come to me and say, oh, we really want to get our developers excited about security so they'll be more motivated to deal with the output of these tools. I was like, well, you're never going to get developers excited about security from these tools. We just need to make them, find them ways to get through these findings faster, and then hopefully they'll free up time to think about other things and think about other aspects of security as well. That's how you're going to get them interested. That's how you're going to get them more bought into it. Ultimately, these findings aren't going to get exciting, but these findings can be— this process can be made easier.

19:34Robert HurlbutSo Josh, we've talked about the importance of these tools, and none of us deny that. And certainly, I know developers, they can look at it and say, okay, I can see some value here. But in terms of this AppSec scanning program, can you describe how you're developing that idea, how you're turning it into a program, and helping out developers with these things they may be struggling with?

20:00Josh GrossmanSo, I guess this sort of thing had always been sort of— this issue had been on my mind. It was an issue I was sort of very acutely aware of. And then I started working at Bounce with Avi, and he's obviously done a lot of training courses related to threat modeling and to .NET security. He's been very sort of very successful, very well known for that. And he asked me sort of in passing, oh, by the way, have you got any ideas about training courses? Because that's one of the few things we do here. And I went home and thought about this a little bit. I went home and all these thoughts suddenly fell out about all this stuff that I've been thinking about, working with these organizations, seeing these struggles, it sort of fell out onto a piece of paper. And I was like, wow, there's sort of quite a lot going on here. There's quite a lot of ideas here. I think this is something that would be valuable to take to developers and to take to organizations. So I sat down, did some more work on it, did a lot of iterations with Avi and also with Adi Belinkov, who was— she's very experienced in application security also here locally in Israel, and she was working with us up to a month or so ago. And by the end, we had a rough outline. We reckoned with a few days training covering all these different tools and thinking about how how we can use them better, and how we can understand them better. I mean, we sort of thought maybe we should write this down in a very long document or a book or something, but actually, I think there needs to be an interactive element as well. We're not going to make this fun, but we need to try and make it in some way engaging. We need to find how to push developers through this process and find ways to actually help them work through it. So in the end, we sort of came down to, I feel, there's a few key areas about what we actually wanted to talk about and what we think we need to get across to developers. And so the first is, as you said, understanding the tools better, understanding this is the tool, this is how it works, and really digging a little bit deeper into what's really going on behind the scenes and the different sort of features and different sort of functionality, completely vendor agnostic, but there's a lot of shared common features between them. And I think it helps the developers to understand that concept, context, and understand that information. Also about configuring the tools. A lot of the more complicated tools have very different ways that they can be run, different modes they can be run in, and understanding that is also very important. Even within the same organization, different products they're developing may be very different profiles. One may be a very, very modern web application using all the latest frameworks, and one may be a very old monolithic Java application. And they may need to think about what's the best way of running the tool on this particular type of application. And again, that's knowledge that you might be able to get from a particular vendor, maybe, depending on how good your vendor is, but there's still a lot of commonality that it's important that developers actually understand it. I suppose the third key pillar as well was stepping back and saying, at the end of the day, we are finding application security vulnerabilities here. Developers don't have any innate training in, okay, here's how you assess vulnerabilities, here's how vulnerability ratings are calculated, here's the thought process that goes into deciding how bad is this vulnerability, how does it affect us? So another important part is trying to present them with that and try and help them, okay, here's the mindset to work through to actually evaluate one of these vulnerabilities, both from a generic perspective and also thinking about the specific types that come out of each tool. Again, one type of tool is looking at vulnerabilities in your own code. One type of tool is looking at vulnerabilities in third-party code that's publicly announced but may not actually affect you. So it's about how to approach that process and try and bring developers into that process. Like I say, obviously, the commonly held wisdom is you don't want to give developers 1,000 findings. You want to do all the triage yourself first. But some organizations, even that's not so realistic. Maybe there's some basic triage that can be done, but it will still fall to people whose primary day job and immersion isn't in security. So at the moment, we've got a 1-day version of the course scheduled for Virtual AppSec EU, which is happening in June. We also ran it past Jim Manico, and he really liked the idea, and he put it in his catalog as well. So that's really interesting. And then I guess the key thing for us is we obviously very much love OWASP and the open resources model and thinking about what we can do to release this sort of information more widely. And at the moment, it looks like the main way we can do that is to support the exercises that will be part of the course. We're trying to prepare worksheet templates to help people think through, okay, here's what I need to think about if I'm implementing one of these tools. Here's what I need to think about if I'm evaluating one of these tools. Here's what I need to think about if I'm evaluating vulnerabilities. And I think the goal is to try and maybe release those more widely to Again, try and sort of build up this thought process and make it easier to access and make this information more available as well. Because I think we need to find a way of communicating this across. I think that we can't build a one-size-fits-all, here's how everything works, here's how you'll do it in your organization, because every organization is different. But we can certainly help developers with the ideas and the thought process behind it.

25:14Chris RomeoYeah, the idea of an implementation guide, is something that really doesn't exist in the OWASP world. We've got cheat sheets which are issue-specific, but one of the things that I've done in the past in helping other companies build SDLs is to create implementation guides in the early days. It's really just a set of process steps and information, and you could do an implementation guide about a SaaS tool specifically to say, here's the things that you have to do, here's things you have to look out for. It's almost like a cheat sheet, but you can't use— cheat sheet has a specific—

25:48Josh GrossmanYeah.

25:50Chris Romeospecific definition in the world of OWASP, right? Like, this isn't a cheat sheet, but yeah, that's really neat to hear that you're thinking about that. And it sounds like a really well-needed topic that a lot of companies struggle with, that they just need that type of perspective. So, I want to change gears a little bit and understand what are some of the specific examples that you've seen of companies struggling with tools? Because I love to hear real-world case studies about, and I think it helps other practitioners as well to be, because there's probably some people that are going to hear what you're about to say and go, oh, it's not just me. Everybody has this problem too.

26:29Josh GrossmanYeah, and that's a great part of being a consultant. You get to see lots of different organizations, a lot of different environments, and you get to sort of bring out war stories that you can sort of anonymize, and I think it's useful to share. I mean, yeah, I've seen lots of different examples of this. A few key ones. So there was an organization I was involved with where the QA team wanted to start using DAST. Someone suggested to the QA team, pure QA, not a security-focused team, they should start using a DAST tool because they were already effectively performing dynamic application testing. They had a whole QA suite of tests, automated, manual, to test of running applications. So it was considered this would be a very neat insert into their processes. Unfortunately, the big challenge there was that they had very unrealistic expectations about what they'd get out of the DAST. Their entire life, they breathed bugs. They're like, okay, we find bugs, we fix bugs. We find bugs, we send them to be fixed. We find bugs, we send them to be fixed. And they were very much completely focused on, okay, are we finding bugs? Are we finding bugs? Are we finding bugs? I was trying to walk them through, okay, well, this is what DAST does, and it will find certain vulnerabilities if they exist, but it obviously has to look at how much of our application we're covering, and we have to look at what sort of tests we want, we have to look at what bugs are interesting to us. And it was very hard to sort of shift them away from this mindset of, does it find bugs, does it not find bugs? Often, you'll scan something with DAST, and it depends on the complexity of the application, they might not find it. DAST is very, very dependent on how well it manages to navigate the application, and how much coverage it can get. And in the end, they just sort of, I think they just sort of got tired of it. They said, we're not seeing the bugs, we're not seeing loads of value from this, so we don't want to use it, because I think they sort of had unrealistic expectations upfront. I think maybe if they'd had slightly more understanding of, this is what this tool does, this is the situation, and sort of been more of a joint work between the QA and security teams rather than being very much just put onto QA, then it would've potentially been more successful. Another organization I was working to help them with their backlog on their SAST tool, on their secure code scanning tool, and they'd had this process where they'd had a large, large list of code-level vulnerabilities they needed to fix, and they were tracking metrics every month. Okay, now we've got 600, and now we've got 500. They were gradually burning down this backlog, and I'm helping them with this. And one day, I go into the tool, and there are thousands of vulnerabilities. And I said to them, what's going on here? Where have all these vulnerabilities come from? There's much, much more than what you've got in the metrics. And they said, oh no, you're looking at the wrong view. You need to change the filter to that view. And I was like, why? They're like, because that's the view that we use. I was like, why is that the view that you use? Well, that's just the view that we've always used. I asked a few questions, and it became apparent that at some point in the lifecycle of this tool, it was decided they were going to focus on particular types of vulnerabilities, in particular, a set of vulnerabilities, and that they had a filter put in to show them those vulnerabilities, and that was it. They were working those vulnerabilities, and years had passed, and it's all become tribal knowledge that this is the view that we use, and Here be dragons, pretty much for everything else. Because they hadn't thought upfront, okay, what is our process? How are we going to work through this? Which stage do we want to take a more strict policy? None of that knowledge got passed down, that was never being reviewed, or is this still valid? Are we still looking at the right set of vulnerabilities? And they were just stuck on this original view that had been given years ago. Another good example is with software composition analysis. I've seen where organizations just have so many different libraries being reported, and they've got to go through all these libraries, even before they look at the vulnerabilities themselves, just say, are we using this library? Are we not using this library? This particular organization had the tool set to be quite sensitive. There were a lot of possible ways it could match a library, and often it would match on something and say, well, this file exists, so you must be using this library, and actually It was just a relatively common file that was in use with multiple libraries. And they were spending a long time just trying to get rid of false positives in libraries, never mind looking at evaluating the vulnerabilities. And then their concern, which is also valid, was they didn't want to start missing libraries. They wanted it to be sensitive because they didn't want to start having libraries they were using but weren't getting detected for some reason. And here, I think they need— I was trying to push them and say, look, you need to stop a second and decide, okay, how exactly are you going to tune this tool to get the right level of detail? Go through all these libraries, get rid of the ones that are false positives, and then move on from there. And if something new comes up, then hopefully you've got a lot less to go through, and you can say, okay, well, we're expecting this new library because we had this new feature that we needed something new for it, or we're not expecting this new library, so where's it come from? Is it false positive or not? And it was interesting, one of the things that almost came out of that was that they were spending a lot of time on this, and there was some discussion, is this the right tool? Should we use a different tool? And there was a lot of pushback about, well, if we change tools, suddenly we'll have to redo all this work all over again. We'll have to start from scratch. And I said, well, if you're spending this much time on it anyway, it may be that over a long term, another tool may actually make this less time. I've seen that where they moved to a different tool and did some evaluation, decided, or they did some evaluation based on that, they moved to a different tool and they established that actually it was going to take them less time going forward, but they'd sort of fallen into the sunk cost fallacy.

32:26Chris RomeoYeah, I was going to say the sunk cost analysis is where you have to ask yourself the question, if I was starting brand new today, would I buy that tool again? Don't forget all the investment I've made, all the knowledge and everything that I think I have. If I had to make the decision today, would I still buy it? If the answer is no, you've already made your decision. Move on, find a new tool.

32:46Josh GrossmanYeah, no, completely, completely. And yeah, when you're putting this much effort into it on a periodic basis, then it's time to put some effort into evaluating, is this really doing what I needed to do?

32:59Chris RomeoYeah, there's a lot of room for innovation still in the world of AppSec tools. I think we're really still in our infancy of really what the tools can do and, you know, where they can go. And so I think I look forward to the next few years as these tools continue to get better and evolve. And, you know, there's some, some new folks in a lot of these categories that are doing things faster and more from a more innovative approach. And so it's going to be fun to watch them kind of push the edge of the industry and drag some of the early companies that were part of this to either step up and match what they're doing or fall out of whatever magic quadrant they sit in.

33:40Josh GrossmanNo, completely. I mean, if you look at, say, features, I mean, one feature that we sort of, again, when I was talking through with Avi and Adi about sort of the content, and we talked about one of the features in software composition analysis of reachability analysis. Okay, you've got this vulnerability in this library. Based on the way you're using this library, would an end user, would an attacker actually be able to get to that functionality in the first place? And if you've got that, that can be a massive time saver because that's a lot of analysis that suddenly you don't need to do manually, and you can say automatically, well, there's this vulnerability in this library, but it doesn't affect us, so it's not going to be our first priority because right now no one can get to it. And I don't think every tool supports that. I was hesitant to talk about that because I was like, well, does every tool support that now? Is that useful? But we wanted to talk about some of the new ones and things as well because these are things that are going to be big, big time savers and big helps. Yeah.

34:35Chris RomeoI mean, push the envelope. I mean, I wouldn't buy an SCA tool today that didn't have reachability analysis. That would be one of my first questions in the demo. And if they said, well, no, we don't, but we're working on it. Okay, next. Bring me to the next demo because if I'm not vulnerable, Why would I want to extend my resources and have people wasting development cycles trying to fix something, a problem that I don't even have? I got enough problems that I actually have. I don't need to fix the problems that I don't have. So, Josh, I'll throw one out here. I don't know if you have any more on your list, but I'll throw one out just because you'll probably get a kick out of it, and maybe you can add it to your list of struggling with tools. But I'm not going to identify anyone who was involved in this. I'm just going to tell the story very generically, but I was listening to a story about a static analysis deployment that a large enterprise had done. And so, they were using one of the top industry kind of providers of static analysis, and they decided they wanted to deploy it in the cloud. So, they were going to put all of the services that were required to make this managed SaaS scanning service for their internal enterprise operating in the cloud. The, the bill for AWS was more than the license they paid for their enterprise to do it. It took that many resources to do it. So they literally had to double, more than double their budget to make this managed scanning service because they needed so many resources in the cloud to make, to even just have a trickle of performance in, you know, being able to process lines of code. And so I heard that, I just, it just made me chuckle and it makes me think of that same category of struggling with the tools. That was a struggle because you should really be able to deliver SAST without needing more cloud services than the cost of your entire enterprise license. But that was just my take.

36:31Josh GrossmanYeah, no, I've definitely seen that sort of thing as well. I think a lot of the time the sort of analysis that's being done can suddenly become very resource intensive and very memory intensive. I mean, one thing that I saw, which is sort of on the border of software composition analysis, is container scanning. a tool that's like, oh, well, we'll just ingest the whole container, we'll scan it. But it turns out the containers are quite large. And once you start ingesting a few containers into this tool, the tool gets quite unhappy quite fast. So yeah.

37:00Chris RomeoThat's another problem. When I think about, and that's a whole other topic, we didn't even talk about CVA or container vulnerability analysis as a category, but That speaks back to like, how do you build containers? You know, people still build these big, giant, fat containers like, and, you know, that's once again, you know, all 3 of us have been in security for a long time. We know limit our interfaces from the very beginning. The less interfaces we have, the less ways someone can compromise whatever it is that we're building. But the average container you see these days are these monolith gigantic monstrosities of, you know, why do we need the entire Ubuntu operating system? You know, why do we need X Windows in our container? I don't know. We just— it's always been how we've built it. We've always had X Windows in our container. Maybe we need a GUI in our container. Maybe we want to access our container remotely over X. I don't know. You know, that's a whole other— that's probably a whole other conversation we could— I'm about to go on another diatribe. I better circle it back around here.

37:58Josh GrossmanYeah.

38:03Robert HurlbutWell, Josh, it's been great to talk to you. So as we come to a close, just curious, what are some key things that you want people to know about about or think about this topic and maybe even a few key takeaways?

38:17Josh GrossmanYeah. So I mean, as we've seen, there's a lot we could talk about this. I could talk about this all day. I guess I'm planning on doing that at the OS conference. But I think there are a few sort of key ideas that came out of this and sort of key things that people can think about regardless. I think it's important to be aware of and to sort of have in mind. I think one of the first things that comes out is we think about, oh, we're going to buy this tool, it's going to cost us this much in license fee, and so we're going to spend this much. And you forget that there's a lot of time cost of actual people's time to actually work with this tool. And you can look at license fee, but that's just one part of it. And the amount of time you have to put into these tools is potentially higher than you'd expect. I mean, people might start screaming at me, but I think if you're using open source and where you haven't got a license fee, then potentially that's even higher because potentially you have to do— there's more work you have to do to get it put in. Even automated tools incur a lot of manual work, and you have to be ready for that and prepared for that and prepared to take that into account. And how you can cut down that work and make it more efficient is a key goal of the course. I think we covered this as well, but I think it's important to highlight that because of all this manual time, you can spend a lot of time on these tools, you can waste a lot of time, and it can become the the developers, the face of security for a developer becomes, oh no, not more findings from these tools. And if you don't have an efficient way of dealing with these findings, an efficient way of processing them, then that's just going to wreck developer morale, wreck developer attitude towards security, and make it a lot harder to, I guess, promote security in general within a development team. So this isn't just a— there's a lot of value in terms of making more efficient processes so that you've got efficient processes, but I think there's a lot of value in a wider security context of how do we want to operate our application security program in an organization, and part of that is going to be making sure there's not a massively negative view of security caused by frustration with these tools. I think the final thing to take into account as well is that these tools are very developer-focused, or they should be developer-focused, and we want developers to be using them. We have to realize that different people are going to have to solve different problems and/or be involved in different processes around these tools. So DevOps people are going to have to be involved in, how are we going to put this into the automation process? Developers are going to have to be doing the analysis of the findings from the code or library perspective because they're the ones who know it best. And you don't want security to end up getting gummed up on, okay, how do we run this tool? What do we do with the findings? When a lot of the tasks might be developers as well. There is a certain amount of, I guess, sharing the love that you want to do and making sure that the right people are engaged on the right tasks. topics. A lot of understanding how we're going to build processes for this is understanding who's going to do this. A lot of the worksheets are about saying, well, who's going to be responsible for this part? Who's going to be responsible for that part? And making you think, well, actually, different personas who should be doing that. And obviously, as part of that, it also gives you an opportunity to sort of say, well, we want developers to do some of this, and maybe we can push more of the vulnerability assessment and reviewing the vulnerabilities to developers, that pushes into sort of security champion territory where we want certain developers to be more familiar with security, which in itself is a good goal. But you do have to make sure that it doesn't all fall on security, doesn't all fall on developers, but the right people are doing the right processes.

41:49Chris RomeoVery cool. Very cool. Well, Josh, thanks for sharing this insight with us on high-value AppSec scanning programs. Say hi to Avi and the rest of the Bounce team for us. And I guess I'll leave our audience with our key takeaway, OWASP Europe this year, the Global is virtual, so you can sign up for Josh's class. Doesn't matter where you are on Earth, it's going to be virtual for everybody. So check that out. Josh, great to talk to you again. We look forward to seeing you in person at a conference sometime soon, maybe even Global AppSec US in San Fran in October, November, whenever that is happening. But once again, great to see you. Thanks for sharing your insight and knowledge with us.

42:29Josh GrossmanYeah, thanks. Really great to see you guys again. And, uh, yeah, thanks so much. Really great conversation.

42:34Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast and on the web at www.securityjourney.com/resources/podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, with application security, there are many paths but only one destination.