Jim Routh -- Selling #AppSec Up The Chain

0:00Chris RomeoWelcome to season 3, episode 9 of the Application Security Podcast. On this episode, Robert and I interview Jim Ralph, and we talk about how best to sell application security up the chain. Jim is an experienced practitioner who has built 5 software security programs over the years, and he has a lot of insight that he can share with you and with us about how to successfully build software security programs, and also how to communicate and sell the value of those programs up the chain. We hope you enjoy. The Application Security Podcast. Here we go. Hey folks, welcome to season 3, episode 9 of the Application Security Podcast. On this episode, we deal with the issue of selling application security up the chain, or how best to communicate with executive and senior management on the whole issue of application security. And so we're joined today by a special guest, Jim Ralph, who is going to speak to us about this topic. And so, Jim, we always start the Application Security Podcast by asking the question, what is your security origin story?

1:41Jim RouthSure. Well, actually, I owe my career in security to my wife. There's a little bit of a backstory here in that I moved to Minneapolis, moved my whole family to Minneapolis after building our dream house in Massachusetts and My wife was willing to go along and so our 3 kids and my wife and her mother joined us out in Minneapolis. And 3 and a half years into that stint, which I thought was going particularly well, my wife notified me at dinner one night. She said, look, the kids and I are moving back east. Would you like to come? That pretty much was a definitive statement that forced me to change direction in terms of my career, I was actually interviewing for a CIO role staying in Minneapolis. And so I quickly called my sponsor who's the CIO of American Express. I said, Glenn, you got to get me out of here. My wife's had too much of the winter here. And of course, I think that winter we had 10 straight days where the high for the 10 days was -20. So that That didn't help much. So anyway, I ended up working in the business in a role that basically running a bunch of data scientists, about 500 folks doing marketing analytics. And then we merged that with the risk analytics team and I reported to the chief risk officer. And about 2 years later, American Express decided they needed a CISO, and so I ended up being the first CISO because of my knowledge of the business, the IT background, and I did a little bit of IT compliance when I was in IT. So much to my chagrin, I was the guy that started a CISO function at American Express. So I owe it to my wife, really. If it wasn't for her, I'm not sure I would have ever ended up in security. Yeah, that's, that's very cool.

3:54Chris RomeoUm, so I guess on this topic of selling application security up the chain, where I wanted to start is kind of from your perspective. I know you get a chance to talk to a lot of people across the industry, a lot of executives. And so from, from your perspective, what's the state of application security right now in big companies? Are you seeing application security as something that is Like everybody's doing it now? Is it the new thing or is it more established or kind of what are you seeing across various industries?

4:24Jim RouthWell, I'll tell you this, I don't call it AppSec anymore. I call it software security because software's so pervasive, comes at us from many different sources. Software, frankly, software currency alone is a major challenge today given all of the vulnerabilities that we have to address. So I think about it a little bit more holistically, and I'd say that the techniques and processes that we have for software security are far more robust and mature than we've ever had in the history of computing. So we have significant benefit. The one dimension that kind of divides really mature software security programs to those programs that are less mature is if we think of security as something that we add onto the software development process, we're missing a significant opportunity. And really, software security is an approach to improve software quality. And so, it really shouldn't be on the security professionals to review what the developers do, It should be on the developers to use the techniques that eliminate vulnerabilities, security vulnerabilities, and also fix security vulnerabilities earlier in the lifecycle. Those 2 things are economic drivers of benefit. The primary motivation for organizations really should be to improve software quality, and improve productivity in the development and enhancement process of software. Those 2 things, the CFO understands just as well as the CIO and the CEO. I think mature software security programs are embedding the security controls in the development process with the designers and developers and integrators when it comes to third-party software. And the economic benefits are significant, and obviously the net result is higher quality software and more resiliency of that software at the same time. So I see the the programs kind of divided into security's doing due diligence and has stage gates and is checking and validating and looking at static analysis results and identifying the false positives and kind of doing testing. And to me, that model, although does improve quality, doesn't improve it efficiently. And the more efficient model is actually giving tools to the developers in a DevSecOps model that drives defect density down. And the net result is much higher productivity and also more resilient software.

7:46Chris RomeoSo, from, I guess, on the quality topic, this is, from my perspective, has always been It's always been an interesting connection when we think about, you know, I've always been a firm believer that you can't really have quality without security. So, these 2 things definitely fit together very nicely. I guess one of the advantages then is, based on what you were saying, is that existing executives— so, are you saying that existing executives like CFO, CIO, other people in the C-suite, they understand quality already? Is this something that Maybe they don't understand security and all the ins and outs, but they do understand quality?

8:27Jim RouthMy advice to any CISO specific to a software security program is never say anything about risk at all. And instead, focus on a software security program to improve productivity for the money that we spend on software. Now, I've got 5 software security programs under my belt. I've never been turned down for money to implement a robust software security program. And the reason is I never say anything about risk. I purely go to the CFO, sometimes with, sometimes without the CIO, and I say, We're spending, let's say, $1 billion to develop and maintain software as an enterprise. What would you say if I could deliver a 15% productivity improvement for every dollar that we spend on software development? We're going to get a 15% improvement in productivity, and we're going to get that 15% productivity benefit year over year. Anybody, any CFO, any C-suite executive would say, yeah, that sounds pretty Pretty interesting, pretty attractive. What does it take? Then I answer by saying, well, if you give me 15% of what we spend as an investment to retool and change our practices, then you'll get a 15% return year over year in terms of productivity gain. Now, the reality is the productivity gain is going to be a little bit higher. Especially if you implement a DevSecOps model. So there's 2 levers that make that economic model work. The first is putting controls in place that eliminate the possibility of a defect, and so you can remove the cost of fixing the defect from the total cost of ownership of IT, or in this case, of software. So, that's one big driver. The second driver is if you can fix a defect during development versus post-development, it's about 10 times less expensive to do it early in the development cycle versus post-production. And of course, the traditional or conventional control for software security is pen testing, and pen testing is similar or akin to An auto manufacturing assembly line where cars are rolling off the assembly line and the cars have dents in the passenger front side door. And those dents require labor to bang out the dents, buff the car, and get it off to the dealership. And there's added costs to fixing the dents.

11:34Robert HurlbutRight.

11:35Jim RouthAnd that's what pen testing is. A better approach would be to adjust the settings of the robotics that are doing door assembly to avoid the dent in the first place or the defect. And then that makes the whole manufacturing process smoother. You don't have to pay extra money for people to bang out the dents and buff the car and prep for the dealer. You can just ship it to the dealer. So it's a much more efficient—

12:02Robert HurlbutYeah.

12:03Jim Routhmanufacturing process. And the same is true in software. If we can eliminate the defect and then also fix the defects earlier in the lifecycle, those are the 2 economic levers that drive productivity. Because it's essentially the developers that are spending their time to fix defects after it's pen tested and thrown back to the developers, then it's again 10 times more expensive to fix it at that point. So, giving them the tools and capabilities to prevent defects in a DevSecOps model, automating some of that, using container-based controls for review and signing of containers, basically puts the burden on creating higher-quality software on the developers, and that's where it belongs. And frankly, they welcome it. Giving a developer tools to identify security defects is like any of us writing Word documents and using a spell check function. It's automatic, it's efficient, you know, you don't have to be the world's best speller, but you can see where the misspellings are, you can correct those right as you're writing the document, it's a higher quality output, and you don't have to pay somebody else to do it, it's automatic. A lot of the tools for reviewing open-source code and for static analysis are essentially spellcheckers for developers. And so, giving them the tools, enabling them, that will actually help them learn and teach themselves how to write secure code just through the iteration of of giving them early and easy ways of scanning code during development. And all of that improves efficiency and improves the productivity of the developers. So, the before and after measures of productivity gain and defect density are the drivers for the productivity gain that ultimately the organization benefits from. And it's not a hard save, it's a soft save. You don't use fewer developers necessarily, but you get more output in terms of business functionality from the developers that you spend money on, and that's what's easy to measure. And a conservative estimate would say 15%. You can have as high as 30% gain in productivity. At Aetna, we have 1 defect per 10,000 lines of code across all the development that we do, but we have 2 DevSecOps teams that have achieved 0.006 defect density. We're now pushing DevSecOps across the entire enterprise, including offshore development, to get the defect density down to 0.006 across the entire enterprise.

15:12Chris RomeoYeah, that's an impressive metric. I just want to second a couple of things that you mentioned here. One of the things that I've spoken in public about before is the fact that developers aren't monsters. So, they're not creating defects on purpose. And a lot of times, it's because we don't actually teach them the right things to help them be able to use the tools and eliminate the defects. They want to fix the problems themselves. And then on the topic of shifting left, we had Kevin Green on, who's now at MITRE, just to kick off our season 3. 3 of this podcast. And he was talking about the whole idea of shifting left and the benefits that you're also describing here of being able to fix defects earlier and saving costs. And I love to hear you throw that 10x metric out there because I still go back to the IBM study from the '70s that NIST kind of brought back out about the Exponential cost of fixing defects as you get deeper into the lifecycle. And it's great to hear you as a practitioner kind of bring out that 10x number and kind of bring some reality to it, 'cause I know you've got a lot of experience from a lot of different programs. So it's good to hear that that is actually still true and there's still value in fixing things early and you've got a good metric to back it up. Yeah.

16:39Robert HurlbutBut Jim, that's interesting about risk in terms of selling application security programs to various executives, not mentioning it. But I'm curious, I mean, as a person who's been a longtime developer who now is in security the last number of years and now basically focused 100% on security, but still working with developers and certainly sold on making sure developers understand and implement security into their work cycles. You mentioned again not selling risk or not talking about risk, but where does risk come in? You know, again, as a security person, I talk about risk all the time. I have to in my line of work, but I'm very curious about that one. Could you touch on that a little bit?

17:24Jim RouthSure. Well, I'm a security person too. I mean, I'm a practitioner and I talk about risk all the time, like you, and that's the norm. But when I'm seeking funding or critical investment decisions or giving updates on major programs to senior business leaders and/or board of directors, the last thing I do is mention risk.

17:55Robert HurlbutYeah.

17:56Jim Rouththe word risk. And the reason for that is that risk in business terms is understood as probability and impact. And risk in cybersecurity is threats, vulnerabilities times assets. And probability of risk, which we have a lot of sophisticated techniques that we can use to measure is never comprehended by business leaders. And so when you get into probability of risk for cybersecurity programs like software security as an example, the lack of attribution of software defects kills the probability side of the risk equation. And there's no really good industry metrics that are well-established and benchmarked to help address that. And the more effort you put into the model, the probability model, the less likely senior leaders are going to understand it. And so it's a trap. If the better approach is to speak the language of the business, and that's economics. Every business leader understands costs and understands productivity and understands expense reduction. And this is the universal language of business leaders. And I'm talking really outside of IT. And so might as well speak in their language. Because they understand it. And so putting, in this case, a software security program in the context of economics, it's actually far more compelling than addressing the risk side of the equation because of the challenge of communicating the probability of risk. So my advice is avoid it with business leaders. Now, we still deal with risk, at a lower level. And so, as any security practitioner, I'm constantly dealing with risk and trying to quantify that in ways. But selling a program to senior business leaders based on probability of risk, there's a— I've done the FAIR model, I've done other models, I've hired consultants to build business cases and put a lot of time and effort into doing that. And Frankly, you know, you end up spending more time talking about the actual technique that was used for figuring out risk probability and less time talking about what's essential, which is, you know, these are the investment decisions that we have to make for the enterprise. So that's— I know it's a little unconventional.

21:02Chris RomeoYeah.

21:02Jim RouthBut that's what I believe, and it's worked so far.

21:06Robert HurlbutYeah, no, thank you. I appreciate it. That helps me understand a little bit better. I just want to make sure I clarified when do you use risk, when do you not. And so it makes a lot of sense in terms of working with business leaders and helping them understand or speak their language. I mean, that's a lot of what we do in security is not just do security, but also communicating with lots of different groups. Helping them to understand what we're doing and why we're doing it. And so, it's really a lot about why. Why are we doing it? It may vary a little bit depending on the person we're talking to.

21:37Chris RomeoSo, I want to dive into— you said— I want to ask, what is the process? You've done this a number of times before. You're getting somewhere in this 15% to 30% improvement rate. So, if one of our listeners is sitting out there and they're thinking, wow, that sounds Sounds like something I could communicate to our executive management and really sell this idea of not talking about risk but talking about how we're going to make these improvements in productivity. What's the process? What's the high-level process that you've gone through that's taken you to that end result of gaining that 15% productivity?

22:14Jim RouthWell, I'll tell you what I don't do, which is sometimes an obvious step that people take when beefing up a software security program. I don't hire a lot of security practitioners. And the reason I don't do that is the software security program really has to be embedded into software development, and the developers have to own it. And you mentioned this as well. And the way we do that is we create We call them mavens, but I've called them champions before. But the model is very simple. It's take a core group of the most influential developers that are really talented and immerse them in the techniques and the tools of software security in a DevSecOps model, and then have them both evangelize and teach other developers. And so, you know, we probably have, you know, somewhere around 3,000 to 3,500 developers, and we have today probably 600 or 700 mavens. Now, they go through a curriculum, of 8 classes and then a test and certification program, and they earn like a green belt. And then there's a yellow belt and black belt if they want to pursue that. And again, it's primarily education-oriented with some actual hands-on testing that they do. And they get certified and then recognized within the developer community for their expertise in They essentially lead and encourage and coach other developers on how to use the techniques and get the positive results. And so, even though we have a very mature software security program, we have a very small team. I think we have maybe 3 people that do pen testing. Now, a lot of times what we'll do is Anybody in the security function who wants to learn pen testing, we give them a shot at learning them. So we have a group of maybe 10 people that are learning. They're not pen testers. They're interested in learning it as a skill. And so they'll do some. We'll coach them through it. But in terms of our core pen test team, we've got like 3 people. And the whole software security group has maybe 4 or 5 people. So, it's relatively small for the size of the program that we have. But we've used BSIMM to measure maturity. We're in the top 3 maturity scores in the world for BSIMM. And we do all that with a very small staff. And it's because of this model of champions or mavens that get certified and then become the evangelists.

25:28Robert HurlbutYeah.

25:30Jim RouthYeah, that's—

25:31Chris RomeoI think that's great advice. And Robert and I talked about that early on this season from the perspective of security champions. And it's good to know that you've got concrete evidence about how that backs up the success of the software security program. So I want to switch gears a little bit here, and I want to ask you about So, from your perspective and your experience, how is software security being— is this a board-level issue? Is this something that's being discussed in boardrooms in general? Are there briefings and things that are happening about software security, or is it still kind of tucked under the general umbrella of cybersecurity and really not focused on?

26:18Jim RouthSo, the answer is that the topic of cybersecurity at the board level is highly concentrated today. In other words, I meet 4 times a year with— the audit committee has 6 board members on it and once a year with the entire board specifically on cybersecurity. So, their appetite and interest level in cybersecurity, and I've been actually presenting to board members and associations, teaching them how, you know, what the right questions to ask for cybersecurity governance. And software security is always a component of that. It's not necessarily a standalone, and I do talk about software security to our board, but most of the time it's sharing with them the economic benefits. The real challenge to software security, and this is historically, it's always been this way, is that there's limited attribution for bad software. In other words, if you put a website up riddled with high-risk vulnerabilities and defects in it, the threat actors will easily take advantage of that, plant malicious JavaScript, and infect people that are browsing the site. Now, when they get affected, the actual fraud doesn't take place for 2 or 3 months later, and the consumer has no way to trace it back to a bad website. So, there's really no attribution that ever comes back to the bad website. There are SQL injection type threats on a high transaction-level web app, and that is attributable back to the enterprise that's running the website. But most exploits, vulnerabilities in software, are not attributable back to the people that write the software or the organization that creates the app. That unfortunately makes it less attractive as a funding mechanism when you have scarce resource and you want to invest that resource in the highest risk because of the lack of attribution. Software security is just not in there and so it's not in board conversations as a matter of course. We can't change that. That's kind of the way it is, which is why I kind of celebrate the economic benefit. And so, when I do board-level education on cybersecurity, which I do once a year, I always cover software security as one of the bases that are foundational and essential. But it's not getting a huge play relative to other aspects of cybersecurity, which are getting a big play today. And I'd say That's continuing with all the significant breaches that happen. It's becoming— it's a hot topic and it's going to continue to be a hot topic and appropriately so.

29:40Chris RomeoSo from the— and that makes sense to me. I mean, I guess it could be a good thing that software security doesn't get a lot of attention. It could be a bad thing in some organizations. Uh, but, but, you know, definitely the fact cybersecurity as an umbrella is definitely getting a lot of, a lot of attention, uh, at least from what I understand and what's happening there. So, um, what— so as somebody who's built a bunch of these programs and is an executive in this space, I want to kind of flip my next question around and, and ask you to talk to the person who is Somebody who is, uh, you know, maybe part of a software security— maybe they're the only person in an organization that's focused on software security, uh, maybe they're a developer, maybe there's somebody who's outside of the security organization— what are some tips that you would offer them in how they can kind of reach up to the executive levels and, and communicate more effectively about, you know, it could be about software security, could be about security in general, but as somebody who's kind of— who's in that executive role, what are some of the things you've seen people do to communicate with you that have been effective and have helped you to kind of want to help them with the problems they're dealing with?

30:59Jim RouthYeah, so I guess what I would say is that there's— there are always trends in security. And what I mean by that, there are patterns. And some things trending positively in terms of higher-level resiliency in the enterprise, and some things trend downward in terms of higher risk to the enterprise. But things, you know, events are trending like that. And so one way of being more effective as a steward of software security and, you know, good IT hygiene, which software security is part of, is to identify the trends. And so for an example, a trend today is vulnerability management, specifically around software currency, is becoming much more challenging. And it's, you know, today with dealing with Meltdown and Spectre and the impact of those 2 things on an enterprise, it's, you know, it's off the charts. You know, Spectre and Meltdown just as— that don't really have— well, I guess they could be considered part of software security. They never have been.

32:17Robert HurlbutRight.

32:17Jim RouthConventionally or traditionally, and appropriately so, but in fact, you know, with Meltdown, it's actually some instructions that are baked into the chip by the manufacturers that are being exploited. And so, to a degree, it's not typically what you think of. It's not software in the operating system. It's not software in the application stack. It's not middleware. It's not in the browser. It's in the firmware, which we haven't traditionally thought of as part of software security, but it has a— I think, this is my opinion, that it's the most complex set of requirements for patch management in the history of computing is what both Meltdown and Spectre represent. And so, I'd use that as a platform for IT hygiene and the importance of configuration management at all levels of the stack and software quality, again, at all levels of the stack. And I'd use that to garner more resource and more focus on just the challenges today of dealing with a very complex set of compute requirements for an enterprise. And of course, the second biggest trend that has an impact is the move to cloud. And so, I'd also point out what things require investment in terms of resiliency for cloud computing versus on-prem computing, what are the implications for software security in both because they're different, and using some industry data on trends, I'd use that to arm myself with the wherewithal to get additional resources focused on the right things for the enterprise based on these 2 dominant trends that are fundamentally changing both our compute architecture and our security capabilities and the currency of our software. Now, the other dimension here is enterprises are going to be accelerating the refresh cycle for devices once firmware, better firmware is available that addresses the Spectre vulnerability. And there's going to be a buying spree of technology. The manufacturers are going to love it. And this is over the next, you know, 2 years at least. It's not— so what I would do is I'd jump on that trend and look for opportunities to make improvements in our capabilities as an enterprise. Software currency is certainly a core component of that. And so, that's what I would do. I hope that answers your question.

35:23Chris RomeoYeah. And I heard Brad Arkin, who's the CSO of Adobe, say this once, and he summarized it as, never waste a good crisis, was how he—

35:33Jim RouthHe's exactly right.

35:35Chris RomeoHe built Adobe's software security program as a result of a gigantic Flash security thing that we were dealing with a number of years ago. Yeah, I think that's—

35:47Jim RouthHe's also an architect of using that champion model of teaching developers security by teaching it to developers and having them teach other developers. So, he's a big proponent of that and he's had a lot of success with that.

36:02Chris RomeoYeah. And that's, you know, I mean, I think that's really the only model that works here. I mean, I've only seen one big company that has a software security team that in triple digits, basically above 100 people focused on it.

36:17Robert HurlbutRight.

36:17Chris RomeoAnd I don't know that they're— I don't know that they can really justify the return on investment that they're getting. So, I think the champion model, I mean, I did it at Cisco with, you know, I ended up with about 500 people at Cisco that were part of the champion model. And so, you know, I mean, I think that's the only— it's really the only way to scale this thing when you're talking about big companies that have tens of thousands of developers, it's just not, you know, it's not feasible. You're not going to build a team of 500 software security people. Nobody's— no board is ever going to sign off on that level of investment, uh, to, to have that big of a team. So, um, so Jim, any last, last thoughts or conclusions that you kind of offer to the audience as they're, they're thinking about, um, you know, how they're going to be communicating and how they're going to be selling this whole software security thing?

37:05Jim RouthWell, I think we have to recognize that even though, and I mentioned this at the front end, that the availability of tools and techniques and DevSecOps practices today has never been better or more mature and giving us lots of choices and options in terms of integrating software security into the core of IT delivery for an enterprise. I think what we have to recognize is that cloud computing changes a lot, just across the board, in terms of the type of technology intensity for an enterprise, and the software security techniques in a DevSecOps model are different from a conventional waterfall approach. An example would be static analysis, which is a godsend for developers in security. Most of the major static analysis vendors have migrated towards almost a factory kind of construct where You scan your code, you get the results back a day or two later, and of course, in a DevSecOps model with agile development, the developers are off on another sprint. And so then they have to get the results of that, go back to the version of the sprint that they were— that they did the scan on, and of course it's changed, and then go back and say, all right, well, how much of the scan results are still applicable to that base of code that we're working on today versus what we were working on 2 or 3 days ago. And unfortunately, the tooling capability really doesn't align itself to a DevSecOps model, whereas a static analysis capability in an IDE is much more It's much easier for the developer to use. It may not find as many vulnerabilities, but frankly, using it early and often outweighs the offset of doing it in a factory model and doing it once and having a core control with outside testers. So, the reality is that tools today, we have choices and options, but those choices are really important. depending on what the migration path is to cloud computing. And we're also seeing a trend, you know, 10 years ago, every major enterprise used offshore developers, and many of them used offshore development for the majority of their development because the economics were so intoxicating. Well, today, There's a shift towards local development, higher quality, more local development, a little bit more diversity in the tools that we're using for developers, and that really lends itself to more of a DevSecOps model. And so, choices of tools supporting the DevSecOps model has to be factored in. And if I'm in an enterprise today thinking about software security, I'm thinking about How do I solve the problems of next year and the year after in the way we're doing computing, the way the evolution is flowing? And that's my primary focus. I may make some different choices in terms of tools and techniques as a result of that. Threat modeling changes quite a bit. Where the software runs changes. Who writes the software And therefore, who do we hold accountable for quality for the middleware components and applications that are being built by third parties? That becomes a lot more important. So, I think anybody in the enterprise today has to look at the trends, understand the choices and decisions that are necessary to support solving the problems next year. But do that now rather than looking at tools and approaches that solve problems, you know, 5 years ago and may be trending towards obsolescence.

41:42Chris RomeoJim, thank you so much for taking the time today to speak with us and share your insights about best ways to communicate and to build programs. And I know I got a huge amount of value out of just being able to listen to your experience in building these programs over 5 times. So, thank you for sharing with our listeners, and we know they're going to enjoy it. So, thank you very much.

42:09Robert HurlbutThanks, Chris.

42:14Jim RouthThanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @appsecpodcast or on the web at www.appsecpodcast.org.