Jim Routh — Secure software pipelines

0:00Chris RomeoJim Routh has built software security programs at some of the biggest brands in the world. He has served as Chief Information Security Officer or Chief Security Officer 6 different times in his career, always staying close to his cyber and software security roots. Jim has hung up his CISO badge and now focuses on serving on boards and advising security-focused startups. Jim's original AppSec podcast episode is our number one listened to of all time. Having the opportunity to interact with Jim and absorb his vast wisdom and knowledge is a treat for everyone. At the end of this interview, my immediate thought was to go back and listen to this one again. Jim talks with us about the impact of DevSecOps on the CISO, security controls for a DevSecOps pipeline model, and shift left as the dominant theme for software security. We hope you enjoyed this conversation with.

0:55Robert HurlbutJim Ralph.

0:56Chris RomeoAt Security Journey, we believe security is every developer's job. We work with our customers to help them build long-term sustainable security culture amongst all their developers. Our approach is to provide security education that's conversational, quick, hands-on, and fun.

1:14Robert HurlbutWe don't do lectures.

1:15Chris RomeoInstead, we let the experts talk about what's important.

1:18Robert HurlbutModules are quick, 10 to 20 minutes in length.

1:20Chris RomeoWe believe in hands-on experiments, builder and breaker style, that allow your developers to put what they learned into action. And lastly, fun.

1:29Robert HurlbutTraining doesn't have to be boring.

1:31Chris RomeoWe make it engaging and fun for the developers. Visit www.securityjourney.com to sign up for a free trial of the Security Dojo.

1:39Robert HurlbutHey folks, welcome to this episode of the Application Security Podcast. This is Chris Romeo, CEO of Security Journey and also co-host of the podcast. I'm joined by Robert Hurlbut again today, just like every other episode. Hey Robert, how's it going?

2:07Jim RouthHey Chris, doing well. Yeah, threat modeling architect. Good to be here again for this year.

2:12Robert HurlbutI always want to add to the stars, threat modeling architect to the stars.

2:16Jim RouthThat's right.

2:16Robert HurlbutThat's where I feel like Robert is, is at right there. And also a co-author of the Threat Modeling Manifesto that we've talked about a lot of different times on this podcast, to the point where people might be getting sick of it, but That's okay. We love threat modeling. We're just gonna keep talking about it forever. We are joined today by someone who holds the honor of the most listened to episode in his first visit to the Application Security Podcast, still holds the honor years later of the most listened to, and that's Jim Routh. Jim, thank you for being with us again today. And we would love to just get an update to start out, and then we're gonna dive in and talk about software security pipelines and DevSecOps and all kinds of fun stuff, but—

2:57Chris RomeoSure.

2:57Robert HurlbutWe'd love for our audience just to get some perspective on what is Jim up to these days.

3:01Jim RouthSure, Chris and Robert, great to be with you guys and happy to give you an update. I've actually been through a bit of a transition, so I finished my 6th software security program implementation at scale at MassMutual. And I say I finished, of course, you never finish software security, but I did finish my CISO gig. So I'm actually retired from doing CISO gigs now. And I'm on to the next chapter of my life. And I'm one of those that I really don't have to work, but my wife likes it better when I work, and I like it better when I work. So I think I'll work for the rest of my life, but I won't work 7 by 24 by 365 because you only have so many of those gigs in your tank, so to speak. So I spent 20 years as a CISO, 6 different organizations, all of them large, all of them implemented a software security program and made a ton of mistakes for which I learned a great deal. And so that's some benefits just from the scar tissue that I'm happy to share with anyone. So that's kind of what I've been up to. Now, the other phenomena that I've been experiencing, which is something I have to explain slowly because it's hard to understand, I don't do Zoom calls back to back every hour of the day. Let me, let me explain that again. I literally do not schedule Zoom calls back to back all day long now. And I actually consider myself working while not scheduling Zoom calls back to back. So it's an interesting experience. I encourage all of you to at least try it and see, see what it's like.

4:32Robert HurlbutAt least for a day, give it a try, block your calendar and just enjoy this thing called getting some actual work done.

4:40Jim RouthExactly.

4:40Robert HurlbutThat's great. I mean, and like I said, we're going to talk a lot about DevSecOps and pipelines, but I got to ask this question because it's, honestly, it's just one I want to know the answer to. 6 software security programs at scale. If there's one thing, I know this is going to be tough because you're going to tell me there's one book that could be written about all the things you've learned along the way, but if there was one thing just to kind of kick off our interview here, what would you tell folks? Like, here's the one thing you got to get right early for that success.

5:11Jim RouthThat's actually an easy question because Geeks like the 3 of us like to talk about the tools and like to talk about the methods and like to talk about the techniques to deploy in the software development process because, well, we're geeks. And it turns out that none of those things are necessarily a critical success factor for a successful program. The context is we're changing people's behavior. And so it's the human element that determines success or failure with a successful software security program, because cybersecurity geeks have to encourage developers to learn that's not necessarily in the critical path for their view of a project. And the key is to enforce accountability for quality of what developers produce. Now, the good news is developers already know how to own accountability and responsibility for quality. What we haven't done, and this really falls on the cybersecurity professionals, is to instrument the measures of quality specifically around software defects from a security standpoint and just bake them into the fabric of resiliency for the software that's produced. So the answer to your question is the one aspect of a software security transformation that's often ignored— there's people involved and people have to change behavior, and those softer skills in terms of behavior change are essential to any software security program success.

7:06Robert HurlbutYeah, that's great advice. For those that are new to building programs and things, I think that's going to be some great things for them to think about very early on. And so we want to talk about DevSecOps and software supply chain— not supply chains, we should talk about software supply chain because there's a whole bunch of problems there in the world, but we'll leave that for another day— secure software pipelines. And so I wanted to start by just getting your perspective, because when I think about DevSecOps and the CISO, and I know you're retired from that, but it was only recently, When I think about the DevSecOps and secure software pipelines, I want to understand what are CISOs— what are they thinking about DevSecOps? What are they thinking about pipelines? Are they thinking about these things, or are these things that are so far below what they're focused in on? I just wanted to give our audience some perspective because I think it helps to understand the mind of executives, how they're thinking about these types of things. Like you said, we as geeks, we're like, oh, DevSecOps, pipelines, these are awesome. Like, let's talk about them all week long, right? So, for a CISO, what— is this something that you're even thinking about as a CISO, or the average CISO is thinking about?

8:11Jim RouthWell, unfortunately, the average CISO is not thinking enough about this. That much is clear. There's a reason for that. There's a bit of a nuance and challenge associated with CI/CD pipeline implementation that fundamentally has to be talked about more and understood. So, there's a myth. The myth is that If you're an experienced cybersecurity professional or you're an experienced IT leader, you've been through transformations where technology has fundamentally changed. You know, mobile— developing apps for a mobile device is a great example of a fundamental shift. And frankly, you know, when we started looking at developing apps, we, the industry, started looking at developing mobile apps, well, they really weren't mobile apps. They were web apps ported to a mobile device. And the user experience sucked, right? And so, over time, we learned that, oh, well, it's a different format, right? And you actually have different usability factors in the actual device that you need to factor into the design of the application. And we got a lot better at building mobile apps. Well, we have a certain amount, as professionals, confidence in our ability to learn and adjust to new technology because it's cool and exciting and we like to do that. It turns out that there's this myth that is pervasive and more pervasive in larger enterprises than smaller enterprises, but in larger enterprises, there's a myth that software development in a pipeline is just software development and it's not different. And that's a fundamental myth that needs to be abolished because it is different. And if you're a software developer, you're doing different things in a pipeline. If you're a cybersecurity professional, you're designing controls and instrumentation differently than you do in an on-prem world. In an on-prem world, no developer has to think about the network architecture. No developer has to think about the infrastructure management. In a pipeline model, they're doing both of those things, and by the way, they're not equipped necessarily to do it. But they're hungry and interested in learning. So this notion that it's just software development is wrong. It's a different type of software development. And if you're highly skilled as a software developer, you have to learn additive skills to configure the packaging of the build, where all of that is provided for you in the on-prem world. And because of that, the whole cycle of development is faster, more iterative, and frankly, more challenging because you have to live with multiple builds of a product over time and support those multiple builds. And frankly, what you do is you end up just looking forward to the next build, and if it's weekly, you fix your errors on a weekly basis, or even a daily basis if you can get your pipeline to that level of maturity. So it's fundamentally different, and you just have to recognize it's not the same animal of software development as it is in the on-prem world. And so if you have 20 years of experience developing applications for web and mobile apps using a lot of cool technology but on-prem, don't expect it to be the same. It's not. And the challenge that CISOs have is that we've learned software security control design and development implementation in the on-prem world, at least I hope we all have, and the actual controls don't translate verbatim into the DevSecOps model. It's fundamentally different. You need different types of controls. You need this notion of guardrails, and there's much more emphasis on the configuration management. Like, An example is, all right, I set up an AWS account. I'm going to do— let's say the 3 of us are going to do some development together. Now, I know you guys are really talented, so I want you on my team. So I'm going to set up the account. I'm going to say, all right, well, we're just going to start with a development environment. It's not going into production. It's not going to be used by anyone initially, so I'll give all 3 of us access, and we'll all have the same access to do whatever we need to do. And that's a good way to get started. It's a bad way of introducing configuration management defects right into the development environment because right off the bat I'm giving everybody access. Everybody has full access. Nobody has privileged access. There's no privileged access monitoring, and frankly, I have no authentication because I'm just saying free-for-all. So if we started that way and then we built this really cool technology, whatever really cool technology would be, But we built that really cool technology and we want to introduce it to other people. We just add other people into this environment. And then we say, hey, let's get some open source components. We'll, we'll get access to a repo and then we'll start to manage and configure this thing we're building, this project. And all of a sudden there's 20 people doing different roles and so forth. And they all have the same level of access, no authentication and no privileged access management. Now we're doing what developers do and we're doing the same things that we did on-prem, but we're creating a cybersecurity nightmare in what we're doing. It might be cool, but it's a nightmare. And so that is a fundamental shift in thinking that we have to apply to the pipeline model that doesn't exist on-prem.

13:55Robert HurlbutYeah. Do you think this is a situation where the people like that are the developers and the security team, have a better perspective of this across the industry than the CISO does, from what you've seen?

14:07Jim RouthI think there's more scar tissue at the lower levels where they bumped into these things and they kind of have a perspective that, yeah, this is a different animal. I'm all in, I want to, you know, participate, but I don't necessarily know what I'm doing and I'm just going to go forth because someone said we have to deliver something quickly, right?

14:30Robert HurlbutYeah.

14:30Jim RouthSo, and I don't think neither the CIO or the CISO in the large enterprise, I think there's a perspective that my massive enterprise can build industrial strength applications. We do that well, so we'll figure this out. It's not, you know, it's not a big deal. And I think what I'm suggesting is A, it's a big deal, and B, put some method to the madness here in terms of it involves fundamentally changing roles. So if I'm a developer in a DevSecOps model, I'm doing fundamentally different things than I'm doing if I'm a developer on-prem. So why do we think we're just going to move developers with on-prem development experience into a pipeline model and say, have at it and everything's going to work out? It just, it's foolish, right? It's not, it doesn't work that way.

15:18Robert HurlbutYeah. It's funny, your privilege, your kind of identity and access management example. I was literally dealing with that this morning, like 2 hours ago, thinking I'm like, and it's, it's the cloud providers make it oh so easy cycle permissions and give someone the permissions you actually want them to instead of full admin. And I'm saying that with a hint of sarcasm because it's not. It's like this giant— like, try to figure out, and then you assign some privileges and then it still doesn't work. And so you're missing one thing. And that's what we were having some fun with this morning, trying to figure out what was the missing. It was like the game of play the missing privilege. Like, there's one missing, there's 20 that are there.

15:56Chris RomeoYeah.

15:56Robert Hurlbutthere's one more needed to unlock all of the capabilities of the other 19 I had on the list. And so then we were, we were having that same conversation like, oh, we could just give full admin, but we're like, no, that's not a proper security approach. We're going to fight through this and figure out what's the right privilege so that it's— we're living in a least privileged world like we know we're supposed to.

16:15Jim RouthBut— And the irony there is we know how to do identity access management. We know how to develop software. So why don't we know how to do identity access management as applied to developing software in a cloud environment? And unfortunately, we don't. We have to learn the hard way.

16:32Robert HurlbutYep. It's always going to be— there's always going to be some difficulty there. Well, let's talk about security controls then from your perspective in a DevSecOps pipeline model for dev. I mean, you set up nicely the kind of tension between on-prem development knowledge people have and going into a cloud world where they're using pipelines and doing this, but I'm curious as far as, like, what are you thinking about as far as the controls you're expecting to see if you come in and you look at somebody's pipeline?

16:59Jim RouthSo let me start with this as a basic construct. In traditional or conventional software security, there's this notion of the developers develop the software and then somebody tests the software. And part of the testing is functional testing for security defects, and that's done. And then a bunch of defects are identified, and then they're assigned back to the development team to fix those defects before that codebase is launched into production. And that's kind of the traditional, conventional set of controls. The notion that a team builds software and somebody else tests that software has to go away. That's fundamentally flawed, I would argue, even in an on-prem world, but it's certainly in a DevSecOps world.

17:49Robert HurlbutYep.

17:50Jim RouthSo just do away with that, and instead, the cybersecurity professional should think in terms of the enforcement of accountability, of quality for software with the owners of the software development process. Guess what? That's not the cybersecurity professionals, right? It happens to be the DevSecOps team. or the DevOps team or the digital team or whatever those avant-garde innovative folks that are getting development money, whatever you want to call them, they own the quality of the output. Our job in cybersecurity is to define as part of the quality that they own resiliency, and resiliency means fewer software defects. It doesn't mean zero software defects. It is software. All software has defects. So you have to recognize that there's nothing like— perfection is not a word that belongs in a sentence that involves software development, right? So, but we have to get the defect density to an acceptable level, which takes place in a sustained way over time, and the ownership and accountability for doing that rests with the DevOps team. The cybersecurity team has to design an— I call it instrument— the controls within the native environment to enable guardrails to allow the DevOps team to create high-quality customer interface type of software at scale with the infrastructure management and configuration management requirements. And it's almost like taking the entire tech support resources that are available in the on-prem world and parsing it up into different responsibilities on the DevOps team. And creating a new set of roles. There's not like one cybersecurity role in a DevOps team. I can think of 4 or 5 roles in the DevOps team. There's somebody might specialize in composition analysis. Somebody else might specialize in inside the container. Others might specify in configuration management of the infrastructure setup and the networking associated with the particular environment. And So those are— there might be 4 people from a cybersecurity skills perspective that all understand and speak software development that are running the DevOps process, and the CISO has to remove themselves from the central IT set of controls and venture forth in this brand new brave world of designing new controls. And I know one of the things you're going to ask me about is this notion of shift left, which I think today I could argue is well established, both an economic foundation as well as a tactical approach to a software security program, is give the tools to identify defects to the developers to use earlier in the lifecycle, shifting left, and you'll get a lower cost of ownership in terms of the end result and much better risk management and resilient software as a result of that. In the DevOps pipeline model, for those organizations aspiring to a weekly build and then ultimately a daily build, there are many organizations today, you know, Netflix is an example, that have dozens of daily builds, right, that are going on. And to move into that kind of turnaround in terms of the pipeline, there are actually both technology choices, some instrumentation of controls in terms of workflow protection, but also an opportunity to fix defects in the next build. That's not— I don't think that's— it's shifting.

22:16Chris RomeoYeah.

22:17Jim RouthI don't think exclusively shifting left now, Again, brave new world in DevOps.

22:23Robert HurlbutYeah, we'll come back to shifting left here in a second, but I had a couple of clarifying questions based on the talk about security controls in the pipeline. You used the term guardrails, and I'm wondering if you would define that because I think I know what you mean with guardrails, but I'd love to hear your perspective on guardrails first.

22:41Jim RouthYeah, so I would argue that in a pipeline model, there could be hundreds of configuration management decisions that need to be made that are largely about the environment in terms of how to get things into the build process and manage the build process. And some of those may not necessarily be translated to cybersecurity configuration, and it could be just application configuration management, or it could be more of infrastructure configuration management, right? In this category of configuration management, think of it this way. It's a bunch of choices that the DevOps team has to make, and the outcome of the choices will have implications in terms of the resiliency of the pipeline itself and the code running. Those decisions that get made aren't necessarily— the downstream impacts of them aren't necessarily clear. There has to be some guidance and guardrails that says, look, you set up a dev environment, no matter how many developers you have, make sure that there's authentication enabled so that whichever developer is using the environment has to authenticate into the environment. And if you think that's silly, go back and read about SolarWinds. And then, and then when you realize that it's not silly, then come back and say, okay, maybe we need multi-factor authentication for every single developer. Now, that's a decision that if the 3 of us were building an app in a pipeline today, we could make the decision one way, which is the first way, which just says we all get access. No, it's a pain to do authentication, no authentication. Or we say that maybe what we're building will turn into something beautiful that'll run in production, So let's put the controls in place today anticipating that it's going to be in production, and those decisions of configuration management have security implications to them, and so let's understand those implications as we make those decisions. That's essentially the concept of a guardrail, that it basically could say if you set up an environment without authentication, a red light goes off. Or an alert goes out, or you're just not able to do that without some level of permission. So that's like a guardrail. And when you drive on a windy road, on a switchback going up a mountain, where you have a guardrail on the end of the curve, you feel a little bit more secure and you're kind of able to see, okay, the turn is going this way and then the curve is going back this way, right?

25:27Chris RomeoIt's guidance.

25:29Jim Routhessentially what it is. Now, if you hit it, the guardrail, it'll bounce you back and you won't go off the cliff. It might damage your car, but that's better than going off the cliff, right? So essentially the same construct is to give guardrails to help make those hundreds of configuration management decisions in setting up and establishing a DevOps CI/CD pipeline model, and there's a lot of infrastructure management that today is software, and there's this notion that's really kind of cool in some respects because it's new and different, but it's also challenging, of policy as code, meaning that you're writing code, sometimes scripts, around the configuration management process that provide the guardrails, and essentially The evidence of those decisions around configuration management become the artifacts that you say, that's our policy. Every time we do a build process, we're doing it this way. And if it's deviating, then we have documentation to show that there's a deviation and some checks and balances about that. So that's kind of what it's, you know, when I started in cybersecurity, Chris, We wrote a Word document. Like, our security policy was like— I'm trying to think, my first one, I think it was at American Express. I think it was like 15 pages, right? And it was a Word document and a lot of, you know, thou shalt, you know, do this and so forth, right?

27:03Robert HurlbutYep.

27:03Jim RouthAnd honest to God, it wasn't updated. It was updated like every 3 years. Honest to God. So that was the policy. We've now come full circle to where policy is actually code that is a bunch of scripts and alerts in the configuration management process for a pipeline. And it looks a lot different than that Word document policy, but it's a lot more effective because it's totally aligned with the configuration management decisions being made in real time. So interesting construct for cybersecurity professionals to wrap their head around, but Definitely, you know, something that we need to do for the future going forward.

27:46Robert HurlbutI want to ask you, you use the word cybersecurity quite a bit to describe the people that are here, and you mentioned software security. I tend to use application security. I'm just curious, is there a reason? Are you purposefully using cybersecurity because as a CISO, you're looking at the big picture and you're like, we're all one team? I'd love to hear kind of your perspective on those words.

28:10Jim RouthYeah, that's— I'm glad you asked the question. It's a good question. First of all, I use software security, and I don't think anybody else does. So everybody calls it application security. I call it software security. Why is that? Because I started as a developer, and applications were things we built or bought, one or the other. But if you look at a full-stack architecture in any enterprise, there's software all over the place, right? And so In my mind, it's not just the application software, right? In a DevSecOps model, it's a great example because the infrastructure management software is just as important from a security standpoint as the application software that gives functionality to an end user. To me, application security is limited to the SDLC and the application functionality that's being provided. I think of software as software comes from manufacturers, software comes from open-source components, software comes Today, you have infrastructure management in a cloud environment, that's software. To me, software is a bigger, more pervasive definition that lends itself to more of the type of Lego block building that we do with open source components, with repositories, and so forth, whereas application is exclusively around the lifecycle itself, the SDLC itself. That's me. That's— so I don't expect the industry to change. I've been calling it software security for 10 years. Nobody pays attention. So I think that's just the way it is. So that's, that's kind of one dimension. The second one, what was the second one you asked about?

29:46Robert HurlbutSo cybersecurity, you use that term kind of generally.

29:49Jim RouthSo forget about organizational constructs and in a DevSecOps world, if you forget about organizational constructs, you're actually making a step forward, not a step back. What I mean by that is, as a CISO, my job is to influence software developers and actually to teach them cybersecurity and to enable them to practice cybersecurity. The more developers that I can get into the fold of having some sort of skill related to cybersecurity, the better the resiliency is across the entire enterprise. The worst thing that I could do as a CISO is build up my software security team of cybersecurity talent and say, we are going to conquer the world. The best thing I can do is keep that team to a few people and make them facilitators and instrumentation specialists, and then create as many software developers certified in cybersecurity as I possibly can. I am suggesting Thousands. And so, and I as a CISO have to influence them with this kind of behavioral change notion that they're accountable. And again, in a DevOps world, there might be 4 or 5 cybersecurity-specific roles that are embedded in the development team, and you might not be able to tell a developer from a cybersecurity professional, but they have the ability and they demonstrate cybersecurity skill and how they do their craft. And as a CISO, I have to influence that, and that has nothing to do with organizational boundaries. In fact, organizational boundaries are a bit obsolete when it comes to software security.

31:36Robert HurlbutYeah, I want to come back and just touch on the shift left question because I know you mentioned that here. And, and so when you're thinking about shift left, are you thinking marketing term? Like, where are you on the spectrum between marketing term and getting stuff done in the pipeline?

31:52Jim RouthSo in the 6 software security programs that I stood up, I never had any trouble getting funding to do a complete transformation of the software development process. It was the easiest thing in the world. Now I say that because most organizations struggle with getting enough funding to pay attention to software security, and the reason is because they're trying to quantify the benefit of a software security program in terms of risk management, and it's actually absolutely the wrong approach to take if you want to get unlimited budget. The right approach to take is to say, we spend— we, the enterprise— we spend money developing and managing software, and just try to put a dollar figure on how much money we spend a year, and I'll use a billion as just a marker because that's kind of on average with the types of large organizations that I've been part of, a billion dollars, spending a billion dollars on software development is kind of the norm, in some cases a little bit higher than that. So if I, if I'm spending a billion dollars to both develop and manage software, then there's a total cost of ownership for what the software that's built. And if I can reduce the total cost of ownership for the software by 15% annually, And I can commit to that to the CFO and the CIO and the CEO. There is no CIO, CFO, or CEO in their right mind that would say no to that. Because 15% of a billion dollars is a lot of money, right? And, and it's every year you get the benefit, right? You'd be crazy not to say, okay, what gives? And my answer to that is, I'll deliver that for you across the entire enterprise. As long as you give me 15% of what we spend on software development upfront to invest on a 3-year program, and by the end of the third year, you're going to be at 15%. By the end of the first year, you'll, you'll probably be a little bit underwater. Second year, break even, and third year, you'll hit the 15%. And again, that kind of decision is a decision that business leaders make every day. they're comfortable making that kind of decision. And so getting upfront capital to invest in the tools and the processes and the people change to support a program is easy. Now, all of the economics that I just described has nothing to do with risk management and everything to do with shift left. So if I can remove defect from the process, then I don't have to fix the defect, and the total cost of IT ownership goes down considerably because I'm not spending labor dollars to fix defects. If I can identify the defect early in the lifecycle versus after production doing a pen test, then I have a 10x improvement in the labor costs associated with fixing the defect in the development cycle. That's the fundamental premise of shift left, is higher quality, lower cost, the more you instrument the defect identification and management process in the software development process earlier on. Now, what changes with that? In a DevOps world, if you can put some software in the QA build process and the same software in the build process that actually protects the workload in real time that not only identifies potential anomalies to the application behavior but blocks certain anomalies to the application behavior using some, you know, decent tooling from an ML perspective that does pattern recognition of the application performance, then I'm insulated as an enterprise against backdoors or malicious DLLs that got embedded in the supply chain. It does not take a genius to figure out that, hey, that might be relevant today. It might, it just might be. There are 5 or 6 companies that I have seen just in the last couple of months that all have products that are in this category of workload protection. Every DevOps organization should consider playing with some of the technology. It's emerging technology. It's not bulletproof. It's not mature. It's not scalable across the entire enterprise necessarily, but it's a place to start. And so that's where if we put those controls in place, are we shifting left? And I would argue no, we're shifting right, right? We're fixing it in production essentially, or protecting it in production, right? So it's not really a shift left, it's more of a shift right, but it feels like the right thing to do, at least in a pipeline model. So I think, you know, I'm a big believer in data science being a foundation for cybersecurity, and that's not common practice, or most organizations don't necessarily embrace that, and I'm used to that, but someday I think they will. And in software security, I do not know why, but we— the number of products that are available that have an ML framework is tiny, and I do not know why. I think we should use data science to help and enable the software development process improve resiliency, and I think there is a great opportunity to do that, and I think in workload protection capability, we are just scratching the surface, but You don't have to be a large enterprise that invests a lot of money in R&D in order to do this. I think all you need to do is choose a product that's available today and try it out and learn from that. If it works and you can learn from that, then broaden it, but try it out in your pipeline. I think, what's the cost of that? We're trying it, we're doing something in R&D. Look, we do something in R&D every day in a DevOps model. Let's just do it from a resiliency standpoint, and we may end up protecting ourselves from supply chain poisoning that's going to be pervasive and is pervasive today.

38:14Robert HurlbutYeah.

38:15Chris RomeoWow.

38:15Robert HurlbutThat gave us a lot to think about here. We definitely— yeah, a lot of— there'll be a lot of days in the future where I'm still pondering a lot of these things. One of the ways we like to end episodes now is What's your call to action for our audience? So we talked about DevSecOps and the CISO. We talked about security controls. We talked about shift left. For somebody who's listening right now, what can they take away? What homework assignment would you give them kind of coming out of this conversation to help them get better at DevSecOps or at being in tune with the organization and ways to connect and make these things a priority where they work?

38:53Jim RouthIf you're in the cybersecurity world today, You need to cease and desist all thinking that you're responsible for the resiliency and the security of the software product that is developed. Stop thinking that. Stop believing that. Stop practicing that. Stop saying that. Make that no longer part of your lexicon and no longer part of how you make decisions on where you're going to allocate your resources. It's a paradigm that is no longer valid. The right paradigm is that ownership of the accountability of quality of software belongs with software developers. It always did, it always should be, and it always will be. I mean, and so let's just embrace that and then start to use our creative powers to create instrumentation to support the quality of software, which means you have to measure the quality of software, and you have to make sure that in the quality measures that defects from a security standpoint are included. If you're a car manufacturer and you have a car rolling off the assembly line and every 10th car has dents in the front passenger door. There's 2 choices you have as a manufacturer. You can hire someone to fix the dents, ergo pen testing, okay, or you can go back into the manufacturing process, adjust your robotics in the door assembly to avoid the dents in the first place. And then when the car rolls off the assembly line, there's no work to be done to fix the defects. So our job in cybersecurity is to figure out how to fix those defects through instrumentation early in the development process and enable those tools to the development team and the DevOps team. And I don't care if they're called digital, if they're called DevOps, if they're just called developers, I'm totally indifferent. If anybody in an enterprise has responsibility for building software, then my job as the CISO and the entire cybersecurity team needs to support that accountability and enable that accountability and move towards a DevOps model, which ultimately will deliver a lot higher quality at a much lower unit cost than what we've been doing. At large enterprises, let's face facts, and this is kind of a harsh reality here, but every large enterprise I've worked with has done offshore development, and sometimes 60%, 70%, 80% of the development is done offshore. Offshore development is a wonderful way of creating mediocre software at scale, right? That's what it does, and it's predictable and it's, it's repeatable, but that's what it is. If you want high-quality software from a user experience, from a digital consumer experience, you want high-quality software, you're going to do it onshore. You do it with a lot more talented people, and you're going to use a pipeline. Why? Because that's what we're doing today. And it works. Now it's much higher cost comparatively to the offshore model, and it doesn't mean offshore model is something to be discarded and is bad. It just means the reality is the DevOps pipeline today is how we're delivering software for the today and for the future in the enterprise. We got to come to grips with that as cybersecurity professionals. We have to instrument that for higher quality. We have to enable that through better controls and infrastructure as code. And ultimately we have to deal with all of the changes from a professional development standpoint to manage that process so that talent can be supportive and not resistant of that. And that's not a little thing. It's a big thing in enterprises today. The enterprises that are all over the pipeline model in a highly mature way, never had a legacy to deal with. They started right from cloud first. Those are the industry-leading DevOps models. Legacy is hard, you know, and big enterprises that you have systems built 30, you know, years ago, it's hard to transform it, and fundamentally the people skills are different, and so part of the CISO's job and mandate is to help with the definition of the new roles outside of the cybersecurity organization, but the new roles that are part of the DevOps team. The best way to do that is take some cybersecurity professionals and embed them in the DevOps team as full team members developing code. It's not conventional, but it works.

43:37Robert HurlbutThank you so much for sharing your insight. I always think of you as one of the people who I look up to and say, here's somebody who's done this a lot of different places, and so I could literally listen to you talk all day long about these, these topics, but we'll stop here for now and just know that in the future we need to do another episode. We'll pick another topic to talk about, I'm sure, where there's a lot of things we could, we could dive into. But thank you for taking the time to share your insight and your experience and history with our audience, and we look forward to having another conversation soon in the future.

44:08Jim RouthChris and Robert, it's my pleasure. Great to see you guys.

44:12Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, security is a journey, not a destination.

44:37Jim RouthThank you.