Chris Romeo — DevSecOps Fails

0:00Chris RomeoHey folks, for this episode, Robert and I decided to talk about an article I wrote called DevOps Security Culture: 12 Fails Your Team Can Learn From. We hope you enjoy this walkthrough of the 12 fails. If we missed any, hit us up on Twitter and let us know what we should add to the list. At Security Journey, we believe security is every developer's job. We work with our customers to help them build long-term sustainable security culture amongst all their developers. Our approach is to provide security education that's conversational, Quick, hands-on, and fun. We don't do lectures. Instead, we let the experts talk about what's important. Modules are quick, 10 to 20 minutes in length. We believe in hands-on experiments, builder and breaker style, that allow your developers to put what they learned into action. And lastly, fun. Training doesn't have to be boring. We make it engaging and fun for the developers. Visit www.securityjourney.com to sign up for a free trial of the Security Dojo.

1:11Robert HurlbutHey folks, welcome to another episode of the Application Security Podcast.

1:16Chris RomeoThis is Robert Hurlbut.

1:18Robert HurlbutI'm a threat modeling architect and co-host of the Application Security Podcast. I'm joined here today by Chris Romeo. Chris.

1:25Chris RomeoHi, Robert.

1:27Robert HurlbutNice to have you.

1:27Chris RomeoYeah, hey, thanks. This is Chris Romeo, CEO of Security Journey, and it's odd, Robert, to be sitting on perhaps this side of the microphone. I guess it's the same side of the microphone I always sit on, but feels like the other side given that we're about to talk about DevSecOps and you're gonna kind of interview me in this process.

1:47Robert HurlbutYes, yes. So recently I noticed that you wrote an article on DevOps or DevSecOps, however we may wanna take a look at it, I really liked a number of things that you mentioned in here, and we were talking about, hey, what's something that we can think about and talk about? This is the article that came back for me. It's just, wow, there's a lot of really great stuff in here, and so here we are today to take a look at it and talk about it.

2:15Chris RomeoYeah, I got the opportunity to write this article for TechBeacon. It's called DevOps Security Culture: 12 Fails Your Team Can Learn From. In this article, I poked a little bit of fun at a number of things in the DevOps world, and so hopefully I won't get any hate mail at the end of this, but if I do, it's okay. I can take it. But yeah, I mean, security culture for me is a big part of how we change how an organization approaches security. If we're going to do that at the speed of DevOps, we've got to be thinking about what is a DevOps security culture. A couple of different facets that I talked about in the article that I'll just mention here. When I think about what is a DevOps security culture, it's really just It starts with knowledge, right? I mean, we need to have our developers and operations and all the people that are working together, they gotta have knowledge. They gotta learn and understand how to build secure software fast, right? We always think DevOps is about building software fast, but if we don't add that secure in there, we've got some other problems. DevOps security culture is also about experience, about how do we improve the process? How do we get the tools to be even better? There's a bit of art and creativity into this as well. We want people to have the ability to be innovative and come up with new security ideas and not just do status quo type of stuff. And then we know DevOps is about a bit of science as well, because whatever gets— whatever matters gets measured, right? How many times have we heard that said in the last 100 years? Probably at least 100. But, you know, in the world, it's this connection between knowledge, experience, art, creativity, and science. These are all things that are the core pieces of a DevOps security culture.

3:54Robert HurlbutThat makes sense. Just diving into the article, the first thing you talk about, which I just mentioned a moment ago about this idea of the naming, tell us about that, the DevSecOps or DevOpsSec or whatever it may be. Talk about that.

4:15Chris RomeoYeah, and I'm not the first person to make this joke about a security fail in the world of DevOps being driven by name and brand. We created this term DevSecOps as an industry, and we really did create a monster here, right? Because you'll hear people say DevSecOps, SecDevOps, SecOps, SecOps with Dev, Dev with a side of security, right? I mean, it's become almost a distraction for us as an industry. We spent all this time thinking about the perfect marketing term, and it's like, Let's just call it DevOps and teach everyone that there's no DevOps without security integrated, right? I'm not the one who came up with that. Julian Vahent, who's been a guest on the podcast before, that was a tweet from him probably 5 years ago, and I literally took a screen grab of it and saved it in my archives because I'm like, that is priceless that he came up with that whole idea. Maybe it wasn't 5 years ago, maybe a little less, but just a priceless idea. Like, we don't need to worry about the name and the brand behind this. That's all marketing. DevOps should have security integrated at every step of the way. That should be the only way we ever think about it.

5:18Robert HurlbutMakes sense. I do like the DevSecOps swear jar. I need to put at least a few dollars already because I've said that several times already in this podcast, but I like that.

5:29Chris RomeoThat's okay. Yeah, I mean, that's in the article I talk about how to change culture as well for each of these, because that's a big part for me is it's not just about making, you know, pinpointing fails that exist in something. It's more about what are we going to do to change this? And so that's one of the things I said is, you know, kind tongue-in-cheek, I said create a DevSecOps swear jar, and anytime somebody uses that term, make them throw in a quarter, a dollar, whatever you want it to be. But it's really a cultural play here for name and brand. It's about we should just call it DevOps and just we all know it has security built into it.

6:02Robert HurlbutMakes sense. And so number 2, you mentioned the infinity graph. So what is that?

6:12Chris RomeoThe infinity graph, I'm gonna just throw it out there, right? We created— somebody created this idea of an infinity, right, to show DevOps as this never-ending thing that just always goes on forever. The challenge is that's not how DevOps actually works. If that was the case, we'd never push to production if it was just a continuous loop that kept going around, right? I mean, DevOps is a pipeline. That's the word that we use to describe this. It's a pipeline visualization. Code comes in on one side, it makes its way through the process, it goes through a whole bunch of tools. If everything goes as it's expected, then that code gets pushed to production at the end of the day. There's no infinity graph here. And so of all the ones that I'm gonna poke some fun at here, this is gonna be the biggest one. Like people gotta start or stop using the infinity graph, and I'm guilty of it too. I used it in the beginning. I was like, ooh, fascinating. What a great way to show DevOps.

7:07Robert HurlbutRight.

7:08Chris RomeoBut the more I think about it, the more I talk with practitioners in the field, it's like, no, we got to ban the infinity graph from our world. Use a pipeline illustration. It makes sense for what it's doing and people can really understand it.

7:19Robert HurlbutSo that helps them think about a beginning and an end of sorts, right? I'm starting and then I actually do deploy something. It's not a continual, I'm never getting off this train. I'm going to— Yeah.

7:32Chris RomeoAnd the other frustration I have with the infinity graph is I saw so many pictures where people would just take their standard DevOps infinity graph and they'd write security and draw and like put a box around it. And then they would say, look, we have security in our DevOps. No, you don't. You just wrote security and put it in a box. I mean, I see what you're— I see the game you're playing here, right? And so that's why I love the pipeline visualization. Like I'm expecting to see security pieces on that pipeline visualization. And then I can say, oh yeah, this is a DevOps that has security properly built in.

8:03Robert HurlbutGotcha, makes sense. So number 3, security as a special team and lack of collaboration. Well, that's a big one.

8:13Chris RomeoAll right, security people out there, I'm talking to you now, okay? So get ready to be potentially offended depending on how long you've been in security, 'cause I'm guilty of this as well. That's the funny part about this. A lot of these I'm coming back to going, I've been in security for a couple of decades now, and yeah, I'm guilty of having this mindset of security is a special team, and we kinda, you know, have all the answers about security. And so how dare you question? Did you not know that I'm from the security team? Look, I have a badge that says security team on it. I mean, of course I have all the answers. the answers, right? And so that attitude, and I'm pointing at myself just like the rest of my security friends around the industry, that's gotten us in trouble because we're known as people that don't want to collaborate. And so that's a fail. If we think of ourselves as a special group of people with all the answers and we don't want to collaborate with the developers that we really need to influence, we're pushing forth this continued approach of siloed organizations and stuff. And it's just not a good way to— it's not a good way to work in the future. And the modern way we work right now should not be about everybody having their own lane, don't step in my lane, that's my lane. No, let's all work together to build secure software.

9:18Robert HurlbutOne sentence you mentioned, or quote, I guess, is, another failure with security teams is having team members who do not know how to code. That's an interesting one because, of course, my background is coding. I've done that for many, many years and switched over to doing more software security, application security. But I've also seen back and forth on that in Twitterville and other places that, well, you know, should you have to code or should you have to know how to code to be a good security person? But so tell them, talk about that a little bit.

9:55Chris RomeoI've come to the conclusion now that if you want to play in the application security or software security space, you got to know how to code. I'm sorry. You just have to. You're trying to influence people who spend the bulk of their day doing what? Coding. You're gonna do— you're gonna go and try to talk to them about a particular feature or some change they should make from a design perspective, and they're gonna say, hey, look at the code, and you're gonna go, oh, sorry, I don't understand. I don't know how that works. A big thing for me in 2021, and I started talking about this in 2020, and I'm gonna keep talking about it in 2022, is developer empathy.

10:33Robert HurlbutMm-hmm.

10:34Chris RomeoWe as security people have to start seeing things through the eyes of the people we're trying to help and we're trying to influence. We gotta stop saying, we're security, we have all the answers. Oh, we don't need to know how to code. You do all the coding. I just do the, you know, checking off the box things at the end. That doesn't work. We have to say, how does this change? How does this new security thing I'm trying to do, how does this impact the developers? If I could get every security person to go spend a week job shadowing a developer, we would have very different approaches to application security as an industry, because those people would go, holy cow, we make you run that? We make you do this with all these tools, and then this garbage comes out, and you have to siphon through all this stuff? Like, when you start to see, your eyes start to open to what the developer's going through. And so I think security being able to code is a part of that, is being able to say, hey, I care about helping your mission enough as a developer that I'm willing to spend the time to invest to ensure I know that I understand the basics of the language and object-oriented concepts and things. And like, right, I'm an example, okay? I've been in security for almost, oh boy, almost 25 years at this point. You don't want my code running in production, okay?

11:50Robert HurlbutMm-hmm.

11:50Chris RomeoYou don't, you just don't. Like, you're not gonna be like, hey, Chris, you're the guy that needs to write this new feature. But I understand the concepts behind code. I understand object orientation. I've written code in Java, I've developed in Ruby on Rails, I've developed, you know, in some other languages that are older than those, perhaps in the past. I'm not a great coder, but I can follow you. If you're walking me through a code example, I know exactly what you're talking about. And if you're like, hey, from a security perspective, what should I do here? I've seen enough to do that. And so I'm not saying you have to go become a senior developer to be successful in application security. I'm saying you gotta just put the time in to support the people that you're trying to influence. And that's what I summarize as developer empathy, is really walking a mile in those developers' shoes and putting forth the effort so that they're like, oh, you know what? These security people actually care about our mission 'cause they took the time to figure out how to code at least at enough of a level that we can have intelligent conversations.

12:50Robert HurlbutAll right, so number 4, vendor-defined DevOps.

12:53Chris RomeoAll right, vendors out there, get ready to get mad at me. Robert, what's your email address in case they need to, you know, send some—

13:00Robert HurlbutYou're right.

13:02Chris RomeoYeah. As I look across the industry, and I've used a couple of different cloud providers, and I'm realizing there's no DevOps standard. There's no IETF, there's no, you know, RFC for DevOps. So who gets to define DevOps and how it works? Do you know who does that? It's all the vendors that provide DevOps solutions. They define what DevOps is. If you're getting your cloud from a certain cloud provider, they have a DevOps DevOps, they have a view of what DevOps is. It's funny how it always kind of comes back to their product space, though. Like, their DevOps is defined by the product solutions they can stitch together to provide you with this experience. My advice to practitioners that are new to DevOps and integrating security is that the vendors don't have to be who sets your agenda as far as what DevOps is, you can figure it out from a best-of-breed perspective, and you may find a cloud provider, and you may say they have a best-of-breed solution from end to end.

14:03Robert HurlbutGreat.

14:04Chris RomeoBut don't think that there's a DevOps standard and they're somehow implementing it. No, they're putting their products together in such a way to try to provide you with a solution. And so just remember, there is no DevOps standard out there. There is, you know, you can define your own DevOps. The vendors don't have— you don't have to let them define it for you in your context.

14:23Robert HurlbutYeah, I like that, that you mentioned there's no DevOps standard. You know, I never really thought about that until I saw that. You're right. I mean, I've heard— we hear things, but that's where we're getting it. A lot of this is from the vendors defining that. So yeah, it makes sense. Number 5, marketing term infatuation. All the terms, lots and lots of terms, it seems.

14:43Chris RomeoI mean, we've got shift left, we've got shift right. I heard someone say shift up and shift down, and apparently we're We're shifting in every direction. And, you know, I wrote this article before our interview with Jim Ralph where he really— the previous interview on the AppSec Podcast, if you want to go back and if you haven't heard it yet, he provided the best definition of shift left as far as in actuality what it is, so much to the point that I'm going to skip over that and just say, no, it's not a marketing term. Go listen to how Jim defined it because he really changed my thinking for something that I would have said, ah, this is a marketing term. Yes, I believe in the idea of starting security on the left, You know, another person, Matt Coles, I heard him say that a long time ago. He's like, it's not shift left, it's start left. Of course, that's what we're, you know, from a secure development lifecycle perspective, it's not about shifting left was a marketing term. Start left is the idea of what we want to do until Jim, you know, changed my thinking about that. But when I think about the fact that we do sometimes get caught up in these marketing terms, that's what I'm just, I'm throwing that as a fail, like being infatuated with these terms. It doesn't change the day, you know, it doesn't change the context of the security in your product. And so, you know, start left, we want to build security in from the beginning. That's what we used to call it back in the, you know, 2000s, build security in. It's the same idea, right?

16:03Robert HurlbutRight.

16:03Chris RomeoAnd then we got some vendors who said, oh, you know, shift left is cool, why don't we shift right? And these are the people that are providing solutions that work in production, which we need those tools, okay? It's all good. But the way I think about it is meeting in the middle, We call that secure development lifecycle, right? It's starting left, it's starting right, it's a process. What is old is new again, but, you know, it's still secure development lifecycle at the end of the day.

16:28Robert HurlbutAnd so, number 6, big company envy. I like this acronym FAANG, F-A-A-N-G.

16:36Chris RomeoI didn't come up with that. I didn't come up with that. That's other people smarter than me in the industry that said, hey, you know, Facebook, Amazon, Apple, Netflix, Google. So we want to refer to them as a group using this FAANG. This one's all about, you know, a lot of times people will look at that and they'll go, well, I heard a talk about how Netflix is doing this and with DevOps and like, wow, they're just killing it. Well, they've been at it for a long time. And if you're brand new to this, a fail is for you to have big company envy and go, ooh, they got it all figured out. And maybe they do, but that doesn't change the day for your program. And your program is not going to be the same as what Netflix is doing. doing, or Facebook, or Amazon, or Apple, or any of these people, right? You're a different company with different requirements and different perspectives. And so, just don't— I'm saying, just don't have that big company envy.

17:23Robert HurlbutYeah. Makes sense. Number 7, overcomplicated pipelines and doing everything now.

17:29Chris RomeoYeah. When you think about DevOps and using pipelines, it is easy very early on to say, we gotta have everything in this. We have to have all of the categories of tools. We might even have 2 of some of the categories to make sure we've got all the right coverage.

17:47Robert HurlbutYeah.

17:48Chris RomeoOvercomplicating anything sets you up for failure, right? I mean, I believe in the keep it simple method. I apply it to everything I do in life, which means I apply it to everything I do in security. If I'm gonna build a pipeline, I'm gonna start with a simple pipeline with 1 or 2 security tools. And then once we get that rolling, then we can add more, we can adapt, but why take everything and throw it all together in a bucket and say, let's just hope it kind of works out at the end? And then that's the doing everything now part of this as well, saying, well, we have to have SAST and DAST, and we have to have SCA, and we have to have other 4-letter acronyms that we haven't even invented yet included in our pipeline. And when we do that, then we build complexity, and complexity is ultimately less secure than simple at the end of the day. You're never going to convince me that a complex system is more secure than a simple system because if it's simple, I can explain it to everyone. I can explain what's happening from a security perspective, and when you try to explain your complicated system, it takes you an hour and you're still not past the high-level architecture trying to tell me all the pieces. So keep it simple. Don't fall into that trap of saying you've got to do everything now. You can phase an approach to DevOps in where you build different pieces of security as you go.

19:02Robert HurlbutSo this next one, number 8, security as a gatekeeper. I think that's similar to the other, but this one in particular, yeah, we are the ones, we're the— was it the department of no, we stop, the buck stops here type of thing?

19:16Chris RomeoYep, and that is a fail for a reason. With security as a gatekeeper, that is something that you don't want to do. DevOps is set up to go fast. Deploy often. And if you're standing in the way with some manual process, all you're doing is, one, you're abstracting, you're taking away all the joy from your development team because they want to move at a DevOps pace. They want to build cool software and have it tested and watch it go to production. And so if you try to stand there and be the roadblock, all you're going to do is build animosity between security and development, which we've already had enough of that classically, historically. We don't need more of it. Let's try to find solutions where we all work together. Once again, developer empathy, And don't try to be the person that's running the gate saying, hey, if you want to get to production, you got to go through us. Work with the team to get the pipeline set up to the point where you've got the security tools you need to gain the assurance about the build, so then you can move at DevOps speed.

20:12Robert HurlbutNumber 9, noisy security tools. How are they noisy? Are they just making lots of noise? No, lots of stuff, right?

20:20Chris RomeoThere's no data center anymore. Like in the cloud world, I can't even go to a data center to hear any of the noise, right? Nah, it's nothing to do with that. It's as far as the output. And I've told this example probably 100 times. I don't care, I'm gonna tell it again. But what does the average team do whenever they get a new security tool? How do they configure the policy for a brand new security tool? Let me just throw out there, I spent 6 figures on this tool. How do they configure it?

20:47Robert HurlbutLet's lock everything down. Let's look for everything and spit it all out so we can—

20:52Chris RomeoLook for everything is the key word. for everything, and you know what happens? What do we send the developer? We send them a 10,000-line report that's 116 pages of all the deficiencies in the code that they just committed. And what does a developer do with that? They have a bonfire when we're not looking.

21:08Robert HurlbutOkay.

21:09Chris RomeoNobody's reading 116 pages of findings that came out of this. And so that's a fail to have these noisy security tools. And you can have noisy security tools later in the lifecycle. You know, you could change policies and make it bad, But I see this happen a lot where people start with a new tool and they're like, I'm going to get my money's worth. I paid 6 figures for this. We are checking for everything because every finding is, you know, we're going to get down to a penny per finding or whatever. What they've forgotten about is the cultural element of the developers are going to hate this tool from the first time it runs in the pipeline, and they are going to do everything they can to work around this tool to minimize the impact it has on their flow. And so if we get a new tool and we install a minimal policy, we can build this thing called trust between the developers and the security team where they're like, eh, you know, they threw that tool in, it didn't overwhelm me, it found one or two things, but when I went and looked in the code, that problem was there so that I could fix it and our code's more secure. Now you've got some trust built in that tool. So now when we increase the policy a little bit, you've made some deposits, right, with those developers. So they're like, okay, they turned the policy up, it's returning a few more things, but They were so good about what the original findings were that they were sending to me that I'm okay with it. Now you're changing culture because it's not an us versus them. It's a we working together on how we're going to find security problems in our code.

22:31Robert HurlbutI do like that part where you say tune the existing security tools. That takes some time. Like I said, you can't just buy it and throw it out there, turn everything on, and just let it run. It does take some time, so there is some effort to be able to make sure that you find the things and set the things that make sense and that are going to not, as you said, also never waste anyone's time with those findings that don't matter. All right, number 10, I think one of our favorites, lack of threat modeling. Not the lack of threat modeling is our favorite, but—

23:09Chris RomeoIt's a fail if you don't do it. Can you believe it? I put it all the way down at number 10. Wow.

23:15Robert HurlbutI know, I was— where is it? Where is it? Oh, there it is.

23:17Chris RomeoCould have been the number one pick on this list. That was number one. Well, I had an order that I used as far as how I was trying to— I was trying to flow down from kind of the business, the high-level, like, industry down into the business, down into the nitty-gritty details. I think threat modeling is a nitty-gritty detail, but I think that's a fail to not do threat modeling. Some people will say, well, DevOps, it doesn't support threat modeling because it moves so fast. Wrong answer. You can threat model anything. It just comes down to where you fit it into your cycle, into your process. And so, you know, you and I, we both love threat modeling. And so we want to see threat modeling be something that's integrated into the pipeline. Maybe you got to put threat modeling more at the, hey, someone grabs a new feature and they start to think about what they're going to do with it. That's when they threat model. And then they start to code, and then they go, they commit their code and it goes in the DevOps pipeline. See, DevOps and threat modeling, they do work together. They're not mutually exclusive.

24:11Robert HurlbutAnd then of course I like the reference to our Threat Modeling Manifesto that's out there too. Uh, definitely check that out if you haven't already.

24:19Chris RomeoYep, and I'll have another article coming out next month for TechBeacon talking about an insider's guide to the Threat Modeling Manifesto and how you can actually use this to take it to the next level and roll it out in your enterprise.

24:31Robert HurlbutVery cool. Number 11, vulnerable code that's in the wild.

24:37Chris RomeoThis is the classic open-source third-party software that's got vulnerabilities built in and nobody knows about it. I mean, we barely even have to talk about this here for our audience that lives, eats, breathes application security. They're running their software composition analysis tools, whether that's open source from, you know, dependency check from OWASP, or whether they're running a commercial tool, they're getting this one. I just wanted to throw it in there as a fail to have a DevOps and not be checking for this vulnerable code that's in the wild.

25:08Robert HurlbutAnd that's pretty— I mean, recently, that's, that's very relevant, right? Just, uh, all the kinds of things that can come in and, and impact our companies and what we're trusting and so forth. So very, very relevant. Number 12, lack of security retrospective.

25:23Chris RomeoWhat, no drum roll? Come on, this is number 12. All right, lack of security.

25:26Robert HurlbutThe final one.

25:27Chris RomeoLack of security retrospective. So It's a fail if security doesn't lead the charge into DevOps saying, we need to look at any security issues we've had and perform a retrospective and figure out what went wrong, a blameless retrospective. Security people, I'm talking to you, and I'm talking to myself too, but I'm specifically talking to us as security people because, you know, for a lot of years we loved to point fingers at people and pass the blame and say, okay, who's responsible for this? It's not me because I'm from security. It's one of the people here.

25:59Robert HurlbutRight.

26:01Chris RomeoRight? You don't have a retrospective where anybody grows, where the product gets better, where the process gets better if you go in pointing the finger. A fail is when we don't have a security retrospective. A positive impact of this is leave the blame at the door, figure out how to grow together, and always practice security retrospectives when we have security failures, just like we would with any other failure. Let's get better from it. Let's not try to pass judgment or pass blame. Those are my 12 fails. There are probably more out there, but if you've got more, Hey, hit us up on Twitter, tell us what they are so that we can add them to our list here.

26:37Robert HurlbutExcellent, excellent. All right, well, Chris, thank you, and, you know, great article, and hope that our folks listening will go check it out as well. There's a lot more interesting things in there as well that you can take a look at, some more details, but DevOps Security Culture: 12 Fails Your Team Can Learn From. Good talking to you about it, Chris.

26:58Chris RomeoThank you.

26:59Robert HurlbutAll right.

26:59Chris RomeoThanks, Robert. Thanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, security is a journey, not a destination.