Mark Loveless -- Threat modeling in a DevSecOps environment.

0:00Chris RomeoMark Loveless, also known as SimpleNomad, is a security researcher and hacker. He's spoken at numerous security and hacker conferences worldwide, including Black Hat, DEF CON, ShmooCon, and RSA. He's been quoted in the press, including CNN, Washington Post, and the New York Times. Mark joins us to discuss his series of blog posts on threat modeling at GitLab. We discuss his philosophical approach to threat modeling, framework choice— spoiler alert, it's a pared-down version of PASTA— and success stories and best practices he's seen for threat modeling success at scale. We hope you enjoy this conversation with Mark Loveless. Are you trying to build a security champions program? Everyone is these days. One challenge of rolling out security champions is, how do we educate all these new folks? Security Journey has your answer. We provide a Security Dojo environment with level-based security education that gives your newfound champions a path to follow. And the best part? It requires almost zero administration by you. Visit www.securityjourney.com to set up a demo and learn how you can use the Security Dojo to connect with your security champions. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am the CEO of Security Journey and co-host of Instead podcast. I'm also joined by Robert today. Robert, how's it going?

1:34Robert HurlbutHey, Chris. Yeah, Robert Hurwitz, threat modeling architect, doing well. And today I know we're going to be talking about one of our favorite topics, threat modeling. Looking forward to it.

1:44Chris RomeoWe just, we can't stop talking about it like every other episode. Like we're going to have to rename, we have to rebrand. We're not the Application Security Podcast. We're now the Application Security/Threat Modeling/DevSecOps/a number of other things. Podcast, but that's not gonna roll off the tongue as well as Application Security Podcast. We're joined today by Mark Loveless, and Mark has written a blog post about how he's approaching threat modeling in his professional environment, and we're gonna get to that. But first, Mark, we always start right away. We jump in headfirst into the deep end with your security origin story. So, how did you get started in this crazy world of application security?

2:26Mark LovelessWell, hello. Good to be here. How I got started in this is decades ago, my father brought home an Apple II, not an Apple II Plus, not an Apple IIe, an Apple II computer. And I started playing around on that and Not too long after that, I got to— ended up being pretty handy when it came to figuring out how to break copy protection on the commercial software so my dad could make backup copies of software. And he kind of had a, I guess, what essentially would be kind of a wares group at his place of employment with all the other Mac owners. And that got me into security. Modems came, you know, and then of course later on the internet came. And that's just— that was it. It was helping my dad make copies, illegal copies of software was what got me started in this weird thing of security. I had no intention of going into computers at all or security. And but I've been doing it for several decades now and I just absolutely love it.

3:54Chris RomeoVery cool. Yeah, that's my origin story has a lot of similarities there to copying. Now it was, I think it was Apple IIe for me and Commodore 64. So I was, we were copying things. And the funny thing is, and I don't know if you ever did this as well, but half the time the fun was just copying it. It wasn't even— I didn't even want to play it. It was just, how can we, how can we copy this thing? How can we make it work? And then I'd get it and I'm like, oh great, I'm not gonna put on the pile over there, you know? Um, so yeah, that was the same thing.

4:29Mark LovelessThat the same thing. Games were the big, the big thing everyone wanted to copy back then. And, uh, because they were, you know, so utterly expensive, you know? So yeah.

4:42Chris RomeoYou're making me nostalgic now. I'm remembering those old days right now and wanting to go back and experience that again. But we got to get to threat modeling. And so the blog post that, the first one in this series that you've written is How We're Creating a Threat Model Framework That Works for GitLab. And so we're going to put that—

5:04Robert HurlbutRight.

5:04Chris RomeoThe link to that in the show notes so people can read the actual article. But I found this article and I immediately reached out to Mark and said, I want to ask you 1,000 questions about everything that you've done. And so, Mark, tell us a little bit about the— is there a series behind this? Is this just a one, a single post, or is this going to be like a series of posts to people?

5:22Mark LovelessThis is the first of 3. Um, this first one kind of gives kind of an overall perspective of where we're at, uh, with the entire thing. And, uh, once we, uh, the— I mean, the second one is probably more— it's more or less written so to speak, and which we're going to go into more detail into what we did to alter everything from what normally people might do, I guess. And then the third one will be a fairly detailed example of one of a particular threat modeling scenario that we did as kind of a test of it, but also you know, what, you know, what we'd learned from it and how it influenced what we're doing.

6:11Robert HurlbutSo cool. So just curious, one of the things that you talk about in the article and one of your focus points is, and actually a good question that we hear quite often, is if you have a DevSecOps environment or trying to set that up, Philosophically, how— what is your approach? How do you approach that for threat modeling? And then also, what's maybe different than the classic approach?

6:45Mark LovelessWell, a lot of times what we saw and what I've seen at previous employers where threat modeling has been involved is there's been a— a lot of times the threat model comes in at the end because the development team, they go to security to get their application approved for release and whatnot, or their code changes or whatever it is that they've done. They go to security and say, okay, here we go, what do you think? And then security gives their opinion and then they're back to the drawing board, and that's inefficient. And as a result, obviously—

7:22Robert HurlbutYeah.

7:24Mark LovelessDevOps doesn't want to go to Sec and have everything interrupted in their flow when they're trying to meet deadlines. And for right now, there's still an approval process. We still do that here. There's a kind of an approval process, but the idea was let's try to get this to the point where the approval process is a little more streamlined. and try to get the developers themselves to do the threat model themselves and not have to involve security, mainly because I— you could say part of this was laziness on our end, is that we didn't want to go in there and do—

8:17Robert HurlbutMm-hmm.

8:19Mark LovelessThreat modeling for every line of code in the company. We want— it doesn't scale. So it's just like we're going to have to come up with some type of system where it's got to integrate in with what they're doing already. And it's going to have to be something that where they get a benefit from it. And the benefit is they end up flying through that security review.

8:45Chris RomeoYeah.

8:47Mark Lovelessthat already is in place at the end of things. So that was kind of the approach that we took as far as threat modeling goes.

8:58Chris RomeoSo I want to ask a more tactical question, still at the 20,000-foot view, kind of looking out the window of the airplane down at the DevSecOps pipeline. One of the challenges that I've heard people say from integrating threat modeling in DevOps is like, where is the best place to do it so that we don't— because you mentioned flow, right? We don't want to disrupt the flow that these developers are operating within. So from your perspective, when does threat modeling start when I'm thinking about when I live in a DevOps world?

9:32Mark LovelessHopefully it starts before you start writing code. Ideally it would. So at least you could— I mean, it would be in the planning stages for whatever the planning was for that particular chunk of code or for the, you know, say it's a new, completely new project, that right at the beginning of it when they're going through the planning stages is, you know, you know, they're like, okay, Alice, you're going to work on this section. Bob, you're going to work on this section. Here's the overall goal. And, you know, here's what we want the code to do and stuff like that. It's just at that point, that's where it would need to start ideally. Now, I'm not saying that anybody on the planet has achieved that, but that is the goal of where you would actually want that to happen. It's just right up front, so everyone's on the kind of the same page, so to speak.

10:33Chris RomeoSo from your perspective in your program at GitLab, is that something— so is threat modeling something that is like attached to a user story or Like, where does it fit, like, in the developer's perspective? Like, are they getting a user story and then they have to do threat modeling of a story before they can go into writing code? Or is it really just at this point, that checkpoint at the end, and they can— they just have to do it from a governance perspective, but they do it whenever they want to or wherever they want to?

11:03Mark LovelessYeah, it's right now we're kind of leaving it up to them. Because again, there may be different ways of, you know, one group versus another is going to work. And also the same thing that we wanted to do at the same time was we didn't want to limit the threat model to just coding. We wanted to make it to where it could apply to any other scenario within the company. Say marketing was going to do they're going to do some type of campaign where they move into a new market. There's risk involved with that. That seems to be a decent scenario to try a threat model. So we wanted to come up with a framework that was still, at its basics, easy enough to understand so that someone in a completely different department could use it. Our infrastructure department, the people that put the walls and floors in place of our online product, that have all those servers that they're managing, all those containers, they needed to do this, and they're not writing code. At best, they're writing configuration files and making sure access is set up properly and things like that. But You know, they also needed to do this. So it was a completely different approach where we said, this is gonna have to adapt to whatever their existing way of working happens to be. And that was kind of a head-scratcher to get around, you know, not so much philosophically, but just like—

12:53Chris RomeoYeah.

12:54Mark LovelessOkay, we're just going to have to leave it up to them. We can't dictate how it is that they're going to do this. We can't just say, do this at the beginning or else. We had to say, do this where it fits best into your existing processes.

13:09Chris RomeoYeah, and this is probably the first time, at least on this podcast, I've ever heard that example of using threat modeling from a marketing perspective, like to help them use this approach. And I'm certainly somebody who says all the time, like, we can threat model anything. Like, that's— I mean, Robert does the same thing. We say this over and over all the time, but you're making a reality here in that. And so I'm fascinated by that. I could probably ask you 1,000 questions just about that whole side, but I'm going to try to keep focused. Okay. Yeah. As our audience knows, I like to go down some rabbit trails sometimes, but Just to ask a clarifying question on how marketing comes together into that, like, is that something that's mandatory for them or is that something that is available, you can take advantage of it if you want, but there's not a governance step that says you have to do it?

13:58Mark LovelessThere's not a governance step that says you have to do it. And truthfully, there's not a governance step in any of this as it stands right now. I mean, it's just that So we're trying to get this to happen— oh gosh, I almost want to say organically. I love that word. Yeah, you know, just we want it to where they reach out to us. This has been the— whenever we have some type of security thing that we're trying to get to happen within the company, we know that we're reaching success when they start coming to us for whatever that is. And that's what we want with the threat model thing. So we're trying to come up with a way to make it appealing and we make it to where they can actually see, I'm better off using this than not using this. And if we can get to that point, then there we have it.

14:56Chris RomeoWhat are some of the tactics then that you're using to do that? Because that's a great approach. Like, I can, from my own experience, and it wasn't threat modeling, it was Security Champions, but In running the Security Champions program at Cisco a number of years ago, I went from the point where I was walking around begging people to become a champion till I had people knocking down my cube wall saying, hey, I want to be a champion. What do I got to do? You know, it went from— because we had demonstrated the value behind becoming a champion so much that people now said, I want to be associated with that group. How do I get in? It went from like, now there's a membership set of steps. So, on the threat modeling, from the threat modeling perspective though, like what tactics are you using to try to get people to get developers to go, duh, of course I have to threat model. This is what we do here. This is how we do things.

15:46Mark LovelessOne of the ways we've been doing it is, first off, we've been educating the rest of the security department because we've got a decent-sized security team at GitLab. And so we've been stepping through this with some of the other sub-departments, like say the compliance people, they have to do evaluations of a product that we're bringing in, just like, well, let's threat model that. Or we're going to be introducing some— they're going to be turning on some new feature in some program. Let's threat model that. And we've had a few that have come in where they wanted a security— and this happens on occasion— where they feel like for whatever reason they need to have a security review earlier on in the process because they, you know, maybe historically they've had problems where they've had flaws that crop up and they say, we don't want that. Those are our targets because they'll— we'll be in discussion with them in earlier stages than normal. And at that point, just like, hey, let's threat model this. Now, of course, they always roll their eyes and panic, and as most people do, because they say, oh, I don't want to go through and learn a graphics program to make charts and diagrams. And just like, no, no, no, no, no, we're going to do it. We're going to do an easy version. And we try to make it as simple as possible for them to to understand. Don't throw jargon at them or anything like that. And just— and then we— and try to make it, oh, I don't know, fun. There's a, you know, there's kind of an element of fun in threat modeling when you say, okay, let's, you know, no holds barred, let's think of the craziest threats we can possibly think of. And you get some really creative, insane attack scenarios. And And that's, that's once you start getting that into that and then start saying, okay, now let's, what's practical and, and, you know, for, for this application and whatnot, uh, you start kind of, uh, uh, getting real work done. That's kind of how we've been, uh, trying to get something in there. And yeah, so that's just get our hooks in there and then we're hoping that via word of mouth. You know, we're hoping that, like, say someone who's the project manager, you know, for some project says, oh, I'm going to do this on every project moving forward. And then next thing you know, they're telling their peers, and now they're wanting to do it. So kind of like you said, you know, all of a sudden people are coming to your cubicle and saying, we want that. You know, we want to be able to do that. we can see that it's working. If it improves efficiency, that's what's going to happen is it's going to end up in performance reviews and everything else, you know. So it'll be— the word will get out. So that's essentially it. That's what I mean by organically.

19:03Robert HurlbutVery cool. So there are a lot of— well, a lot, I guess not really a lot. There's a few tools and frameworks and so forth to choose from. Which framework did you choose and maybe why also? And did you alter it or how did you alter it if you did?

19:26Mark LovelessWe looked at a bunch. We used PASTA. The main reason being is that there was an— one, it was fairly inclusive. So you could, you could get back to, you know, the very beginning of the project and, you know, where you're actually starting to scale out the project from a, even from a financial perspective. Okay, what's the scope of this? What's the, where does this, you know, lie within the business and all that? But also the using historical data. So you're able, you know, part of that involves say, okay, historically for this type of code, what are the threats we normally have seen involving this type of code? And since we— and that seemed to be a really good fit for us because we've got a pretty— we're pretty pleased with our HackerOne bug bounty program, which has been going really well. Well, or really bad depending on how you look at it. They found a lot of bugs. No, it's actually, truthfully, it really has been going really well. We've gotten a lot of things fixed and we really enjoy the, you know, just getting all this great stuff in and getting things fixed. We can use that historical data from things like our HackerOne data and be able to point that at code and say, okay, here's, let's say, the top 5 things we pay out on. So we need— so in our threat modeling, anything that comes close to those 5 things, we want to make sure that we go through and cover and that we remediate up front so that we're not paying out later on when some of these fine bug hunters find this stuff. So the fact that pasta has that kind of in there really lent itself to our environment. And that was also— that also makes it a pretty good selling point as well, where you can go to a department that you know, that maybe has had a history of, you know, their code ends up paying out more than others. And you could go to them and say, hey, we can help with this. You know, you'll be able to meet your deadlines easier because you're not going back and patching. And so that kind of lends itself to that. So we ended up with PASTA, and then the main modifications we made to it was PASTA has 7 stages, and I'm going to definitely get into this in a lot more detail in the second blog post when it comes out. But we are going to go through and— oh, Jess, where was I? I lost my train of thought. What were we talking about? Give me a second.

22:38Chris RomeoAltering PASTA.

22:42Mark LovelessWhat you did. Oh, yeah, yeah. Talking about altering pasta. Out of the 7— yeah, it was the 7 phases of that where we would mainly try to focus on 3 of them in the middle, which is basically what are your risks, what are the, you know, most likely of these, and, uh, you know, let's come up with mitigations for them. And so we mainly stuck in that area.

23:17Chris RomeoMm-hmm.

23:19Mark LovelessThat was the main thing we did. And just like, okay, let's just focus on that area only. And that way, uh, it was something that was much easier for them to understand. And at the core of it, what we're wanting out— I mean, what anyone wants out of threat modeling is, particularly for, say, people that are doing code development, it's just like, don't have security bugs in your code. So it's just like, we didn't need the fancy charts and diagrams and everything. And for them to learn jargon, it's just like, nope, just what could potentially go wrong?

23:59Chris RomeoYeah.

24:00Mark LovelessLet's mitigate it upfront. Mm-hmm.

24:04Robert HurlbutAnd for our listeners, uh, if you'd like, you can go back to, I think it was either first season or second season, Chris, I can't remember exactly, but we did talk to Tony UV about pasta way back when. So that's a reference to the past if anybody's interested.

24:19Chris RomeoFar, far into the past as well. So, um, Mark, as I'm, as I'm listening to kind of you describing the pieces of pasta that you've pulled forth. Sounds like there could be maybe a little bit of commonality with Adam Shostak's 4 questions as well, when he kind of boils threat modeling down to its most simple component. And that's where I've kind of found where I've started to drift now. And the 4 questions being like, what are we working on? What could go wrong? What's the third one? Why can't I think of the third one? What are we going to do about it? Of course, mitigation is the thing I always say is the most important. And then, how did we do? Is his 4th question. So, it sounds like there's maybe some similarities. Like, you pulled out the pieces of pasta that were the most important for you, but it sounds like there could be some commonality between the 4 questions as well there.

25:09Mark LovelessOh, yeah.

25:11Robert HurlbutThere is.

25:12Mark LovelessThere is, most definitely. I've known Adam for, oh gosh, easily 2 decades. So, And so we've kind of come from the same world and kind of grew up dealing with the same stuff. So yeah, that's essentially it. You're just boiling it down to its simplest thing. And to make it even— and even then, like, just, you know, we just— we've been trying, like I said, we make it as easy as possible. Like when they were saying, oh gosh, how do I write this up? And we're just saying—

25:49Robert HurlbutYeah.

25:51Mark LovelessLook, we use, you know, we already dogfood our own product and we're used to working in Markdown. Just do it in Markdown. Just create a doc. That's all you got to do. Just do it in Markdown. And if you need a diagram, you know, Mermaid, that works, right? Just include that in with the— you're used to doing that anyway. So it's stuff you already know. We're not going to make you—

26:17Robert HurlbutRight.

26:19Mark Lovelessyou know, download some piece of software or, you know, or anything like that that you have to do anything extra. Just, we're just going to use the tools we have and it'll just, it'll just go in there.

26:31Chris RomeoHmm.

26:32Mark LovelessAnd it's, it's, uh, very satisfying when you see someone, they're working on a project, uh, it may be, let's say, an internal, you know, repository that we're looking at And right in there, threatmodel.md is included in the— it's a part of the project. And it's just like, awesome, this is excellent. This is what we want to see.

26:55Chris RomeoYeah, that's very cool. So let's explore. I know you've told us kind of as we were preparing for this that you had some success stories that you can share about moving the conversation left and You know, having developers do their own threat models. And I know we've, we've talked, we've talked some about developers doing their own threat models, but you also mentioned shortening overall coding time. So I'm curious, you know, what kind of success stories you have, because a lot of times our audience are people that are trying to do the same thing, and they're going to be listening going, how did GitLab do this? Like, you know, what are some, what are some success stories that'll get people kind of pumped up about, hey, I can do the same thing Mark did? And bring that to our company?

27:37Mark LovelessWell, basically kind of the things I've been alluding to. It's just, we had a developer had come to us and, you know, they've been working on— they're working on a project, they need some help, they had some security questions. And one of our guys, he said, yeah, I can help you with this and let's do a quick threat model. And that's how we approach it. We don't say we're going to do a big detailed overarching threat model. They're just like, let's do a quick threat model. And literally within an hour or 2, they're looking at the code. They're talking about what, you know, where all the inputs and outputs are going to be and all this. And they, you know, come up with, you know, I don't know, maybe 10 different items or so that of potential threats, and a lot of them, they're already, you know, they're mitigating on the fly. They're already mentally saying, oh, well, if we do this, this would completely prevent that. So all of a sudden, you know, their code, it just flies through the review process because it was— I think it was someone else that actually did the security review of it. And this was fascinating for this developer. And the very next thing they worked on, she actually did the threat model herself and again flew through the review process simply because she'd already considered all these different security aspects. And to actually go to the security department and say, look, here's, you know, particularly when she's talking with, you know, someone that's in application security that's used to looking for problems and saying, we looked for these problems and here's what we did to address them, was a huge, huge time saver. And it really was just a few hours up front. I'd say a few hours, 2 at most. It really wasn't a lot of time to get this kind of thing done. And then that saved them from having to code and recode and, you know, and then re-go through the process of being certified by the security team as okay. So that helped. And then there's been one other success was On the GitLab standalone product, I went through and did a real detailed threat model on that, found some flaws that have turned into, well, there were already issues open for some of the things that I found. This gave it weight.

30:37Chris RomeoMm-hmm.

30:38Mark LovelessAnd so some of those things actually got some attention, they weren't severe things, but they were things that needed to be addressed and were going to have to be addressed at some point. All of a sudden, they're getting attention, they're getting fixed because they're just like, oh, we went through and threat modeled and here we saw this. And someone said, okay, so this issue we've had sitting there for a month or two, okay, maybe we should go ahead and get to it. And so things are getting solved that way. And that goes into the overall product. itself. And so those are good things.

31:11Chris RomeoYeah, I could see the saving time in the security review. That's a big, audacious, helpful thing for the developer, though, by them being able to say, hey, I did the threat model, I invested a couple of hours upfront, and I got to the security review and it was easy. And I'm just like, I'm on to my next feature. I cleared the security No problem. They were happy with what I saw. That's describing a good, strong security culture because there's that connection between developer and security, and it's not a gate. It almost sounds like you're not running a gate. You're running a coaching service that helps to open the gate for the people when they come through. It's a collaborative work in progress together.

31:56Mark LovelessYeah, it kind of goes to the philosophy. My philosophy is that the main thing a security practitioner needs to do is to try to work themselves out of a job. Okay?

32:09Robert HurlbutLove it.

32:09Mark LovelessI mean, we're never gonna get to the point where we're unemployed, but that should be the goal. That really should be the goal. Just like things are gonna be so secure, we're not needed anymore. And just, it's the journey of trying to get there and push as much stuff as you can off on, off of your plate and onto the plate of the people that actually can do something about it, like developers, that helps. They appreciate it. They think they're just, oh, well, this to them is cheating. Oh, we're going to cheat. We're going to do security's job before they get to it. So they're happy. They're happy with it and we're happy with it.

32:47Chris RomeoYeah. Mark, we think way too much alike. This is dangerous. Like, you're speaking and I'm like, How does he know all these things already that are in my brain? But I think what you've described in the program is just— and I'm looking forward to the next 2 blog posts. I'm going to be on the edge of my seat waiting, and I will be one of the first readers when I see it come by because I can't wait to see where you're going to go. I think we got a little bit of a preview. Sounds like we got a little preview of part 2 in this conversation, but very excited to hear that next part.

33:18Robert HurlbutYeah, definitely. And likewise, looking forward to it as well. So, Mark, thanks for joining us today. Just to sort of close out, curious, could you give us some key takeaways as well as a call to action for our viewers and listeners?

33:33Mark LovelessA key takeaway would probably be, well, kind of like what we just said. It's just like we— this is a step towards time efficiency for all parties involved. It allows threat modeling to scale. It allows coders to get through something a lot quicker. And it's— if you make it easy enough for them to understand, and they feel like they're cheating when they do it, and they're then it's perfect. So it just— that'd be my main takeaway. It really truly is a lot easier. I probably made it sound more complex than it actually is. If it sounds like, well, that sounds really too easy, that's it. It should.

34:28Robert HurlbutExcellent. And a call to action?

34:31Mark LovelessSearch the GitLab Handbook. We're a pretty open company. You can search the GitLab Handbook. The blog post will have links to the threat modeling portions of the handbook, and there's links to the templates we use. There's links to that describe everything that we have so far in detail. So hopefully that will be enough of an explanation. And I think we've got one One good example in there so far, uh, which is what the 3rd blog post is going to be, uh, we'll be discussing. But we got at least one in there, uh, right now that's in detail that we just decided, well, let's just put it out there for everybody, you know. And, and so just go there.

35:23Chris RomeoVery cool. Well, Mark, once again, thank you for sharing with us and with our listeners your threat modeling program and philosophies and all of those things. It's been very insightful for me to, to hear how you're putting this together in an enterprise-grade company. And so we look forward to a future conversation. We can talk about threat modeling again, or we can talk about something else as well. But thank you for being with us today.

35:48Mark LovelessAll right, you bet. Thank you.

35:50Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast. Podcast and on the web at www.securityjourney.com/resources/podcast. You can also find Chris on Twitter @edgeroute and Robert @roberthurlbut. Remember, with application security, there are many paths, but only one destination.