---
title: "Matt Clapham -- Development Security Maturity"
url: https://appsecpodcast.com/matt-clapham-development-security-maturity/
date: 2016-10-11
duration_seconds: 2883
guests: ["Matt Clapham"]
topics: ["Threat Modeling", "Secure Development"]
audio: https://www.buzzsprout.com/1730684/episodes/8122735-matt-clapham-development-security-maturity.mp3
transcript: true
---

# Matt Clapham -- Development Security Maturity

*October 11, 2016 · 48 min*

with [Matt Clapham](https://appsecpodcast.com/guests/matt-clapham/)

on [Threat Modeling](https://appsecpodcast.com/topics/threat-modeling/), [Secure Development](https://appsecpodcast.com/topics/secure-development/)

[Audio](https://www.buzzsprout.com/1730684/episodes/8122735-matt-clapham-development-security-maturity.mp3)

## Show notes

Can you learn enough about a development team’s security practices in an hour to give it useful direction? Matt Clapham joins Chris and Robert to explain his lightweight approach to estimating development security maturity through five key behaviors. They discuss preparing for the assessment, learning from defect records and existing artifacts, and holding a conversation that encourages honest answers rather than anxiety about being graded. Matt walks through scoring practices such as threat modeling and vulnerability response, then explains how the results can guide improvement. The discussion also examines the experience an assessor needs, the model’s limitations, and its relationship to broader approaches such as BSIMM and SAMM. The goal is an actionable view of how a team actually works.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
Security Journey provides application security education for developers and everyone in the software development lifecycle.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Matt Clapham:  
→ [Matt Clapham on LinkedIn](https://www.linkedin.com/in/mattclapham)

Mentioned in this episode:  
→ [RSA slides: Estimating Development Security Maturity in About an Hour](https://www.rsaconference.com/writable/presentations/file_upload/str-w05-estimating-development-security-maturity-in-about-an-hour-final.pdf)  
→ [BSIMM](https://bsimm.com/)  
→ [OWASP SAMM](https://owaspsamm.org/)

Chapters:  
00:00 Estimating development security maturity with Matt Clapham  
01:32 Matt’s product security background  
03:15 What development security maturity means  
05:15 Why measurement can make developers anxious  
08:20 Why a maturity assessment is useful  
11:03 Preparing for the assessment  
16:33 Understanding the application before the meeting  
19:49 What defect records reveal about a team  
21:10 Running the development-team conversation  
22:30 Communication that encourages useful answers  
26:38 Scoring security behaviors  
29:54 Evaluating threat modeling practices  
34:39 Vulnerability response as a maturity signal  
37:52 Can a team outgrow the model?  
42:12 Limitations and advantages of the approach  
45:30 How the model relates to BSIMM and SAMM

## Transcript

*9,038 words · assemblyai*

**0:05 Chris Romeo:** The Application Security Podcast. Here we go. Hi folks, Chris here. Robert and I are joined today by Matt Clapham. Matt makes products more secure. I mean, hey, his Twitter handle is @ProdSec. The topic of this interview is what Matt calls Development Security Maturity. This concept is based on Matt's research and also a talk he delivered at RSA. Matt created a simple process to measure the maturity of development security by looking at 5 key behaviors. We cover the what and why of development security, the 5 key behaviors, and scoring and reporting. As a conclusion, we discuss how to make the results of an assessment actionable. We hope you enjoy. So our topic for today is dealing with what we call development security maturity, and we are have a guest with us today, Matt Clapham, who is someone who's done a lot of thinking and a lot of research into this topic. And so, Matt, first of all, welcome to the show.

**1:31 Matt Clapham:** Thanks.

**1:32 Chris Romeo:** And every guest that comes into this, this interview process with us, you got to tell us your superhero origin story. How did you get into security? This, this crazy thing we call security?

**1:44 Matt Clapham:** Oh, gosh. That's a good question. Years ago when I was a software tester, I started thinking about oddball corner cases. I like to break things. I still like to break things. It's, you know, it's one of the things I do. And then from there, I started being the weird one who would experiment with, you know, not running as admin all the time and finding new and amazing ways to break things. And that kind of got my mind thinking and working into security. And then at my job, when we had a big push for a security focus and needed to have security champions on all the product teams, I really kind of dove into it. full bore and just really got to know the space a lot better, how to, you know, deal with things like testing it and dealing with it from a development perspective all the way up through the IT operations and then, you know, prevention and operational space. So I guess my origins come from that, that being an experimenter and trying to look how to break things that were then focused on the security space.

**2:43 Chris Romeo:** And so now you're applying those skills to the The good side, right? Or not, I guess there's no good or bad side in that analogy, but you're using those things that you learned in figuring out how to break things to make things better on the other side.

**2:58 Matt Clapham:** Absolutely. I use that skill and that experience every day because I think it's critically important, as you discuss here on this podcast, things that we go to design security in, make it the norm, make it easy to use by default so we don't have to worry as much about it later.

**3:15 Chris Romeo:** Yeah, that's certainly advice that we have all the time. So I know, Matt, that you've had a chance to talk about this idea of development security maturity pretty far and wide, and you had a chance to do a talk at RSA last year on this topic. So why don't you set the stage for us a little bit first and tell us what is this thing that you call development security maturity? Sure.

**3:39 Matt Clapham:** To me, development security maturity is How good is the organization, the overall development organization, at designing, building, deploying, operating, maintaining a secure solution? It could be a simple off-the-shelf software package that the customer actually runs, or it could be something as simple as a component, or it could be something as grandiose as a large web application. But, you know, what approach does that dev and/or ops and/or DevOps team put towards making sure that they've worked through the system and gotten all of the different defensive layers in place to try to minimize the security problems as much as possible. And so we can measure that in a way that I came up with in a quick fashion.

**4:26 Chris Romeo:** Okay, and is this something that's going to give us like a high, medium, low, green, yellow, red? Is it that simple of a measurement, or is there more depth into it?

**4:40 Matt Clapham:** Well, it's got some facets, and we'll certainly talk about those in a moment. The key thing, though, is it's just a rubric, right? It's a 5-point scale. It's got some kind of plus or minus. It's almost like a grade point average, if you will. But the thing it gives you is just a reference. It's by no means perfect. It's by no means a real deep dive, but it gives you something to say, oh gosh, we're doing a whole bunch of— we're doing nothing at all, or we're doing a bunch of stuff. And we should, you know, either think about improving over here or not worry about doing as much over there. It really kind of gives you that yardstick to know how scared should you be.

**5:15 Chris Romeo:** Yeah. So now, Robert, based on your experience as a developer, if you were to put on your average developer hat and take off your security guy hat and put on the average developer hat, how is the average developer going to react to that description of what Matt just provided about this maturity thing? Is this going to frighten people?

**5:36 Robert Hurlbut:** It might a little, because especially when you say measure, you know, everyone's thinking about, especially if you've been in a place for a while or even just starting out, you're thinking about, well, as a developer, what are they looking at? How are they measuring my progress? How are they measuring what I'm doing? You know, they may look at lines of code I'm writing. How many bugs am I fixing? And those kinds of things. And now you're adding another—

**6:01 Matt Clapham:** Dimension.

**6:01 Robert Hurlbut:** measurement of how are we implementing security and how well are we doing these things. So initially, that might be a little scary to think about that, but I think that it's another tool that could be useful to gauge where we are because even just like I said, the lines of code and how many bugs are we fixing and so forth really help not only developer in understanding their own progress, but it helps the organization. I think the same thing. It's just another great tool. And so, you know, I think in terms of understanding it from those perspectives, it's another way to help not only the developer to get better at what they're doing, the organization to understand how are we as a whole doing in our progress.

**6:49 Chris Romeo:** Yeah, and I didn't realize that the developers dealt with that. I guess it's kind of like, it sounds like a bit of an anxiety from a measurement perspective. And there could be— is that because they're always being measured against new things, or is it just the same old things that are really prevalent in their world?

**7:11 Robert Hurlbut:** Well, it varies. I mean, like, for example, the lines of code, that's really an old thing. Because if you think about it, I've seen code, for example, that had thousands and thousands of lines of code that made no sense in one method.

**7:28 Chris Romeo:** Wait, that's all my code you just described right there.

**7:30 Robert Hurlbut:** Yeah. So, you know, you can't say that that's a good piece of code versus another method that has a reasonable number, you know, 10, 20, and many, many methods that do things that make sense. So, that's an old way of looking at things to say, oh, how many thousands of lines of code did you write today? I mean, that's an old way of looking at things. But there are still shops that do that. Based on not understanding, you know, how do you do development in the right way today versus just counting how many lines of code did you write? That's not exactly the best way to— So, yeah, there's old and new pressure, I guess, ways of doing things, old ways of doing things, newer ways of doing things. So that's where I throw that in. It depends on the shop.

**8:20 Chris Romeo:** Yeah, okay. So, Matt, based on this definition of what development security maturity is, let me ask what I think of as normally the million-dollar question, and that is, why does anybody care?

**8:36 Matt Clapham:** Well, as I've been developing my method, I've thought of a variety of different ways that it does help. Imagine you're a dev team and you want to know where you're at. and how you can get better, some quick, easy wins, things to focus on, areas where you've got a real glaring deficiency. So, it gives you useful feedback there if you're a dev team. If you're in, say, a position in IT operations and you're trying to find out how scared should you be about that little kludgy one-off app that somebody insists on running that is gonna be used for a critical business process and and, you know, using to calculate financials or something like that, you can kind of get a sense as to what that company that built it has done to really make sure that they're not going to be your undoing. And so, you can use it on both ends of that spectrum and then use that feedback to just really target where to go next, what to improve.

**9:33 Chris Romeo:** Okay. So then, from the— I guess from the average developer perspective, How would you answer the concerns that Robert had of— not the concerns he had, but him sharing with us the average developer's perspective? How would you explain to that average developer about why they should care and what the benefits are for them, the person that's actually writing the code?

**9:58 Matt Clapham:** Excellent question. I always caution, and maybe the use of the word measure, it can be a bit of a loaded term, and it certainly can scare people. I totally get that. It's not— that's just a description of what it is we're doing, right? We're just looking. We're just kind of seeing, hey, what are you doing? And the reason why we can— and the benefit a developer might get from that is not just the feedback that I mentioned before, but also it helps whoever's doing that evaluation, whoever's looking and saying, hey, how are we doing? It can be a great reference to say, yeah, we are doing a lot of good stuff. We're doing all the core things. We're doing above and beyond on several of the core things. So, really, We're doing pretty good right now. Maybe there's some above and beyond things we can do over here in this corner case where we had, you know, the least of the things that we were doing. But overall, we've got a better warm fuzzy, right? Or we can say, hey, we've taken that rubric, that yardstick, and it says we're doing really well. So let's measure it even further and now go into some sort of deeper dive scenario and really get into the nitty-gritty about where we want to improve.

**11:03 Chris Romeo:** Okay. So I think we've set it up here. Let's dive into this process and so we can get a better understanding for how it would actually apply to the average developer, to their team, to their bigger organization at all those different levels. So, where do we start in this process?

**11:19 Matt Clapham:** So, usually, I start out with some pre-meeting research, right? They're gonna talk to the team that's making it eventually, but you gotta know the layout. You gotta understand why are you even looking at it. Is it, like, one of those 2 scenarios that I described earlier? Is it something different?

**11:37 Chris Romeo:** So, just let me— sorry to interrupt here, but from a context perspective, is this a process that only is designed for big companies, or is this something that can be universally applied regardless of how big your company is, or is it something that is really focused on the enterprise?

**11:55 Matt Clapham:** Oh, do you want to hear about how I got to the method?

**11:58 Chris Romeo:** Yeah, let's hear the origin story of the method. That's even cooler.

**12:02 Matt Clapham:** Sure, sure, because that helps put it into perspective, I think. A friend of mine and former coworker, Michael Howard, of Writing Secure Code fame. He and I were talking one day about the core things. If you just boil down a secure development lifecycle down to 4 things, what would they be? I looked at that and said, wow, that's a good starting point. If you just measure the— or not measure, but just say, how are you doing these 4 things? You've got that warm fuzzy.

**12:27 Robert Hurlbut:** Then as things—

**12:29 Matt Clapham:** I got thinking about it more, I got looking into saying, well, as we're moving into this always online, web-connected, the products are more than just product now, they're both part product and part service. We've got to go into the next layer of saying, well, how do we do some sort of security testing on that to double-check? And so I added one more, and that kind of gave me this real simple rubric. And then it dawned on me that, well, I can not only use that as a person working at a large enterprise, but anybody who's looking at and kind of be evaluating the security of something they want to use can use the method to kind of look at it themselves. It might be harder to get some of the answers. And maybe if you're looking at some project, you don't have access directly to the development team, you might have to go in directly and infer, but you still have the capability of using the method to look at it and say, well, hey, where's my warm fuzzy measurement, right? Where am I at on thinking about, should I be scared about these? And so it scales down to, you know, one person looking at an open source project that he wants to use all the way up to a large enterprise that's using it to triage.

**13:34 Chris Romeo:** Yeah. Okay, yeah, that's kind of what I thought. I kind of figured it would be applicable to organizations of all sizes, but it's great to hear how this thing came into— it came to be, and it wasn't just a theoretical exercise that you were doing writing on a whiteboard. This was based on the experience that both you and Michael had had in years of applying Secure Development Lifecycle.

**13:57 Matt Clapham:** Yeah, and I also had the chance to use it when I was in the appropriate role in my enterprise. and actually helped refine it, kind of get the patter down. That's how I, you know, those examples that I've had to say, hey, if you see this, that's a warning sign, all that comes from that runtime I actually used to kind of work the kinks out.

**14:15 Chris Romeo:** Ah, so you've got the scars that go along with the creation of the tips.

**14:20 Matt Clapham:** Absolutely.

**14:21 Chris Romeo:** Yeah, that's good stuff. Okay, so I think that even helps us more to understand kind of where this fits in in any given organization. With that, this process is starting with this pre-meeting research, and let's go back to that and hear about what that step entails.

**14:35 Matt Clapham:** Sure, sure. So the pre-meeting research is kind of understanding what the app might be used for. That's critically important to understanding, you know, hey, should I really care? Is it just a little tool to do some little one-off function, or is it going to be a critical part of the business process? And you can even, as an individual, you can say, well, I'm going to use that to run my web server, or I'm going to use that to make a zip file or something like that, right? I mean, there's a— that scales up and down as well. But you're going to do that pre-market research just to understand the use case if you haven't already. Because, you know, if you think about it, in an enterprise, they're often brought in kind of after the fact or later on in the stage, and you need to understand why is somebody asking to use this thing. From there, you can go to the vendor site, whoever that vendor, that supplier, that creator of that software or that service is. and kind of look around and say, well, you know, are they open and honest about their security story for this thing, or is it something that they've never even thought of? You can also look at other sources of security feedback like the CVE, right? And you can say, hey, this project is known to have these flaws. They've been sitting out there for a while. They're not fixed yet. The company that's making this thing hasn't addressed them. Or you can look at it and say, yeah, we know about those. Those have been fixed. You also might find out that there are a couple that are still open, but the company has been, you know, has a communications plan in place. They've thought about the impact, and they've got a warning. They've got some workarounds. They've got some detail for how you can at least deal with it until they come around and in some, you know, near-period term actually get that thing fixed. So it gives you that background of how does that maker of that software widget approach, making it more secure, building it more secure, communicating out the issues. And it gives you the ability to spot— I don't want to say malfeasance, but misdirection or inaccuracies as you go further through.

**16:33 Chris Romeo:** Okay. So as you're in this pre-meeting, then, are you considering— are you actually doing like the equivalent of a risk or threat profile? of the application itself to help you try to make a call as to where you think they are before you sit across from the actual developers?

**16:51 Matt Clapham:** It certainly helps with that, yes, because you've got the ability to see, well, here's what they're being public about, here's what you could just tell with, you know, as they might say, open source intelligence. And that gives you that pre-detail and also helps you highlight where to go and where to maybe drill in further when you're actually talking to the dev team.

**17:12 Chris Romeo:** So now I got to stop and just make sure I understand this because it sounds like, based on some of the things you said in the pre-meeting discussion here, that you are speaking to a third party or an outside company that's providing some type of software or some type of tool to you. And I thought this process, I thought it was for an internal person to focus on an internal development effort. but that could also be applied to the external side. So is it, is it for all intents and purposes for both, for internal and external, or tell me a little bit about that because I'm a little fuzzy.

**17:48 Matt Clapham:** Excellent point. Yeah, I kind of missed the applicability to internal projects. Well, certainly the description I was giving earlier was about, you know, maybe a commercial project or an open source project that's publicly available, widely known. But you can also apply that to an internal project. Maybe it's something that's only self-made, it's only used internally. But that team usually has a SharePoint site or a Confluence site or a, a something, a Yammer site, something that they use to keep themselves coordinated, in tune, in sync, maybe post specs to, you know, something that they use to, to say, to stay coordinated with each other.

**18:24 Chris Romeo:** Okay.

**18:24 Matt Clapham:** And that can often be used as that source of intelligence. So yeah, you don't have the ability to go to the CVE because it's not public, it's all internal, but you can see maybe their list of bugs in Jira. Right? Something you can look at and say, oh gosh, there's a bunch of security bugs they haven't addressed, or wow, they've really thought about it and they've got all these fixes except for these couple down here that are minor.

**18:43 Chris Romeo:** So, you know, it sounds like you're profiling a little bit here in this step.

**18:48 Matt Clapham:** Absolutely.

**18:49 Chris Romeo:** You're trying to come to the meeting prepared. It's almost like the old adage about lawyers never ask questions that they don't know the answer to. So it sounds like you're doing some of that initial thinking so that sometimes some of those questions you ask, you may already know the answer to. But you're just trying to see and gauge what's their awareness and what is their— I hate to say truthfulness, but are they speaking to me straight or are they trying to snow me a little bit?

**19:15 Matt Clapham:** Absolutely. And there's a certain amount of double-checking that they know their own stuff, that they really truly understand it. Because if they both publicly advertise it or talk about it on their site and they can speak to it, In the interview, then you've probably got the right people on the call, and it's not just talking the talk. They're likely walking the walk as well.

**19:39 Chris Romeo:** Yeah.

**19:39 Matt Clapham:** Yes. Now, they could still be lying to your face and playing a confidence game. That's certainly a risk.

**19:44 Chris Romeo:** Yeah. There's always—

**19:45 Matt Clapham:** But at least you've got some of that coordinated check.

**19:49 Robert Hurlbut:** Yeah.

**19:49 Chris Romeo:** There's always the 5% of people that are going to do evil in any given situation. Maybe it's a little less than 5%, but it's something. So, Robert, I was thinking about a question from from your perspective as a development team member is the things that you've seen development teams create behind the scenes. What are going to be a couple of the most juicy things that you know about from the development side that are created that I really, if I'm the person that's doing the assessment, that I want to get my eyes on within the SharePoint site or the Slack group or wherever the team is keeping stuff?

**20:26 Robert Hurlbut:** Well, definitely, how are they tracking defects? How are they— what are the— what's the information like? Any documentation? But even beyond that, certainly bug defects, and how are they responding? When are they responding? What's their backlog? Those are the kinds of things that I would be interested in knowing about, because those tell a story about how committed we are to Not just identifying issues, but also resolving those issues. If you see a really, really long backlog that's not been addressed in 6 months and you're still releasing every month, there's something— there's a disconnect somewhere. So those kinds of things can tell you a lot.

**21:10 Chris Romeo:** Okay, so the bug database is going to be a good place for me to mine some interesting information that'll feed into Matt's process here. Okay, so Matt, now let's— we're done with our pre-meeting. We've done our work here to prepare. Now we go into the actual meeting itself. What does that look like?

**21:27 Matt Clapham:** Well, when I start to talk to the dev team to try to get answers to those— how are they doing on the different 5 different processes? You've done your meeting prep, right? You've done your pre-market research. Now you've got to make sure you're getting the right people into that meeting. So usually you're going to want to talk to— plan for half an hour to an hour. You're gonna wanna talk to several people from the development team, or a software tester, or a software test manager. You're gonna want a small representative sample of the people who actually worked on making the product. You don't wanna talk to the CISO unless he was right there helping to write the code. He's a great guy, don't get me wrong. He's concerned about risk. He would love to have the feedback about, you know, hey, how can we improve? But he's not gonna know usually the nitty-gritty of how that product was put together and is being built or whatever, right?

**22:16 Robert Hurlbut:** Yeah.

**22:16 Matt Clapham:** So, you've gotta have the right people in the room. Typically a project manager or program manager who knows the right people to bring in. You're going to want to have a dev or a test, and then, you know, maybe somebody else like an architect if the project is laid out that way.

**22:30 Chris Romeo:** Okay, so talk to me a little bit about the communication style that you're going to use in this meeting, because I can tell you that one of the things that I think has been a huge part of my success in my career is that I can just talk to people wherever they are, at their level. If they're really technical, I can talk to them really technical. If they're brand new to security, I can speak at a very foundational security level. But I always try to be collaborative and try to treat them like I want to be treated. So I never talk down to anybody. So give me your perspective here as somebody who could— I could see you being seen by some teams as the enemy as you're walking into the room because it's kind of the auditor thing, right? If they— someone hears auditor, they think, uh-oh, that's the enemy. Don't tell them anything. Tell them only what we have to and answer yes or no. So what's, what's your take on how are you communicating? What's your style when you come into this meeting?

**23:24 Matt Clapham:** Uh, I try to be similar to what you described there, Chris— open, uh, congenial, friendly. Uh, I, I don't want to come into that meeting, um, being standoffish at all, right? It's not about that. It's about, hey, I'm trying to figure out how much work do we all got to do because this is, you know, I don't want to have to throw up alarms any more than you do. So, you know, help me work through this and understand, and then we can all benefit in some degree like I described earlier, right? And so that helps build that rapport. Also, having a development background myself, having worked on product, it's helpful to find some sort of common ground in terms of experience with development, right? Talk, ask them maybe something like, what's your favorite programming language? And we've got a whole podcast just on that topic alone. But then that breaks the ice, that gets them realizing that you're there, you understand their situation, you're not just there to swoop in, take a dump in their pool, and then fly away.

**24:22 Chris Romeo:** Yep. Okay, so the meeting, we've got the right people in the room, we've talked about communication styles. Let's jump into this first key behavior that we want to measure. So, I remember from looking at your slides that this is the developer training and the secure development happens to be an area that I'm super passionate about. But tell us what this means from your perspective.

**24:46 Matt Clapham:** Sure. So, as you mentioned, there's a need to make sure that everybody who's working on that project has some modicum of understanding of their role in developing a secure software product. And it could be just knowing that, hey, I've got to use the static code analyzer and it'll do its best to tell me when I've got a buffer overflow. Or it could be somebody who really wants to go deep dive on something and is really concerned about some particular aspect, like maybe cross-site scripting or whatever, and really wants to get to be an expert in that area. But we need to have that base knowledge of training and that base knowledge of understanding across the entire dev team if possible, or at least the majority of them, because that's going to help. So, you know, the probing on that topic goes around, well, what is your team's approach? How do you keep in touch? Is it just sending Bob off to RSA every year and then waiting for him to come back and say, I found some cool new testing tool? Or is it, you know, hey, yeah, we all do this thing, we all went to some local conference, we've talked to vendors about some other stuff and how their tools might help and understanding their space around what their tools do and don't give us. I mean, what has been that development team's approach to making sure everybody has that base level of education?

**25:59 Chris Romeo:** Yeah, that makes perfect sense there. I think of this as the security community angle as well as the training side because community is even bigger than training. Community is scheduling lunchtime sessions where you gather anyone who was interested in learning about some new thing in security to come together. And there's the networking side and the community side, and then there's the knowledge side as well. And all of them are beneficial to what you're trying to do.

**26:28 Matt Clapham:** Absolutely. And even people who just want to go and watch a video from some security conference online and spend an hour doing that, hey, I'll take that. That's at least something.

**26:38 Chris Romeo:** Yeah. So how are you measuring this then? So you've got— so what, is there a numerical scale or something that I'm using here To decide how— where you are on the maturity spectrum?

**26:49 Matt Clapham:** Oh, do you want to talk about the, uh, this kind of the scoring system right now?

**26:53 Chris Romeo:** Yeah, let's do it. Let's do it. Well, let me ask you this: is it, is it something that's different for each, each behavior, or is it the same scale for each behavior?

**27:01 Matt Clapham:** It's the same. It's, uh, well, you— this behaviors are different, each of them, but the concept is the same across all.

**27:08 Chris Romeo:** Okay, so yeah, why don't you tell us what the, um— tell— give us a breakdown on how you're going to rate somebody based on their Okay, sure.

**27:16 Matt Clapham:** So as I'm going through each of these 5 things, what I'm looking for is first and foremost a real— and I know it's not black and white, it's not binary, but are you doing something? And is that something, you know, as with my professional opinion, because admittedly the method depends on, you know, having at least some background in security and understanding some of the stuff, but in my opinion, does that represent that they're at least doing something?

**27:40 Robert Hurlbut:** Yeah.

**27:41 Matt Clapham:** where it could be as simple as that, watching a video, all the way up to sending half the team to professional training through some vendor, right?

**27:47 Chris Romeo:** Okay.

**27:48 Matt Clapham:** And then those other things like the, well, is it real rudimentary? Is it a pure, you know, pure code review and that's their solution? Or is it something where they do all that training? Those become the pluses and the minuses, right? Are you doing it? And then how many of those different things are you doing above and beyond the base, did you do it, that add and become modifiers.

**28:12 Chris Romeo:** Okay.

**28:15 Matt Clapham:** So back to the training, I gave some examples right there.

**28:17 Chris Romeo:** So some can detract. So there's— if you're not doing something that your model says is crucial and should be done, they're actually going to lose points.

**28:29 Matt Clapham:** Kind of like a grading scale, right? Where you have an A-, B+, and you still got a B. But you're kind of on the trending downside. Yeah.

**28:40 Robert Hurlbut:** Okay.

**28:41 Matt Clapham:** Yeah. Okay.

**28:42 Chris Romeo:** All right.

**28:42 Matt Clapham:** So, so let's, let's move on to the next one. Right. Talking about threat modeling or design review. Right. There's first the just, are you doing something? And maybe, for example, a minus in that might be, well, yeah, we send Bob off and he goes in his office for an hour and he comes back with a little Word doc that says, here's all the threats I could think of.

**29:02 Chris Romeo:** Yeah.

**29:03 Matt Clapham:** Or is it more of a, you know, something in the middle where it would just be neutral as we— yeah, we're used— we're doing something. We've got a few people in a room using a tool that's trying to tell us the threats the tool can think of, and then any additional ones that we might come up with as part of that.

**29:18 Chris Romeo:** So that's—

**29:18 Matt Clapham:** or is it, you know, they're using a tool, they've got some custom stuff built with templates, they're thinking about how to mitigate those, they have requirements across the board to make sure that they're mitigating everything. That would be like a plus or a plus plus.

**29:29 Robert Hurlbut:** Okay.

**29:30 Chris Romeo:** So that's— so you've just taken us from the maybe C minus to a B to an A plus then. in that journey between kind of the 3 examples you just gave. Am I getting the grading scale right?

**29:44 Matt Clapham:** You're getting close. You're getting a point and a point minus or a point plus because all the points add up at the end. Okay.

**29:54 Chris Romeo:** So, Robert, you've spent a lot of time focusing on threat modeling and teaching people how to threat model. What is your take on that as a scoring system? Is that make sense from your perspective, or is there anything else that, as somebody who does a lot of threat modeling, that you want to profile there from a threat modeling perspective?

**30:12 Robert Hurlbut:** Yeah, no, that actually does. I like the scoring from one person doing it, which I always say, don't let one person do this, but if that's all you're doing, then at least it's something to start with, to having more people involved. And then finally, the final part there I thought was really good, which is Really getting involved and getting some output, perhaps using a tool, using some templates, using things that apply directly to the organization, to the product, and getting some good results out of it. I think that's a great scoring system.

**30:46 Chris Romeo:** Okay. All right, so Matt, let's talk about the next one. We got static code analysis.

**30:52 Matt Clapham:** Yeah, I'm sure you're familiar with what that is conceptually. But what we're talking about is just, you know, is the team, after they've trained the staff, they've put together the design, they've reviewed the design, tried to get some of the design kinks out, now they're going to the actual implementation stage. They're starting to put, you know, hands on keyboard code out there. And let's look at that code and see what kind of security things can we spot while that code is freshly written, right? And so the team might be doing something Really simple. Maybe they're just— they've got the code, it's checked into a version control, and they run a lint on it. I wouldn't exactly call that static code analysis, but hey, well-written, well-formed code is certainly readable and easily maintained. So that's helpful later on when you have to deal with something. Or it could be that they've got a challenge in that they're using some newish programming language that doesn't yet have a tool available. So they would like to do something and maybe they have a manual code review process, but They can't do anything because nobody wrote a static code analyzer for Lisp or something like that. I don't know, pick a language, right?

**32:02 Robert Hurlbut:** Okay.

**32:02 Matt Clapham:** Or it could be that they've got all that, they've got rules in place, they've got a standard tool that's commercially supported, that's updated regularly, they have a program in place to make sure that it's a check-in requirement, and they even go back and when the tool gets smarter, they go back and review all the problems in their codebase from before, right? Whatever that tool can now tell them about that they didn't know about before, they go back and at least know that it's there, right? Maybe they're not fixing it yet, but they know that it's available and that they can do something about that, right? So then you've got— they're doing the behavior and they've got a span from minus, they're almost doing nothing, up to, yeah, they're doing their darndest to make sure they write good code.

**32:37 Chris Romeo:** Okay. And let's go right into dynamic analysis and keep rolling on these behaviors.

**32:43 Matt Clapham:** Sure. So dynamic analysis is, you know, like your web application scanner. And even in the case of something where it's just an application you write on a run on a client, there are tools available like AppVerifier and others that will help tell you if you've got just, you know, general potential problems that could turn into a security vulnerability— dangling pointers, buffer overflows, that kind of thing. And so, um, you know, if they use that tool, it could be something as, as simple as the real basic case of, well, they check to make sure that the, the binary is compiled with all the latest greatest switches for stack cookies and address space layout randomization and insert, you know, defense-in-depth platform tech here, all the way up to they're doing all that and they're, they're looking at both, uh, dynamic scans in an automated fashion, some sort of tool. They might be doing dynamic scans via some sort of pseudo-manual fashion, you know, one of those vendors that actually has a small army of independent contractors who will go and happily bash on your app and pseudo-penetration test it, kind of a continuous application security testing, up through even something as advanced as a penetration test where they've actually hired a specific company to come in for a specific target and specific goal and really, you know, tear that thing to shreds, either just within the confines of the app or even deep dive into the bowels of the app and the infrastructure around it and try to really, you know, take it over and And then provide that feedback back to the team. And then also you want to make sure and ask them, in this particular case, well, what do you do with those vulnerabilities, right? Robert mentioned it earlier. Well, you know, they have a huge backlog. They've got all this great feedback from their tool, but they never do anything with it. So, you know, you kind of put that all together and find out what they're doing in that span, and then, you know, that kind of leads into the next one. Yeah.

**34:39 Chris Romeo:** Okay. Yeah. So, that's the vulnerability and response process. So, what is that behavior?

**34:44 Matt Clapham:** I like to summarize it this way. Who gets the call at 3:00 AM when bad stuff happens?

**34:51 Chris Romeo:** In this podcast, that'd be Robert.

**34:53 Matt Clapham:** We should hear about his experiences there sometime. I'd love to hear some fun stories. But basically, that's it in a nutshell, right? You've got an instant response. You've got something that it's gone to the point of it's not just somebody compromised the server, you're reasonably confident that there's a problem in the software that the team wrote. And, you know, explain that scenario to them and say, what's your plan there? You know, do you have somebody who's on a call tree and, and they know, they know who to call? And if that person's not available, there's a backup person. Is that all worked out? Or is it really a, a, what I've, I've seen more often than I, I'd like to admit, It's a, well, that'll never happen to us, so why would we bother?

**35:35 Chris Romeo:** Oh yeah, I hate it when I hear that.

**35:36 Matt Clapham:** Yeah, yeah, really. You know, I know a couple of software companies who have, you know, had to double their coffee bills, uh, for the, the coffee for all their employees because they've had some of these incidents, right? So to think that it'll never happen to you is, is somewhat naive, right? By what evidence do you base that assessment?

**35:53 Chris Romeo:** Yep. Yeah, that's a good, that's a good, uh, word of advice here. If anybody that's listening to this ever says If you ever look at something and someone or something, you present a threat and somebody looks at you and says, that'll never happen to us, run away screaming or demand that we, that you fix it right then because we know that there is no place to hide on the internet and any security vulnerability that potentially exists in your product will eventually be found. So you need to squash them all just to try and make things as good as they can get. So, from a— Matt, from a scoring and reporting portion here, one of the recommendations I think I'd make for our listeners is for— and we'll provide a link to this in the show notes— and that is for them to go take a look at your slides from RSA last year that are posted on the RSA website.

**36:44 Matt Clapham:** Mm-hmm.

**36:44 Chris Romeo:** Because it's— I think it's hard to visualize the mathematical formulas that— or the mathematical system that you have here. But I think if somebody grabs those slides, they'll be able to take what your explanation was And put it together with, with the picture to really help understand.

**37:01 Matt Clapham:** Yep. So here's the, probably the simplest way I can think of to describe it without, without visuals. Each of those behaviors, each of those processes that you've looked for are a check, right? Are you doing it? Are you not doing it? And then you've got pluses and minuses. And so now you can sort of count up the checks and average out the pluses and minuses. So you did 3 of the behaviors, so that's 3. And then you had 3 pluses and 2 minuses, so that's just 1 plus. And then you kind of, you know, average that out and say, well, that's a 3+.

**37:33 Robert Hurlbut:** Hmm.

**37:34 Matt Clapham:** That's a general indicator that the team is doing some stuff. They're on the— they're doing more stuff than not, and they're trending in the right direction. Sure, they might have had a couple little misses on some stuff, but they're trending in a positive direction because they're doing stuff and they're doing more than what they just need to do for the basic behavior.

**37:52 Chris Romeo:** Now, can somebody peg this in all 5 behaviors? Do you reach the point where you outgrow— is it possible to outgrow this model because you get so good?

**38:00 Matt Clapham:** I think I've only seen it once, but yes, you could have a company that is, or a group that is so seriously concerned about this stuff that they're doing all of that and the bag of chips.

**38:12 Chris Romeo:** Yeah, I guess if they outgrow your 5 key behaviors here, they don't need you anymore anyway. They're already passionate. people, and they're gonna go make their own plan. They're like, hey, thank you.

**38:22 Matt Clapham:** Exactly. I'm gonna get the hell out of their way.

**38:24 Chris Romeo:** Yeah, because they're, they're ready to roll if they've got that level of focus. And that's consistent with what I've seen as well in the industry, is, is I would say that there's not very many people that are pegging all 5 of these key behaviors. But at least this gives you a place to start as an organization, because I know a lot— this is a daunting world that we work in here if you're looking in from the outside. I always have to remind myself Okay, everybody in the room doesn't know all the terms that keep floating in your brain here and you keep throwing out. Sometimes you got to stop and just explain what you're talking about here. So it can be scary looking into the world of application security, but I think your 5 key behaviors here give a nice place to start if somebody says, I don't know, I don't even know what to do. You know, we're in such bad shape.

**39:09 Matt Clapham:** Yeah, and by just looking for those 5 simple things, you can give them that quick feedback. So one of the things you can do with that report is, you know, first say, well, you know, you're only doing 3 out of the 5 and you're trending in the right direction. Well, actually, 3 out of 5 would be pretty good. I'm not too worried about that. But say they're only doing 1, 1+, right? You could say, well, you know, you really aren't doing much. I need to see some more stuff. And here's some examples. Here's some suggestions as to what you could do, right? And if that's an internal dev team, then you maybe you can give them a lot more feedback, even a little more context-specific, right? Because you can sit with a team and work through some of that more explicit examples for them. Or if it's just an external company, then you can say, well, look, if you're doing— if you were doing static code analysis, then you really ought to be doing dynamic analysis and threat modeling too, because it's going to help you with that whole space, right? So, you know, take it that next step and do 2 more things. You can give them that feedback at a different level depending on the relationship that you have, the ability and the flexibility and the time to give. But it provides a value to that report so that developer, right, can look at it and say, oh, I never thought about using static code analysis. I'll have to go investigate that, right?

**40:17 Chris Romeo:** Yeah. So, Robert, from your perspective then, when you're working with different development teams, are you normally coming into development teams that don't know any of these 5 key behaviors, or are you finding that they've got some of these things that they're— I mean, where do you think the average development team that you work with when you first get to them, where are they going to fit in Matt's system of maturity here?

**40:43 Robert Hurlbut:** Well, what I'm finding is more and more are starting to look at static code analysis, which is good. Certainly some dynamic analysis and testing where they might start to run some automated tests against, let's say, a website or application or maybe even considering a pen test from a third party. So I see those things in terms of security for a lot of teams. What I don't see as often, and that's why I'm out there trying to help them mostly on threat modeling, is threat modeling is missing and developer training is still either lacking or it's not up to date. They may have had training 3 years ago, but Some of the things that we knew about then may or may not apply today, so it's out of date. And this whole idea of tracking the vulnerabilities and what do I do if we do have a problem, that is also lacking. So I see about 2 out of 5 and maybe 3 with the developer training, but still I don't see as much in there, and I still see developers who don't know anything about threat modeling and secure design. What does that mean? What are you talking about? So that's what I'm seeing today in most organizations if you were to apply these 5 things about where are they now and where do they need to be.

**42:12 Chris Romeo:** Yeah, and that's good feedback. That helps to get some more real-world perspective on how these types of behaviors are playing out in modern organizations. So, Matt, as we start to close and summarize the development security maturity model that you've got here, I'll ask you the harder question first. So, what are the disadvantages of this model? Why would I not want to use it?

**42:40 Matt Clapham:** Well, it's by no means holistic. It does not, you know, look at very specific aspects of a given behavior. It just kind of says high-level, did you static code analyze? Right, not, you know, do you have that feedback loop, do you have all the other things, just the kind of the general thing. And then the other stuff becomes pluses and minuses that are on the subjective side, right? You can look at it and say, well, I think that's a plus. And in my slides, and as we were going through kind of a mini training session on a scoring method in my presentation at RSA, there were some people who came up with slightly different pluses and minuses averages, right?

**43:16 Robert Hurlbut:** Mm-hmm.

**43:16 Matt Clapham:** Because they're thinking through some of my examples and saying, well, I would have called that a minus. So there's some arbitrary aspect to it that certainly biases the outcomes. So those are kind of adding up to some of the disadvantages. Also, it can take time. If you've got somebody who needed to ship yesterday, an extra hour, hour and a half that it would take just for the actual execution of the method, you may not have, right? In the DevOps world where you're trying to ship every 2 hours, you probably don't have an hour and a half to spend sitting around thinking about it.

**43:47 Robert Hurlbut:** Right.

**43:49 Matt Clapham:** Yeah.

**43:49 Chris Romeo:** Okay, so that's— so now that we've talked people out of it, let's talk them back into it. So what are the advantages of using this type of a model?

**43:57 Matt Clapham:** The speed. You know, it might be too slow for a few corner cases where you really don't have a whole lot of time, but if you've got a couple weeks, a week even, even just a few days, you can get that quick feedback on a warm fuzzy. Right. Should I be scared because there's really nothing going on here that's been done with respect to security, or is it something where I've got at least a, you know, they're a 4- and I can, you know, back off and not be as worried about them? It makes a good triage mechanism.

**44:29 Chris Romeo:** Yeah. Yeah. And the other, the other thing that I think is a real advantage, and I would say that this is a model that's based on just do something, is what I wrote down when I summarized it and tried to think about it. And I love the fact that You're getting people in motion as the foundation of this model. So, and what I mean by that is you're giving them a checkmark if they just do something in that category, which a lot of times just doing something is the bigger part of the battle than refining all the more complicated points. It's starting the push of the boulder so it goes over the edge and starts down the hill. Once it starts going down the side of the mountain, it's going to pick up speed as it goes. and take off and, you know, have a life or a kind of a speed of its own as it goes.

**45:17 Matt Clapham:** Yeah, that's a fine way to think of it, right? How far, how much momentum do they have in rolling that boulder down the hill? Are they still trying to break it loose, or have they actually got it halfway down the hill and trying to make it pick up speed?

**45:30 Chris Romeo:** Yeah. And I know people that are more experienced security folks that are going to be listening to this, they're going to be thinking that— they're going to be thinking, Well, what about BSIMM? What about OpenSAM? And so we want to have you come back in the future. We'll do a deep dive into those methodologies and really give it— because that's a full hour-long conversation just in itself. But just quickly, somebody who asked that question about how do you— how does your maturity model relate to BSIMM and OpenSAM? Give them just a quick 2 or 3 sentence answer to that so that they can be appeased.

**46:04 Robert Hurlbut:** Sure.

**46:04 Chris Romeo:** long enough until we create a new episode to talk about those things.

**46:07 Matt Clapham:** Sure. Fast. A lot faster than either of those. I like the depth of some of those. I like that, you know, things like the OpenSAM, you can— it's all available. You can do the analysis. They have the checklists and whatnot for you. But even the OpenSAM can take a couple of weeks and something like a VSIM, probably closer to a month end to end. Plus, you need a lot more people for those projects as well. So, my method is quick, it's dirty, it just involves an hour and a half of somebody's time plus a little bit extra to talk to the dev team. They can spare an hour to make the security guy happy and get him off their back.

**46:45 Chris Romeo:** Yeah.

**46:47 Matt Clapham:** Okay.

**46:47 Chris Romeo:** Well, that's a great advertisement for the future when we'll do a deeper dive into those 2 assessment methodologies. But Matt, I want to thank you for your time today. And where would— if somebody wanted to find you to continue this conversation, where would you recommend that they find you?

**47:04 Matt Clapham:** Oh, on LinkedIn, of course.

**47:07 Chris Romeo:** Okay. And on Twitter as well? That's a possibility?

**47:10 Matt Clapham:** Oh yeah, absolutely. Follow me on Twitter. I'm @ProdSec. Okay.

**47:15 Robert Hurlbut:** That's—

**47:16 Chris Romeo:** so folks, feel free to reach out to Matt and continue this conversation, ask him questions about the methodology. And we'll provide a link in the show notes to the RSA slides that he did so that you can dive deeper into the methodology. Matt, thanks for your time.

**47:31 Matt Clapham:** Well, thanks for having me.

**47:32 Chris Romeo:** Thanks for listening to the Application Security Podcast.

**47:37 Matt Clapham:** Our intro music is 8-Bit Kung Fu by Boring and TJ, and the outro is Southern Delight by Stefan Kartenberg. You can find us on Twitter @appsecpodcast or on the www.appsecpodcast.org.

---

Source: https://appsecpodcast.com/matt-clapham-development-security-maturity/