---
title: "Glenn Leifheit -- An Inner Glimpse of the Microsoft SDL"
url: https://appsecpodcast.com/glenn-leifheit-an-inner-glimpse-of-the-microsoft-sdl/
date: 2016-11-02
duration_seconds: 2983
guests: ["Glenn Leifheit"]
topics: ["Security Culture", "Careers in AppSec"]
audio: https://www.buzzsprout.com/1730684/episodes/8122731-glenn-leifheit-an-inner-glimpse-of-the-microsoft-sdl.mp3
transcript: true
---

# Glenn Leifheit -- An Inner Glimpse of the Microsoft SDL

*November 2, 2016 · 50 min*

with [Glenn Leifheit](https://appsecpodcast.com/guests/glenn-leifheit/)

on [Security Culture](https://appsecpodcast.com/topics/security-culture/), [Careers in AppSec](https://appsecpodcast.com/topics/careers/)

[Audio](https://www.buzzsprout.com/1730684/episodes/8122731-glenn-leifheit-an-inner-glimpse-of-the-microsoft-sdl.mp3)

## Show notes

How do you keep a secure development lifecycle useful as products, teams, and delivery methods change? Glenn Leifheit joins Chris and Robert to share lessons from Microsoft’s SDL and its evolution beyond a rigid sequence of security tasks. He explains the importance of adapting the program to a product’s context, treating developers as customers, and making security help easy to access. The conversation covers community, practical learning, constraints that encourage creative solutions, and the need to evolve alongside development practices. Glenn also describes mentoring across roles and seeking ideas outside the security industry. His central message is that a sustainable SDL depends on partnership and repeatable habits, with processes simple enough to survive staffing changes and the pressures of everyday delivery.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
Security Journey provides application security education for developers and everyone in the software development lifecycle.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Glenn Leifheit:  
→ [Glenn Leifheit on LinkedIn](https://www.linkedin.com/in/gleifhe)

Mentioned in this episode:  
→ [Microsoft Security Development Lifecycle (SDL)](https://www.microsoft.com/en-us/securityengineering/sdl)

Chapters:  
00:00 Inside the Microsoft SDL with Glenn Leifheit  
02:48 Where the SDL came from  
10:54 Adapting security to the product and business  
13:55 Making developers want to ask for help  
23:35 Evolving the SDL with development practices  
31:36 Learning from constraints and outside influences  
34:54 Treating developers as customers  
35:30 Building a learning community  
38:56 Mentoring across roles and experience levels  
42:42 Learning from organizations outside security  
46:28 Making security processes sustainable  
47:07 Partnership, adaptability, and practical next steps

## Transcript

*9,687 words · assemblyai*

**0:05 Chris Romeo:** The Application Security Podcast. Here we go. Welcome, friends. This is our second interview from ISC² Security Congress. We are joined by Glenn Leifheit, an InfoSec and Development Evangelist at Microsoft. Microsoft is the grandparent to almost every secure development lifecycle across the industry. This is an in-depth discussion about how to actually do SDL from the people who made SDL. Glenn shares some things during this conversation that I've never heard in public before. about the internals of Microsoft's SDL process. You will take something away from this conversation that you can apply to your program. Enjoy. Okay, folks, we are here at the ISC² Security Congress. Where are we in the world?

**1:12 Glenn Leifheit:** Orlando, Florida.

**1:13 Chris Romeo:** Orlando, Florida. I should know that better, but I feel like I've been on too many airplanes lately. But we're here with Glenn Leifheit from Microsoft, and Glenn has been very involved in the secure development lifecycle process. And he was doing a talk here at the conference about SDL, and so we wanted to sit down with him and just understand what he was talking about here and share that with you, the listeners here. So, Glenn, what, what was the, the talk that you did here at the conference?

**1:39 Glenn Leifheit:** I did a talk on the life and times of the secure development lifecycle. So the idea behind it is historically, where do we start from and where are we today and where do we need to get to in the future? As part of that, you know, a lot of times we We see things that are changing within the business. We see things that are changing within development. A lot of times from a security perspective, we look back and we say, you know, you'll have a development team come up to you and they're like, well, we've been agile for 9 months, but where are you guys? You need to help us. And a lot of times we continue to be behind the ball as opposed to up there with the development team making adjustments as they're making adjustments.

**2:18 Chris Romeo:** Yeah.

**2:19 Glenn Leifheit:** And growing in that way. This is more— my talk was not just about the evolution of where we've been, but where we need to get to in terms of not only joining in a partnership with the development teams as opposed to just them being the developers, making sure that it's a real partnership between the 2 groups so that as they change, we change. Because the only way we're going to be able to keep up with the lifecycle is to know the fact that they did change.

**2:45 Chris Romeo:** Yeah.

**2:45 Glenn Leifheit:** If you tell me 9 months later, well, That's 9 months I'm behind.

**2:48 Chris Romeo:** Yeah. So where did we come from then here in the grand scheme? Because when I think of secure development lifecycle, Microsoft is the grandfather of SDL, of putting those principles together. And I know most of the programs in the industry either spoke to Microsoft behind the scenes or just looked at the website and went, yeah, that's what we need to do. So tell us a little bit about where did we— where do we come from in this SDL world?

**3:12 Glenn Leifheit:** Yeah, from an SDL world, I think the real pivotal moment, you know, way back In the early 2000s, we had, you know, the Code Red and all these viruses that ran rampant. I remember going through the pain, and luckily, my favorite part of my career is the fact that I was actually on vacation when Code Red hit. I never actually had to deal with that until the after-aftereffects. Well, listen, I was still very painful.

**3:36 Chris Romeo:** I was working incident response, and I did have to deal with it.

**3:39 Glenn Leifheit:** Yeah, and I tried to go on vacation, but we had to come back. I was at least out— I was completely out of communication range, so we're It worked out perfectly, but I had to deal with several of the others. And I think that one of the interesting things that we looked at back then was we were so busy trying to fix the problem that the virus was causing that we weren't really taking precautions as to what happened. And back in early 2002, Bill Gates sent out the Trustworthy Computing memo at Microsoft, which has become pretty famous from that perspective.

**4:10** Yeah.

**4:10 Glenn Leifheit:** And that's really kind of when, at Microsoft, we started to take stock as to what the heck do we need to do from a security perspective. That's how it— that's really the initiation of how we changed from, you know, the poster children for being insecure to being poster children for being secure.

**4:26 Chris Romeo:** I remember back in those days when it used to be— that used to be the running joke of— but now when you think about where Microsoft is today, it's— Microsoft is synonymous with security. You got some of the best security in the— from an operating system perspective that's out there.

**4:40 Glenn Leifheit:** Yeah. So from From that perspective, there are a lot of things that have changed, and we've adapted over the years. For instance, back then, it was very waterfall-based. And just like any SDL, we had to try and fail at some things. We tried and we succeeded at some things and adapted some things because the reality is when you go out and you try to do it, even if we're Microsoft, we're gonna go out there and say, okay, we should do this. And then the development teams go through it and go, what the heck is that? And it becomes clunky and figure it out. It takes a while to build a process that's like that, that you can just go publish on the internet and people can go look at.

**5:13 Chris Romeo:** Yes.

**5:14 Glenn Leifheit:** There was a lot of work that went behind it. As we've gone through that history, one of the things that we've seen is that at Microsoft, and this is probably about 2004, 2005-ish, is when we really started to recognize, from a Microsoft perspective, there's really at least 2 ways that you need to deal with the SDL. From the first point is you have to deal with it as a product. which is fantastic, but for products you're building broad, you have to have the security features that all the customers want. That's a lot more than an IT team wants. There's 60 ways to do something and 44 ways to do another and 100 ways to do another. From an IT perspective, running the line of business of Microsoft, from an SDL perspective, we have to go, this is the configuration we approve for as an organization so we can drive forward.

**6:03 Chris Romeo:** You guys have multiple business units too. It's not like you just had people writing Windows-based software. You've got Xbox and you've got other platforms and other types of devices that all have to fit into this.

**6:15 Glenn Leifheit:** Yeah, and I think that that's one of the things that we've— as we went through this, originally everybody was kind of under the same SDL. Now we serve on an SDL committee that helps build the original SDL that, you know, you see on the website and things like that. We all have a seat at the table. But one of the things that we do is we look at it and go, what do we need to run our business the right way? I think you need to adapt to whatever that business is. So, for instance, a good example is Effect. From a product perspective, they specialize in, you know, a lot of the products are built in C, C++, things like that. On the internal side, it's mostly web applications that are internal and things like that. And there's no shortage of those either. You know, being the world's largest IT shop, we get a few of those. in the numbers of thousands. So there's a lot of things, different things that we have to secure. And the more complicated pieces, we take the things that they're developing in the product groups and move them forward within IT. So I'm actually part of the IT organization.

**7:17 Chris Romeo:** Okay.

**7:18 Glenn Leifheit:** Just to help clarify where I'm going with some of this. But as part of that, we actually use these products before the pre-release. So they're alphas. So we were using Windows 10 a year and a half before Windows 10 was out.

**7:32 Chris Romeo:** Oh, wow.

**7:32 Glenn Leifheit:** We were using Visual Studio we always talk about is v.next because that's the only one we're working on, and we know that the name it's under isn't gonna be the name it is in the future. So, we're just working on v.next, and we keep working on it. And, you know, we have to work with all of our partners to help enable security. You know, how do you do static code analysis on stuff that isn't released yet?

**7:54 Chris Romeo:** Mm-hmm.

**7:54 Glenn Leifheit:** And so, we have a lot of different issues that we have to deal with from a Microsoft perspective that have forced us to be very adaptive because we can't keep up with it. And where other organizations may not necessarily have had that level of bleeding-edge type of scenarios, but there's still a lot of morphing that needs to happen because the reality is the one thing that doesn't change is the development team and the business are continuing to adapt constantly to whatever the—

**8:22 Chris Romeo:** So this is actually fascinating to me because I always thought of Microsoft's SDL as one I was thinking everybody inside the company was doing the same thing. So each of these lines of business then has the ability, so they take in that SDL that the committee puts forward and they're able to adjust it so that it makes sense for their business.

**8:43 Glenn Leifheit:** Right. And I think one of the things that, when I first joined Microsoft, so I've been there about 5 years, and I was telling one of my coworkers, and I told him, yeah, I'm going to go work on the SDL as part of part of what I'm going to do. And he looked at me with the strangest look. He's like, but it's on the website. It's done. It isn't done, and there's a lot to do. And I think one of the things that we, as we've adapted through that time, it's really been able to drive home the fact that differentiation is needed. I think one of the things at the last place I worked, FICO, the credit score company, which is where I was at. They, we talked about doing the S— we did the SDL and rolled out the SDL there. And one of the things that we looked at was we've got to have, we always felt like we had to have something that was the same. You know, it's gotta be consistent. The reality is you've got, I had 50 applications there. I had 50 one-offs. They were 98% the same, but there was a one-off for the business. This business needed this and this business needed that. Some were waterfall. Some were just starting to look at agile.

**9:47 Chris Romeo:** Yeah.

**9:48 Glenn Leifheit:** All these kinds of things. And let's face it, even when a development team is going through and making their move to Agile, the reason that from a security perspective we tend not to do as well as, you know, we tend to really struggle when teams make adaptations like that.

**10:08 Chris Romeo:** Yeah.

**10:09 Glenn Leifheit:** One of the things I did was I actually went and sat with several development teams and just said, Let me just sit in on your scrum meetings. I just want to understand what's going on in your group. What are the dynamics? A few teams let me in. One of the things I noticed is it's not that we're not laying a strong foundation. It's that when you lay a foundation on something that's going through an earthquake, it's going to bounce up and down. The reality is they're trying to figure it out.

**10:33 Chris Romeo:** Yeah.

**10:34 Glenn Leifheit:** What it's going to be like this time isn't going to be what the next release is, and it's not going to be the next release. It's constantly iterating. And so we set our plan up as if it's something that's solid and we're setting it on concrete. But in reality, it's like setting it almost like on a sea where the waves are going up and down and we're like, hey, why am I feeling seasick on concrete?

**10:54 Chris Romeo:** So what do you recommend then to those teams that need that adaptability? How do they get adaptability that's specific to their product set and their line of business?

**11:03 Glenn Leifheit:** I think one of the things that I really talk about As much as I really love to go technical and really deep, this isn't a technical problem, this is a people problem.

**11:11** Mm-hmm.

**11:12 Glenn Leifheit:** The reality is that we need to really understand who our developers are. And the way I think about it that way is we no longer talk in terms of developers at Microsoft. We talk in terms of customers. This has to be something that they have to want to come to us to purchase from us in all intents and purposes, to be able to subscribe to the surfaces, to be able to do this. So, that means that— does that mean that we have to work with them and maybe adapt it so that we understand in advance what they're going to do.

**11:44 Chris Romeo:** So the developer— so I got to make sure I'm clear on this. The developer at Microsoft is your customer, is what you're saying.

**11:51 Glenn Leifheit:** So you're treating them—

**11:52 Chris Romeo:** so instead of treating them as somebody that you're cracking the whip on that has to do this, you're flipping it around and saying, you're my customer now. So now I got to make sure I've got return on investment and that you're satisfied, you're happy. Okay, wow, that's a neat way to approach it.

**12:07 Glenn Leifheit:** I think the real issue— so we A few years ago, 4 years ago now, we started rolling out static code analysis pretty broadly across IT. Rolling it across 18,000 developers takes some time. But when you do that, the thing that we started running into is, well, we were doing well with that. We started selling it kind of as a customer experience kind of thing and stuff like that. But we learned that all of a sudden there was tool fatigue, there's all these other things that people just... they were seeing us too much.

**12:36** Mm-hmm.

**12:37 Glenn Leifheit:** Yeah, they had security fatigue to an extent. And it wasn't that it was security fatigue, it was the fact that again, we were coming to them with things to do, and therefore we were the instigators. The reality is it's their release. They need to come and get— they need something to get through that. And the— as we look through it further, we'll I'll talk a little later about where we're kind of trying to go. And I think that's, that's really where this will help, but being able to not only have them come to you for— think of it from, I mean, you still need processes that connect and all of these other things, don't get me wrong, but treating them as if they're the customer completely changes your dynamic with them.

**13:20 Chris Romeo:** Yeah.

**13:21 Glenn Leifheit:** Now you're trying to attract them.

**13:22 Chris Romeo:** Mm-hmm.

**13:22 Glenn Leifheit:** You're trying to have them come to you. The reason we want to do that is application A over here, It's something you've been using for a long time and it's been around for years. That team splinters off and starts going to a new app. You don't know this happened. Nobody tells you anything. It's just something, some, some guy in another group got some funding. Hey, cool. Go, go build this kind of thing. Oh, there's an SDL. What, what, what's that? We need to do that. We need those guys to actually want to call us and say, hey, I got this new app. I need to be set up on this and I need to do this and I need to do this and I need to do this. Let's go.

**13:53** Yeah.

**13:54 Glenn Leifheit:** And that's what happens now.

**13:55 Chris Romeo:** So how do you— so I'm fascinated by this. How do you attract— what are you doing that's actually getting those developers or customers to call?

**14:05 Glenn Leifheit:** Yeah, and I think part of that is about relationships. I think one of the things that it all— for a long time we were a very manual shop. We took very strong pride in our code and manual code review and spot checking and doing doing great process historical work at Microsoft. Over the last 4 years, we've really adapted to the fact that things are changing. For instance, when I started 5 years ago, we had roughly 3,000 applications that came through our SDL on an annual basis, and each one had 4 releases a year for waterfall. Now we have about 5,000 applications, and they release anywhere from weekly to daily. Nobody's gonna scale that many humans, and I don't think there's that many in the industry to be able to maintain that. So from that perspective, we had to change, and we kind of got forced into it. And as we started doing that, we really started looking at it, um, from an engagement point of view, and that the development teams need to be truly engaged and invested as part of this. So some of it is about the fact that, you know, We have a— I'm not gonna get— don't get me wrong, we do have an advantage in the fact that we did have the Trusted Computing Medal that came from our company.

**15:21 Chris Romeo:** Yeah.

**15:21 Glenn Leifheit:** We've got some buy-in within the company that does help us, which some companies don't have. And I've battled through that in the last company as well. But I think that from our perspective, one of the things that's changed this relationship is that we really, again, customer-centric focus, the adaption. So, when we rolled out static code analysis, we didn't say, hey, go do this. We said, What's your process? How does it work? How can we fit into your process? I don't want you to learn a new technology. I want you guys just to have it working. If I've got 300 build engineers, I don't want to have to teach all 300 build engineers a new scripting language because they decided to do it. What are you using? Okay, great. Here's how you can do it there. And we roll them into—

**16:01 Chris Romeo:** So you provide the integration as part of that adaptation. You actually, you don't just hand them the tool and say, here, go.

**16:07 Glenn Leifheit:** This was a full onboarding process. It took a long time to do, and we're still going through it with some teams. But the whole idea is the fact that, you know, we basically stood it up as a dual-stream process. So, for instance, the first stream is the build stream. So whether you're a waterfall process that has a real build environment that has to be integrated in, we had a person who specialized in build who helped you guys out.

**16:35** Yeah.

**16:35 Glenn Leifheit:** Help to tweak things if needed and stuff like that. And then if you're really agile and we're in maybe Visual Studio Team Services or some of those other online services for source repositories and doing continuous integration, we worked with them to be able to help make sure that what they were doing was everything. And then we worked with our vendors to be able to help saying, hey, we have these needs. How can we help meet them faster? And there's some really good stuff coming down the line that we've been helping our vendors work on that is helping us, but we're also trying to change it so that we're helping the industry. So everything that we're asking for from that perspective, we are asking for as a product from that perspective. And we don't actually— one of the other things is we don't actually do custom rules at Microsoft, at least internally. One of the things that we do is we actually turn those rules over to our vendor and they actually roll them into the—

**17:30 Chris Romeo:** Okay.

**17:31 Glenn Leifheit:** overall for the industry. So everybody gets to share in those vulnerabilities that we may be looking for.

**17:36 Chris Romeo:** So based on this model that you just described here, I know many people are going to be wondering, because we know that there's some small security organizations where they have one person and they— I hear this often and I heard this in past jobs. They'll say, well, you're Microsoft and you have 50,000 people to do this. I'm one person, 10 people, whatever I do. So how big is that And just scale, is that group that provides that consultative process, is it one-to-one with a product to—

**18:10 Glenn Leifheit:** Oh, by no means are we anywhere close to one-to-one. So we are— I'll give you 2 things. I used to be the person who was one person. So the last company I was at, I was on a 9-person information security staff. We all did everything. I concentrated on the SDL and PCI compliance. I owned both. For a financial company. For a financial company. So I know the workload that's behind all of that and so forth. We do do things at a different scale at Microsoft, but there are tons of learnings that come from— we're still way outnumbered from that perspective. Once in a while we do things very different. When I used to go do training, I'd throw some PowerPoint slides together and do some stuff. Now I go to a recording studio and actually record training in a in a studio with a producer. Still boggles my mind, but because I'm still so used to doing it the other way. But the reality is having, you know, that's an advantage that I have there. But at the same time, that doesn't mean that the other training isn't valid. It's very valid and very easy to do. It's not maybe quite as scalable, but you know, you can, you can record PowerPoints, you can record those things. You just have to think about how you're implementing things in a customer-centric way.

**19:20 Chris Romeo:** Yeah.

**19:21 Glenn Leifheit:** I think that's one of the things that historically, one of the things we've done is, hey, I built this PowerPoint and I've got this great training that we do, and I'm going to make sure that everybody needs to spend 6 hours with me to be able to get through training. One of the things we learned at Microsoft is, well, we started with a half day of training, or actually it was 8 hours originally. We condensed a 2-day training into 8 hours. And then we realized, well, development teams aren't willing to spend that much time. So it didn't matter. And then we moved it to half a day. Now we're at 2.5 hours. And it's not that we're training them on less. We take key points that we need to drive home. There's plenty more that they need to learn, but keeping it in bite-sized chunks that, you know, if I say I need all your developers for a day, now you've taken 20 resources and you've affected a sprint at that point. When I say, hey, I need your resources for a 2.5-hour meeting, I gotta waste a 2.5-hour meeting every week. I can, you know, I can deal with that.

**20:18 Chris Romeo:** Yeah.

**20:18 Glenn Leifheit:** And this is gonna— this is gonna help me get through that STL. That's fine. And then so we continually go back, we work with the groups that work the relationships with those teams and make sure that we have things scheduled. So maybe we have one now and we have one in the future, and then we have one a month later, and we keep going back and building— again, building that relationship and building credit. Because the reality is In a way, I kind of look at it as political capital. One of the things that we look at, what we've been doing at Microsoft, is building political capital with all these development teams, with all these other groups, you know, and being able to say, okay, they trust us now. They've been— we've been able to build that bond so that when we have to go out on a limb, there's trust there.

**21:00 Chris Romeo:** Yeah, I've seen that as well.

**21:01 Glenn Leifheit:** And we— the problem at a security level, one of the things that I fear for most people is we try to use that capital way before we've earned it.

**21:11 Chris Romeo:** Yep.

**21:11 Glenn Leifheit:** I mean, we're way in the negative when we're doing it most of the time, and that's really painful for each group, and that's one of the reasons why we have trouble with teams and we don't get along. Again, that goes back to that whole customer relationship point of view. If you continue to drive it from that perspective, then you continue to address people in what would be a quality way. The way I look at it is First, you have to get in the ballpark. So, you know, you're the developer, they're the coworker, then they become your customer, then eventually they become your partner. And as you work up that stream, the partners are the ones who are calling you left and right. Your customers are the ones who will reliably come to you on a regular basis and say, hey, okay, we've got this release, or hey, we've had some business change. We just dumped a crapload of money in this app. And we're gonna do all these new features. And we're gonna say, okay, well, which ones matter from a security standpoint? Which ones are just small? And we'll adapt to that and figure it out. And that's one of the things we've been a very rapid response. We've moved to a much more rapid response model. So, we are, one of the things we did is we drove, for the last 4 years, we've been really trying to drive upstream. Part of that was implementing static code analysis to be doing that. But from a design perspective, we're also upstream. Having those conversations with the So, the goal for us is to be embedded within the planning meetings. Because in the planning meetings, at Microsoft anyway, the planning meetings are where you're going to have not just the managers there, but the product groups there. And they're having that conversation of, hey, I've got this idea, and it's this application. And it's the beginning of how are you embedded at the beginning. You're not always going to be able to get in those, which is why you need those partners to tell you.

**22:54 Chris Romeo:** Mm-hmm.

**22:54 Glenn Leifheit:** But in the meantime, You need to be upfront and say, okay, maybe I don't get to know about the next new app, but I do get to know, hey, we just funded a— you know, they're the ones who are talking about how much money they funded. Hey, I funded this thing. It is going to do all this. Fantastic. Well, what's all of this? And can I take a look? Let's see. Let's do it securely from the start so you can get through without a problem.

**23:18 Chris Romeo:** Yeah.

**23:19 Glenn Leifheit:** We want them to feel enabled to be able to get through the SDL. So fast that they, that, you know, it's not an issue. It's just part of their process, and they don't even think of it as something separate. We've spent way too much time being separate and not together.

**23:35 Chris Romeo:** Yeah, so let's— so there were probably hundreds of nuggets of information in there for people that are trying to do something, those that are new in the SDL world. There's many different things that you can learn and take away from there. I'm interested in your thoughts about the evolution side. So this was a— we kind of brought— we transitioned our listeners from where we've been into some good best practices for how things are happening today. Now, what's your recommendation for the future?

**24:04 Glenn Leifheit:** Okay, so just to take a step back real quick, you know, one of the things that I think about is the evolution of other things that we do. I'm very strong, I have a very strong belief in outside influence. You know, from a development perspective, they take in some outside influence. From the security perspective, we really never, never really done outside influence. And I usually step way outside the box compared to most folks when it comes to outside influence. In my talk, I actually did the comparison between music and how the music from the 1950s to today while all based on the same things, are drastically different from a language perspective, from a tone perspective, from a speed perspective, from a loudness perspective, from a how you play the instruments perspective. They've all drastically changed over the last 60, 70 years. And how did that occur? How many different influences? And I put up a chart. I actually went through and did some research and grabbed as many as would fit on a screen. There were probably 40 additional influences that I didn't get on the slide because it wouldn't actually fit on a PowerPoint, it was so big. But the reality was is that the music industry has had tons of influences from other countries, from other— actually even other industries that have actually influenced how things happened. You know, the invention of certain instruments and how things are played, how, you know, the fact that synthesizers can now do— and keyboards can now do things, the advent of technology into music. All of those things have drastically changed the landscape. Yeah, within business, things have drastically changed the business over the last 10, 15 years. But what's changed in the SDL? Not a whole lot. You know, we've adapted a little bit, we've tried to be agile, we've adapted for that perspective. So when I start looking at the future from a customer-centric point of view, what do they really need? And that's where we've really started talking about perceived zero touch. And I always put— we talk about zero-touch, but I add perceived in there because really it isn't that we're just, hey, go ahead, have at it, you know, no security for anyone. It's actually the exact opposite. Security for everyone. Here it is. It's enabling teams to be able to have everything they need when they need it to be able to do their job so that you're not a roadblock. It's just there and you're able to work forward.

**26:32 Chris Romeo:** Yeah.

**26:34 Glenn Leifheit:** And that can be done in the smallest teams. The reality is, if you're going to do some type of static code analysis, whether that be through a free tool or whether that be through something, you can automate that. You can make sure that you understand when the release dates are. You can make sure that you understand how long it takes them to usually get something into their backlog and to fix things. So, you can work with them to say, okay, Fantastic. We want to get you into some type of rhythm that's going to make it, you know, as little impact as possible because that's really what they want. We want to reduce the security friction to enable them with security. And by doing that, you know, if I go through and say, hey, you know, how many releases you got? We're going to have 4 releases in the next 2 months. Okay, great. So, that means that you're going to have, you know, maybe every 2 weeks you're going to have a release. And so how, so how are we going to actually get you the static code analysis results so they get back into your workflow? Because we can complain and say, oh, look at all the security vulnerabilities I found, and I'm gonna put them in remediation and go send it to somebody who— or make them put in a big remediation plan and all of these things, which you need to recognize the risk and you need to do that as part of your process. But again, that doesn't mean that they have to be part of part it.

**27:52** Right.

**27:52 Glenn Leifheit:** of it in that way. They need to know. They need to know the things that are in the backlog so that they can actually fix them. Because a lot of times, one of the things that happens is we have separate systems. Most times people will talk about, hey, yeah, well, I track this in a small team. Hey, we track this, the risks in Excel. Fantastic. It's very viable to do, but the development team doesn't see that Excel document at all.

**28:16** Hmm.

**28:16 Glenn Leifheit:** Their management team doesn't see that document at all. until it's sent around from the CISO who may send it over, or director of security, and he says, here's all the new risk you've got. And he looks at it and goes, where the hell did all that come from? It's not in my backlog. How am I supposed to fix it? So the reality is that's one of the biggest problems we've had is having differential systems. So one of the things we're working to is we're actually working with them to ensure that all the security vulnerabilities are logged And they're within their tracking system. We used to let it be up to them, and they could log it in however it worked for them, which made sense except for the fact that they had to do the work. And now one of the things that we do is we are moving to a situation where, okay, everybody's going to kind of have it logged in a similar fashion. So, and I can't, you know, from a security vulnerability standpoint, the other problem is, hey, if I go through static code analysis for scan, and you get all these results, you can really skew your metrics from a vulnerability point of view. They can't even find the other stuff after you do it. I mean, I did, I was testing an automated process at one point in my career early on, and I said, well, let's just automatically put all these into your source repository. Hooked up the automation, wham, 30,000 vulnerabilities in there, 30,000 new features required.

**29:35 Chris Romeo:** Yeah.

**29:36 Glenn Leifheit:** It basically made their tracking system worthless.

**29:40 Chris Romeo:** 30,000 less friends you had at that company.

**29:42 Glenn Leifheit:** Exactly. It was one of the best learning experiences, but one of the worst things I could have done. So, in hindsight, the way I look at it is, what's a bite-sized chunk and how do these teams work with it? Some teams, one of the things we ask is, how do you work with this? Then log it in that fashion. Do you tend to try to fix vulnerabilities by type? Do you try to fix, you know, I have SQL injection problems, I have cross-site scripting problems. A lot of teams do that because the reality is they're trying to understand what the changes are and they understand, okay, this is how you fix this. Let's just go do it. Well, before we forget how to make all of this work and lose track. Other times, other teams are like, well, we're only in this part of the application, so do it by file. I say, okay, great. Then you can get this whole area solved. and this other group, you know, we'll get to as soon as we can. But it allows them to have the flexibility to go about their way fixing those vulnerabilities.

**30:38 Chris Romeo:** So this was under the heading of perceived zero-touch.

**30:42 Glenn Leifheit:** Right.

**30:43 Chris Romeo:** What else are you thinking about from an evolutionary perspective?

**30:46 Glenn Leifheit:** So from an evolutionary perspective, part of it I look at almost as a disruption. So, and it's It's kind of weird to think about, but it actually came out of reading— again, outside influencer— it came from reading a book that I was reading called Disrupt Yourself by Whitney Johnson.

**31:06 Chris Romeo:** Mm-hmm.

**31:06 Glenn Leifheit:** One of the things they talk about is taking a step back and not just plowing forward with what you're saying. It's kind of a career-based book, but it has a lot of applicability all over the place in terms of We tend to get stuck on this curve and we go through on maybe an S-curve and think, okay, well, this is the right spot, but eventually you get to the top and you got to figure it out. So one of the problems with security is we think we're on this upward S-curve, but we're kind of on this flat line.

**31:36 Chris Romeo:** Mm-hmm.

**31:36 Glenn Leifheit:** You know, from our heads, we're seeing we're being productive, we're doing these things, but we're not really disrupting anything and we're not really being as successful as we should be from that perspective. So the way I looked at it is, you know, being able to, you know, shake things up, bring in outside influence, bring on— bring in the impacts of the business, bring in these other things that you have surrounding you. Um, one of the best ways to learn how to do the SDL is to have scarcity of something as you're doing it. The reason I know as much about the SDL as I do from my early career is the fact that I implemented it at implemented it at FICO. I tell this story a lot. I went in and said, you know, we've got to do this. I got a seat at the table. I went in and saw the CTO, my boss, and his boss, and we had this conversation and everything was going great. You know, she's buying in. All this makes sense. Then she looks at me. She says, so what is it going to take to do this? I lay out the plan with the resources and the people, and she says, I trust in you. Well, thank you, but— No, I trust in you. So, the 4 people I wanted? You. Okay, I get the picture, but do it. And the reality is I had to figure out how, as an individual, to roll this across 50 applications that were sold box products to customers. that I had a very rapid— I won't talk about what the turnaround time was, it was fairly insane. And we had to be able to do it. And how do you implement something and grow a process that's really difficult without thinking way outside the box and thinking of scarcity and how that works? What was my scarcity? I didn't have resources from a people perspective other than my own hours, which I abused my own hours way too much. But the other piece of it was just being able to take on these other things and using that ideology behind the scarcity piece, taking that and then being able to stretch that into allowing for zero-touch. The idea of, well, I know I can't get them to do this, but what can you do with the resources you have Or some other resources you have that you don't realize you have. One of the ways I did it at FICO was I actually, I was having trouble getting development buy-off, obviously, because it was just me. I had to work with 50 other teams. I just couldn't scale anyway as it was. But in conversation, I found a QA guy who was really interested in security. So we had some conversations. He's like, yeah, you know, you could do some of these things. So I taught him how to do some fuzzing in his applications. and stuff like that. Lo and behold, he started kicking those bugs back to developers. He's from QA. Of course he can kick anything back. And so he's throwing vulnerabilities back and vulnerabilities back. And about 3 releases later, I'm like, I go approach that team. I said, you know, I think it's really time you started working on the SDLC. He's like, well, why do we need to do that? I'm like, well, you're already fixing vulnerabilities. Why don't we do it in an organized fashion so we can speed you along? Now it became a, I'm now servicing them.

**34:52 Chris Romeo:** Okay. That customer-centric model.

**34:54 Glenn Leifheit:** It's very customer-centric. I'm able to service them in a way that's going to help them. I'm going to help their process and be able to enable that as part of it. And again, that all comes as part of that perceived zero-touch.

**35:08 Chris Romeo:** And I've seen that connection as well with where you start to build a security community. I think that's such a powerful thing, and that's an example of what you did here is you had that QA person who became the first member of the security community, you were able to pour into him and then he influenced 5 or 10 other people. So that's what the model that I've gone after as well in my career.

**35:30 Glenn Leifheit:** And one of the ways we actually even focused on it is the idea of kind of more community-oriented is we occasionally hold meetings with the developers just saying, you know, kind of a Q&A kind of thing. What do you want to know? How can we help? Those types of things. And it really— and when we do do those, They turned into almost like user groups. People want to come, people come and see it. And, you know, maybe I give a 10 or 15-minute spiel and say, hey, this is what we're doing right now and stuff like that. But it's very reminiscent of going to the .NET users groups I used to go to in Minnesota when I'd go there and work a lot in the community. And, or it would be very similar to, you know, maybe the Java users group. And they, cause they go in and talk about a subject and it became something that developers wanted to go to.

**36:13** Yeah.

**36:13 Glenn Leifheit:** because they wanted to advance themselves in their career. And that shouldn't be lost on a security professional that these are, these are real career opportunities for the developers as well. You don't necessarily want to stage it because, you know, no company wants to say, hey, you can go get a great job somewhere else because you, because you can get security. But let's face it, a secure developer who can code securely is worth their weight.

**36:36 Chris Romeo:** Yeah.

**36:37 Glenn Leifheit:** And, and, you know, they can be at a pretty good scale. So from that perspective, you know, behind the scenes, it's kind of one of those undertakings that in the end, we're really trying to help your career too. Yeah, as part of that, and they need to understand that.

**36:50 Chris Romeo:** I think we're going to see much more of that in the years to come here, of security and development coming together from the people that really get security and build secure stuff and can prove that they know how to do it. They're going to start to filter. They already are, but there's just not as much in the industry awareness about these people are really—

**37:08 Glenn Leifheit:** these are the ones. And they tend to filter to certain industries right now. You know, they'll filter the financial industry because they've got certain requirements. They'll filter to other places because they have certain— like Microsoft, because we have certain requirements as a company. And so we'll hire those folks. But the— I think that the interesting thing is, as we go through it, is inherently it's someplace that we need to promote the industry to, um, in the grand scheme of things. So that's one of the reasons why I also really advocate for going after the new developer, working with the teams, the folks who are right out of school, the folks who maybe have 1, 2 years of experience that you're working with, who are your coworkers. From a security perspective, go talk with them. They're open. They're trying to adapt to this new world. They're the ones who are coming to their coworkers saying, so how does this health insurance thing work? And dental? What do we do with that?

**37:59 Chris Romeo:** Right.

**37:59 Glenn Leifheit:** I don't understand why it's separate and how does that work? I have 3 millennials that work across the hall. I hate using that term, but it helps for identification purposes.

**38:08 Chris Romeo:** Sure.

**38:08 Glenn Leifheit:** But they do, you know, we have a lot of those conversations, but all those conversations end up leading to career conversations and to security conversations and all these other things that they need to know because those little conversations that are very non-business focused are the gateway into building the relationship with them to be able to take that next step and be able to really do things.

**38:33 Chris Romeo:** That's a great reminder for everybody listening to this. If you're in the security industry now, if you can't name the 5 people that you're mentoring and influencing, then you need to go find them because they're out there and they need people. And that's a great example of what you're doing here. You're not really selling them on security. You're not trying to convince them. You're just making them aware of here is a way you could go in your career. Right.

**38:56 Glenn Leifheit:** And from a mentorship perspective, I really look at really 3 areas. It's another one of my pet topics. So I apologize for jumping off here, but I think from the first perspective, you always have the new folks who are newer in the industry or newer at your company, and you can affect them well because it's easier to do. Then there's the peer mentors. One of the things we don't do well enough in the industry is actually mentor each other. You know, there's a lot of coworkers who have really good skills in one area, but they could really learn from another. And that goes from a development perspective and a security perspective. I think that the— then the other piece that we've actually done is stretching, or, you know, from a— and this idea actually came out of when I relocated to come to Microsoft. And one of the things that I sorely needed was relocation mentoring.

**39:44 Chris Romeo:** Mm-hmm.

**39:44 Glenn Leifheit:** Because there's all this stuff that's happening around around me. I'm like, heck, I don't even know where the heck to get my hair cut, and you're yelling at me about, you know, how to get this other stuff done. I got so much on my mind. But from that perspective, cross-boundary. So, security mentoring a developer, developer mentoring a security guy, those types of mentorship opportunities that really are there if you just ask. I think that we're too afraid to ask because it feels like it's a weakness. mentorship is a strength on both sides. Being mentored is a strength. I have 4 mentors right now. I mentor 6 people. You know, I spend way too much time mentoring sometimes, but it's one of the best things I've done for my career.

**40:26 Chris Romeo:** It's one of the most rewarding things too.

**40:28 Glenn Leifheit:** It really is.

**40:29 Chris Romeo:** And I'm the same way. I have multiple mentors, people that I reach out to even though I've been in the security business for 20 years. I still have people that I need to bounce things off from and say, what do you think about this?

**40:39 Glenn Leifheit:** I think I— and just like everything with my thoughts on outside influence is have varied mentors. I think that for yourself especially, as you look at it, because you're responsible for finding those mentors in reality. And so I've got some internal Microsoft mentors that are really good for understanding the politics of Microsoft and understanding security within Microsoft, or really in depth in security, or how do I get to that next level. But I also have mentors that are outside of Microsoft. And part of that is about how about some are industry mentors that I just have occasional conversations with. And others are people who are completely outside the industry that can give me completely unfettered feedback on what I'm talking about and give me the ability to recognize, am I way off base? I have actually, one of my mentors actually came from, I participate in Toastmasters a lot.

**41:33 Chris Romeo:** Okay.

**41:33 Glenn Leifheit:** And have been doing that for, about 10 years. And I've got one of— I met one of my mentors through that. He's actually someone who is building a career as a life coach and these other things. And so he's able to bring a whole different thing to the table that a security mentor is never going to bring from that perspective. And I think to be better at what we're doing, we need that other influence and that other idea. So I encourage you to look out within the industry, outside of the company, as well as outside your field.

**42:03 Chris Romeo:** Yeah, because those people can provide you a sounding board. We forget that many of us that work in security, we all speak the same language. You can start talking about something and we're already with you. When you take an idea to somebody who's outside of the security industry, they don't have all the history that goes with it, so they can give you a fresh look, fresh perspective, and they might just look at you and say, that's the dumbest thing I've ever heard in my life because I don't understand what you're talking about.

**42:32 Glenn Leifheit:** But dealing with, STL is a great example of something, it's a process. Anybody who's dealt with processes at any company can tell you things that work, things that don't work.

**42:41 Chris Romeo:** Sure.

**42:42 Glenn Leifheit:** Things they've failed at, and things they've adapted. And those are things, learnings that we need to have. When I first started working on the STL, I really concentrated on outside influence because we were a rapidly changing organization. I had products that were changing dramatically. And I'm like, how can I figure out how to adapt this monolithic SDL that we want to do? And that was part of the problem, is it was monolithic and we had to go step by step and not give them everything at once.

**43:10 Chris Romeo:** Yep.

**43:10 Glenn Leifheit:** Um, but the other piece was, what are the right pieces and how can I get the best bang for the buck? So I actually reached out and talked to a local school district, local fire department, and a local police chief, and reached out to them because they were in a community that had grown from 2,000 to 100,000 over a 15-year period. It's dramatic change.

**43:30** Hmm.

**43:30 Glenn Leifheit:** And the tax dollars come way the heck after. So they're doing it on a shoestring beyond a shoestring. You know, that thing's so thin at times, it's, it's really scary. So I talked a lot to them about how did they adapt? How did they protect the citizens? You know, fire departments, police departments, those are pretty damn important. And how were they, how were they able to scale when they had scarcity of resources and other things? And I wasn't looking at it as, you know, sometimes security and police, well, that lines up. But it isn't about lining up. It was about how did they deal with situations. They had a good example of it is one of them, they had a road that people were just flying down. Speed limit was 45. Average speed was like 70 going down this road. Everybody was just flying by it. And they're like, well, I can't afford— they can't afford to put speed bumps in.

**44:20** Yeah.

**44:20 Glenn Leifheit:** They can't afford to patrol it. There's very little they could really do. But what they did do, did have, was a couple of broken-down cop cars. In the middle of the night, they towed the cop cars, which didn't have engines that worked, plopped some radar detectors in them, and lo and behold, the average speed on that road went down 10 miles an hour because that was sitting there and there was that threat and view that Okay, they're here. It didn't even mean they're going to pull you over. They never got pulled anybody over. They never— nobody ever saw any influence. But what, but what it did do is reminded people, hey, maybe I shouldn't be going so fast. And people slowed down. So it's all these little, little things that we're able to do that we can take from other industries that can do that. Now, you know, how'd that portray— that taught me a little bit about, you know, I recognize that, that I can't do all the entire SDL at once, but as I filter things in, it can start really small. It doesn't have to be huge. When I talked about— when I was at FICO, one of the things we talked about was the threat model. How the heck, from a threat model perspective, I'm like, hey, you owe me a threat model for the whole application. That doesn't work. They had 12 years worth of that application. They're not gonna threat model it in a day. We'll get back to you in a week. You know, a year, maybe stretching it. So what I started with was, you know, just an Excel document. Just list your risks, list the riskiest spots in your application. Let's start with that, and then let's grow it from there. And really starting with core basics, because the reality is, is that while it's fine, and Danny, that Microsoft, and we have that, the threat modeling tool and things like that, Yep. but you have to bite off the pieces that work within your organization. And what works for one organization is going to be different for another. And, you know, I think that the other piece that I thought about is the more you do that, you really want to make sure that you're building something that can last beyond the person. I think that's one of the real keys is so many processes die because somebody left.

**46:26 Chris Romeo:** Yeah, it was all tied to so-and-so.

**46:28 Glenn Leifheit:** Yeah, and as soon as they leave, things go. And when you're in a small company, that's even worse. Because the likelihood of that, you know, just by percentages is higher.

**46:36 Chris Romeo:** Yeah.

**46:36 Glenn Leifheit:** And it's just going to happen and you're not going to have backups and think of people and stuff like that. So building something that, that's simple enough for them to do that is really sustainable so that even if somebody had to push come to shove and work a ton of hours because they had to cover because they're shorthanded, they still could at least implement that process. It's not so specialty.

**46:57 Chris Romeo:** Yeah.

**46:59 Glenn Leifheit:** it's unimplementable because, hey, George just did this thing in the corner and we're not really sure what it was. You need to be able to share that knowledge as part of that.

**47:07 Chris Romeo:** So just to kind of wrap this up, if you had to give one call to action based on what we've talked about today, what would you recommend that our listeners go and what should they do and take away from this?

**47:19 Glenn Leifheit:** So I think there's a couple of things that I really pinpoint that kind of combine. So I think that the first is recognize that from a security perspective or from a development perspective, we need to work in partnership, but we also need to adapt as close to the same rate as we can. If you're going to go to DevOps and all of a sudden have all these developers doing operational stuff, well, that's part of life. It's starting to happen now.

**47:47 Chris Romeo:** Yep.

**47:48 Glenn Leifheit:** The separation of duty stuff is kind of falling by the wayside, but at the same time, We need to be there to help make sure that those folks know how to do things securely and are doing things securely.

**47:59 Chris Romeo:** Yep.

**48:00 Glenn Leifheit:** So adapting to the processes that are happening around you, using outside influence to help you come up with ideas, using the constraints to help bolster your thoughts on things. I can't do this, but I can do this, or maybe this other group can help me. Time and time again, I've seen that happen at Microsoft and other places. And I think the other piece is really just be open-minded and teach your— keep your— treat your developers or your security, everybody as a customer. I think that the real key change that we did was that, and changing it, you know, from a vulnerability standpoint, we don't even talk about vulnerabilities anymore. We talk about what you should do, not what you did wrong.

**48:44** Okay.

**48:45 Glenn Leifheit:** Keep it positive. Keep it a customer focus.

**48:48** Yep.

**48:48 Glenn Leifheit:** And if you can do all of that, it's amazing what can happen.

**48:51** Yeah.

**48:51 Chris Romeo:** Hey, Glenn, thanks for your time here and all of your wisdom. We look forward to sharing this with our listeners.

**48:57 Glenn Leifheit:** Thanks, Mitch.

**49:00** Thanks for listening to the Application Security Podcast. Our intro music is Eight Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Kartenberg. You can You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.

**49:16 Chris Romeo:** Thank you.

---

Source: https://appsecpodcast.com/glenn-leifheit-an-inner-glimpse-of-the-microsoft-sdl/