---
title: "Steve Lipner — The Past, Present, and Future of SDL"
url: https://appsecpodcast.com/steve-lipner-the-past-present-and-future-of-sdl/
date: 2019-12-20
duration_seconds: 2032
guests: ["Steve Lipner"]
topics: ["Threat Modeling", "Secure Development", "Security Testing"]
audio: https://www.buzzsprout.com/1730684/episodes/8122620-steve-lipner-the-past-present-and-future-of-sdl.mp3
transcript: true
---

# Steve Lipner — The Past, Present, and Future of SDL

*December 20, 2019 · 34 min*

with [Steve Lipner](https://appsecpodcast.com/guests/steve-lipner/)

on [Threat Modeling](https://appsecpodcast.com/topics/threat-modeling/), [Secure Development](https://appsecpodcast.com/topics/secure-development/), [Security Testing](https://appsecpodcast.com/topics/security-testing/)

[Audio](https://www.buzzsprout.com/1730684/episodes/8122620-steve-lipner-the-past-present-and-future-of-sdl.mp3)

## Show notes

How did Microsoft's Security Development Lifecycle become a repeatable engineering practice rather than a one-time security push? Steve Lipner joins Chris and Robert to trace that history from early computer security work through security response, Trustworthy Computing, and the formalization of SDL. He explains why reviewing software at the end cannot scale, how threat modeling and testing became development activities, and what organizations learned as tooling and expectations evolved. The discussion connects that history to teams starting their own programs: establish a way to receive and respond to security reports, use the protections already available in development tools, and build from there. Steve closes with guidance for mature programs that need to keep learning instead of treating yesterday's process as finished.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
Security Journey provides application security education for developers and everyone in the software development lifecycle.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Steve Lipner:  
→ [Steve's website](https://www.stevelipner.org/)  
→ [SAFECode](https://safecode.org/)

Mentioned in this episode:  
→ [Microsoft Security Development Lifecycle](https://www.microsoft.com/en-us/securityengineering/sdl)  
→ [Microsoft Security Response Center](https://www.microsoft.com/en-us/msrc)

Chapters:  
00:00 Introduction  
02:36 Early computer security at MITRE  
04:32 A career spanning security products and practice  
06:28 Joining Microsoft security response  
08:40 Trustworthy Computing and the security push  
14:29 Defining and sharing SDL  
16:46 Why end-of-cycle audits do not scale  
18:32 Formalizing the lifecycle in 2004  
21:12 Threat modeling, tools, and security testing  
25:12 How secure development keeps evolving  
28:35 Advice for starting a security program  
32:12 What mature programs should remember

## Transcript

*4,847 words · assemblyai*

**0:00 Chris Romeo:** Steve Lipner is a pioneer in cybersecurity, approaching 50 years' experience. He retired in 2015 from Microsoft, where he was the creator and longtime leader of Microsoft's Security Development Lifecycle, or SDL, team. While at Microsoft, Steve also created a number of initiatives to encourage industry adoption of secure development practices in the SDL and served as a member and chair of the Safe Code Board. Steve joins us to talk about all things SDL, and I must say, I was super excited for this interview with way too many questions for some someone who was there on day one of secure development lifecycle. We hope you enjoy this conversation with Steve Lipner. I want to take a moment to introduce you to Security Journey. At Security Journey, we believe security is every developer's job. We work with our customers to help them build long-term sustainable security culture amongst all their developers. Our approach is to provide security education that is conversational, quick, Hands-on and fun.

**0:59 Robert Hurlbut:** We don't do lectures.

**1:01 Chris Romeo:** Instead, we let the experts talk about what's important. The modules are quick, 10 to 20 minutes in length. We believe in hands-on experiments, builder and breaker style, that allow developers to put what they learned into action. And lastly, fun. Training doesn't have to be boring. We make it engaging and fun for the developers. Visit www.securityjourney.com to sign up for a free trial of the Security Dojo.

**1:35 Robert Hurlbut:** Hey folks, welcome to this episode of the Application Security Podcast. This is Chris Romeo, CEO of Security Journey and co-host of the podcast. I am joined by Robert. Robert, how is it going today? Going well, Chris, thank you. And this is Robert Hurlbut, Threat Modeling Architect. Robert, are you buried underneath snow at the moment or— Not, not We've had some, but we've dug out and we're okay. Digging out. I live in a place, a part of the world where, or part of the United States that requires almost no digging out. So I'm away from snow. Well, super excited today to introduce our guest who is Steve Lipner, who is someone that I have been following throughout the world of secure development lifecycle for seems like at least a decade, more than a decade at this point. And so I'm so excited to have Steve with us today. And Steve, we're just gonna jump right in and Our listeners love that we just ask this question right off the bat. What is your security origin story? How did you get started in security?

**2:36 Steve Lipner:** I started my career after college, after graduate school, as an engineer with the MITRE Corporation, which is a defense contractor that works out of the Boston area. And after about a year there and one sort of successful project assignment, My boss asked me to take responsibility for a new project to look at how we would achieve multi-level security, that is, protecting classified information in shared computer systems for the Defense Department. And I protested. My degree is in civil engineering, and I protested that I did, you know, I knew something about computers and software applications. But I protested that I didn't have enough background in the operating systems or other things that would be necessary to, you know, to really help lead a project looking at multilevel security. My boss replied that, well, why don't I do what I could with that assignment until they found the right person, and then they'd let me go on to something that I felt like was more in line with my experience and abilities. Of course, they never, never found the right person and probably never had any intention to. So, so I kept, I kept working in security. I should, I should add that that exchange about doing it until they found the right person took place in 1970. So next year will be my 50th anniversary working in, in what we now call cybersecurity. And in fact, a bit of a shameless plug, I have a session at RSA next February talking about the mistakes I've made over 50 years working in cybersecurity.

**4:32 Robert Hurlbut:** I'll definitely be in the audience to hear that. And wow, that is such a cool story to say that, hey, you've been doing this for almost 50 years at this point. And so, multi-level security, government space, what's kind of the next step that you took. How long did you stay at MITRE before you moved on to something else?

**4:54 Steve Lipner:** Well, I was at MITRE for about 12 years, and then I was at Digital Equipment Corporation, if people remember that, merged and merged again, and some of the products survived. I was at Digital Equipment Corporation for 11 years, and I ran a group there that was trying to build a high-assurance Orange Book, you know, continuing with the defense security assurance theme, trying to build a high-assurance A1 timesharing system through the '80s. And that, that was an interesting exercise at the time. We wound up canceling it rather than shipping it because nobody wanted a system that secure. Ran the firewall business at a little company called Trusted Information Systems in the '90s. And went out to join Microsoft in 1999. Skipped a couple of other things along the way, but those are sort of the highlights.

**5:52 Robert Hurlbut:** To hear that list of companies though, it's bringing back a lot of memories for me as far as like, I guess the first big Unix system I ever worked on was a DEC Alpha. And yeah, I remember TIS as well. TIS around Maryland somewhere, right?

**6:11 Steve Lipner:** Glenwood, Maryland, halfway between Baltimore and Washington, pretty much in the middle of nowhere.

**6:17 Robert Hurlbut:** Now it's a metropolis, but maybe back in those days it was—

**6:23 Steve Lipner:** Less nowhere than it was 25 years ago.

**6:28 Robert Hurlbut:** And then in your time at Microsoft, I think you did a number of different things, right? What were kind of some of the big things you did for security at Microsoft?

**6:39 Steve Lipner:** So I was hired at Microsoft actually to lead security response, the function that deals with externally discovered vulnerabilities in Microsoft products, what's now called the MSRC, the Microsoft Security Response Center. I had been working for another defense contractor in the Washington area, found that job was somewhat boring, and so Microsoft offered me the security response job. And I said, well, it would be, it would be interesting and it would be working on something that was important to a lot of people. So I went out to Redmond at the end of '99, ran the Security Response Center for almost 4 years. But a couple of years into that, I also got a new position that included responsibility for what we were trying to do to improve the security of Windows, basically, instead of just issuing security updates when people found vulnerabilities, actually to try to do what we could to reduce the number of vulnerabilities. That job led through a couple of steps that we can get to in a minute to the creation of the SDL. And I was really associated with the SDL from its creation until I retired in 2015. I had a couple of other roles at Microsoft too. Back to the government software assurance business, I was involved a lot with common criteria and the ways that governments evaluate products. Worked for a while on the programs where Microsoft provides controlled access to the product source code to governments who want to gain assurance that that the code is as advertised and trustworthy. So a batch of different things over the years.

**8:40 Robert Hurlbut:** So you're at Microsoft then when the Gates— what we know of, what we think of as the Gates Security Memo came out. And so were you— so you were already kind of focused on security. So you were in that organization that that memo was then saying to the rest of the company, hey, look out, we're taking security seriously here and we're going to do some stuff about it.

**9:01 Steve Lipner:** So there were actually 2 things that went on in parallel. The trustworthy computing email came out of the chief technology officer's office under Craig Mundie, with Howard Schmidt as his principal security guy. Howard passed away a couple of years ago, you know, real pioneer in security and had a lot of very significant roles over the years. And so Howard and Craig were working to try to get a corporate commitment to security. I was in the Windows division and we were trying to figure out how to make Windows more secure. And this was in 2001, you know, Code Red, Nimda, something called the UPnP vulnerability in Windows XP. And it was just, you know, sort of hit after hit and embarrassment after embarrassment, and Gartner sort of urging people to reconsider whether they wanted to rely on Windows or not. And so we, we had a meeting of my, my management staff at a little conference center in Bellevue, Washington. We were talking, you know, about, you know, all the problems that were besetting us. And how could we sort of get ahead of the situation? And there had been an effort by the .NET Common Language Runtime, the .NET Framework team, to do a better job of security on their version 1 release. Came time to ship, they still found that they were having a number of vulnerabilities that they didn't like. So what they did was basically to take their entire team and stop work and delay ship and say, okay, we're going to, you know, take everybody and put them onto security review and security testing until the vulnerability discovery rate goes down to an acceptable level. Michael Howard, who was involved in putting that effort together, was talking about it, and I said, I wonder if we could do that with Windows. And so we talked about it, and then I went to my boss and said, hey, you know, I think what we need to do is take everybody off development of the then ongoing Windows release, which was what became Windows Server 2003, take everybody offline, train them, and have them review the code for an extended period until we get to the point where we've actually gotten the security situation in hand. And the initial reaction was, you know, it won't work. We don't have enough preparation. You know, the developers won't take it seriously. They'll roll their eyes and then go back to doing whatever they were doing, and it won't work. But my boss also said, you know, well, I have some questions. You know, how would you do this? How would you do that? Came back to him with some answers. He had some more questions. The next meeting where he asked me to come back was with him and the vice president who ran the Windows Server software development. Then the next meeting was with those folks and the senior vice president who was responsible for the Windows Core system. The following meeting, was with the group vice president who was basically all the enterprise and server products as well as Windows client. And by that last meeting, the tenor of the discussion had changed from here are all the things wrong with your dumb idea to what resources do you need and when do you want me to preempt the Microsoft conference center so that we can train all our Windows developers. And that business with what we called the Windows Security Push, the planning for that really paralleled the creation and agreement to the trustworthy computing email so that both basically happened together. Bill sent the trustworthy computing email to all employees and it went public. And one of the folks at Gartner said, well, you know, Microsoft has these terrible security problems and they've released— they've sent out a press release. But, you know, at about the same time, it became evident that we were stopping development on Windows and training 8,500 developers in how to actually improve the code and committing them to to what we thought was a month of code reviews and testing and analysis to do that. And so, you know, those 2 things, I think without trustworthy computing, we couldn't have done the security push. And without the security push, trustworthy computing would have, you know, been interpreted as just a press release. So it was really sort of a confluence of positive things that happened together that enabled us to get some momentum behind actually delivering secure code out of Microsoft. Yeah.

**14:29 Robert Hurlbut:** And when I think about all of the secure development lifecycles from other companies, I mean, everybody points back to what you and the team did at Microsoft. I mean, I was at Cisco for a number of years and the Cisco secure development lifecycle was built with the same components, the same pieces, and even a lot of the advice from the folks at Microsoft in how to do it. And that was one of the really cool things about being part of CSDL was that we were able to have those conversations with our peers at Microsoft. And Microsoft was so open to say, well, here's all the things that we've tried. Here's the things that didn't work. Here's the things that were awesome. And you can try the things that didn't work, but we're just going to share that knowledge with you. And so, that That was a huge kind of development push for me to be able to see, hey, this is how Microsoft has done it. But I wanna back up for a second now. Now that we kind of dive into the SDL side, I wanna start with just asking you for a definition of SDL. Kind of what is your definition? As somebody who really helped to define what this thing is, how do you define it?

**15:42 Steve Lipner:** So I define it as a set of specific activities to build secure software or improve the security of software that are conducted by, executed by a development team and driven by learnings and feedback from the sources of software vulnerabilities. So I think those are sort of the critical things. The fact that secure development or an SDL is driven by actual experience with what results in software that isn't secure. The fact that there are specific requirements or specific activities that make up the SDL, you know, run this tool and fix exactly these bugs, for example. And then the third thing is that the SDL is executed by the developers, not by some outside security team. Those are the 3 things that I think of that that make for an effective SDL process or a real SDL process.

**16:46 Robert Hurlbut:** The execution by the developers is such a big deal because when you're talking about an organization the size of Microsoft or other large organizations that have tens of thousands of developers, we know that there's no way we can get to the point where we have enough security people to try and match up with the number of developers that we have. I mean, say you had 1,000 developers, You'd need, what, probably 100 security people to truly be able to do the work and keep up with them. And so that whole idea of spreading it across, security is for everybody, is such a powerful concept.

**17:24 Steve Lipner:** It depends on the organization and really their culture about how that works. I've talked to organizations trying to create SDLs, and if you have sort of a classic security department or an IT oriented security department. I shouldn't belittle IT, but, you know, a lot of organizations have tended to have their security departments, you know, sort of an audit function, come in after the fact and find out what's wrong and make the people who built it fix it. And that just— that doesn't scale. You can't get enough people. You know, you're always too late because if you start at the end, then you're delaying delivery of capability. So there are just a variety of reasons where the only really viable way to deliver secure software is to have the developers build, design and build secure software. The SafeCode member companies all have processes that really focus on integrating security into development, and that's very much the way, the way it's got to be done.

**18:32 Robert Hurlbut:** I want to approach Now that we have a good definition of the secure development lifecycle, I want to approach this from kind of a past, present, and future perspective. And so, I would love to get your perspective and your take on what's happened in the past with SDL and some of the things that were going through your mind as you were building the Microsoft SDL.

**18:54 Steve Lipner:** So, when we started building the SDL, you know, I sort of set the scene. possibly taken too long with the security push. You know, after we had done the security push and we felt that it was reasonably successful, you know, we had sort of 2 years of managing security pushes with other products and feeling our way. And then actually in 2004, just about a little more than 15 years ago, we decided, okay, we really do need to have a process that continues indefinitely and is a little more formal. And so we basically took the security push and the stuff we had the developers do there, and we structured those things into a set of requirements, and we documented those and said, okay, this is what we expect you to do to all the software that you build going forward. And the way— and that, again, down to specific static analysis bugs, specific tools, specific errors that they had to fix, specific coding constructs that they couldn't use. And so, We rolled out that initial version of SDL in the summer of 2004. And at the same time, we said, okay, you know, we're going to update this over time as we gain experience, discover new classes of vulnerabilities, discover new ways to build more secure software. And those, you know, those updates I think the first one was 6 months after the initial version. Almost from the beginning, we had a transition rule. If we update the SDL and you're about to ship next week, then you get some slack before you have to meet the requirements of the new update. Because that was really important, was the You know, the feedback and continuous improvement piece.

**21:12 Robert Hurlbut:** What are the activities then that exist in that first SDL? Because I think of the SDL of today where we have requirements or user stories, threat modeling, we have static analysis and code review, dynamic analysis, you know, all the things that go into that, what I think of as the SDL of today. But what were the activities in that very early version?

**21:36 Steve Lipner:** The initial version, we did require threat modeling, but that was before we had sort of the modern threat modeling tools and approaches that didn't come out until 2006. So we required threat modeling, but that was, it turned out that was hard for people to do. We required static analysis, and we had specific static analysis requirements for C, C++, and C#. which was really all that Microsoft used back in that era. We had, I think, a general requirement for security testing, you know, but that was before people were widely using fuzz testing for, or dynamic analysis for, as a security tool. And so that didn't, you know, fuzzing, for example, didn't come in until later. We said we would have a security push at the end, You know, just as a revisit to make sure that everything that weren't— that problems didn't remain. And that was sort of the penetration test model. And then we had what we called a final security review, which was focused on making sure that all the requirements in the SDL had been met. It's a sad story. I don't think anybody has a copy of the original SDL, what we called version 2. I don't think anybody at Microsoft has a copy of the original SDL version 2 as it was created at the time. So now if we think about kind of the modern era or the present of where we are right now from an SDL perspective, what have you seen change since that initial version So when I talk about SDL today, I really put much more emphasis on organizations defining requirements based on experience, reported vulnerabilities, and feedback to build continuous improvement. And that's a trend that was sort of implicit in the earliest SDL, but I think it's so important That, you know, we try to— I really try to make that a very clear fundamental for building an SDL for an organization. Other things that have changed today, and these are things that came in at Microsoft over the years and the, you know, the Safe Code members have added over time as well, are, you know, different languages and different frameworks, different programming models, the fuzzing and dynamic analysis. came in fairly shortly, but it wasn't in the initial SDL that we had. And the third thing that is super important is components or third-party code. When we, early on in the creation of the SDL, we said that development teams had to do what we call tracking giblets, tracking little parts that they got from somewhere else, and so that they'd know if there was a a vulnerability found in some component they didn't write, that they could detect it and respond to it appropriately. That's become very important. The 3 things that have changed over time are, you know, the diversity of languages and frameworks, you know, the increased importance of dynamic analysis compared to the very initial version, and the super importance of third-party components.

**25:12 Robert Hurlbut:** When you think about the future, Steve, of SDL, is there anything that you see kind of in your purview that you think we're missing today, and you're hoping like in 10 years from now we'll look back and say, hey, SDLs now have X, Y, and Z?

**25:28 Steve Lipner:** I think the biggest thing, you know, the static analysis tools, I mean, we have them, there are lots of them, you know, certainly more than there were even 5, 6 years ago. Some of them are very good, but the impression I have, and some research suggests this, is that different static analysis tools find different kinds of errors. I think if we really had a better or clearer characterization of which static analysis tools are good for which errors in which languages, that would really help us up our game in terms of doing secure development. The other thing, I think there's been a lot of progress in threat modeling, but there's still, you know, still a way to go maybe there. And then a final thing, you know, people don't come out of school as developers knowing about secure development, and so companies that hire people have to train them on that. I think it would be great if at least some level of introduction to security was included in college computer science software engineering programs so that people would understand the basics that when they're writing software for other people to use, they have to pay attention to security in terms of resistance to attack. Those are probably some things I think might be trends for the future.

**27:32 Robert Hurlbut:** Yeah, and I'd have to agree with you there, hoping that we do get better static analysis tools, that threat modeling is more prevalent even than it is today, and that we do get some education. I've always been curious as to why we teach Java programming, but not secure Java programming by design. It seems like it would be easy when you're teaching someone how to get something from a database in a web context that you just explain SQL injection at that moment.

**28:03 Steve Lipner:** I mean, I think the reality to that is that the people who are doing the teaching, people who are writing the textbooks, They haven't been exposed to security or the need to write secure Java code, for example. When you say you want them to teach people how to write secure code, you're asking them to do something that they themselves don't necessarily know how to do. So it's a challenge.

**28:35 Robert Hurlbut:** When you think about advice that you would give or actionable items that you would give to someone who is perhaps new to the world of secure development lifecycle, We'll start with somebody new. What would you give them as recommendations or things to look at or next steps for somebody who's new to this?

**28:55 Steve Lipner:** I actually gave a talk on getting started with SDL at Black Hat in 2018. I think that that talk and the slides from it are linked from the SafeCode website. People if people want to find that. The short version is, if you are creating an SDL process, start with the basics, make sure you have a security response process, and that you are capable of dealing with discovered vulnerabilities in the software you ship. Then I talked probably at more length than you want me to now about things that basically you can build into your environment that are free, like enabling the security options on tools that you have to use anyway, like compilers. Another big thing is managing third-party components for security so that if there's a vulnerability discovered in some component you're using, you know about it and you're able to respond to it. Actually, again, there's a SAFe Code document on use and secure use of third-party components that provides more detail there. But that's a thing. Then start with training and using the tools and doing code security and threat modeling and so on. Really, I did try to think about Okay, from from a perspective of how do you how do you do you know how do you adapt adopt how do you build a software security program? You know what are the things that are most important? The the other thing I haven't mentioned the the original the original SDL that we created at Microsoft was was somewhat targeted toward the sort of. waterfall or spiral development cycle, but slow cycles that we did back in that era. People don't do that today. They do Agile or they do DevOps or other models that are very fast. That says when you're adopting an SDL, One of the things you have to do is to take the tools and take the steps that are part, that have to be done, and then overlay them so they're integrated into the way that the developers do development. One of the things I've told people is that you'd rather have desktop static analysis that reports the errors right then to the developers, and they can fix them as they go, rather than have some model where you're gonna do static analysis in a batch mode every 2 weeks or 4 weeks, 'cause that's just not consistent with the way people develop today.

**32:12 Robert Hurlbut:** And I was gonna ask you a separate question about what would you recommend for somebody with a mature SDL, but I'm gonna answer that one for you, and I'm gonna say, go check out Steve's talk from Black Hat as well, because when I hear things like, What are some of the options you can turn on with tools? I know that that's something that's gonna be interesting to somebody that even has a mature program, and there's gonna be some things in that talk that they likely haven't thought about, even though it is called Getting Started with SDL.

**32:42 Steve Lipner:** Well, I think a lot of organizations have done most of those things, or a lot of those things, but maybe something for everybody in there.

**32:53 Robert Hurlbut:** Most definitely. Well, Steve, thank you so much for taking the time today to give us a bit of a history lesson and then talk about the modern day of SDL and then also a little bit about the future. So it was great to chat with you and we will definitely reach out again for next season. I probably wrote down 3, 3 or 4 things we could talk about in a future episode. And so we'd love to have you back again. Thank you for the time today.

**33:14 Steve Lipner:** Thanks, Chris. I enjoyed talking with you and I'd be happy to come back.

**33:22 Chris Romeo:** Thanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, security is a journey, not a destination. Thanks for listening.

---

Source: https://appsecpodcast.com/steve-lipner-the-past-present-and-future-of-sdl/