Chris and Robert -- Security Champions

0:00Chris RomeoHey folks, on this episode of the Application Security Podcast, Robert and I are going to talk about all things security champions. We'll introduce what is a security champion, who are they, when do you need them, why do you need them, and how do you set up a beginning security champion program. So this is designed to be a basic introduction to security champions. We hope you enjoy. The Application Security Podcast. Here we go. Hey folks, welcome to this episode of the Application Security Podcast. This is season 3, episode 2. And the title of our episode today is Security Champions. So, in this episode, Robert and I are going to take a deep dive into this topic that seems to be all the rage right now, Robert, across the industry, this idea of security champions. Are you seeing this thing all over the place, too?

1:18Robert HurlbutI have. I can remember just a few years ago, a couple of articles about it. And at that time, it was still relatively new. Nobody was really talking about it. Nobody knew what it was, I think. But now, I'm seeing companies that are implementing it or at least trying to. Certainly, a lot more talk about it out there than there used to be. So, yeah, this is pretty exciting.

1:40Chris RomeoYeah, definitely. And I'm coming at this from kind of a different perspective than a lot of people in that the large technology company I used to work for, one of the things that I did was I actually ran the Security Champion program there for a couple of years. And so, I've got kind of a unique perspective on taking a Security Champion program from around 30 people people when I picked it up till the time I left that company. The champion program was around 500 people. So, I'm going to weave some of those lessons in here. But I think the first thing we got to do, Robert, is start with what is a security champion? And what is that from your perspective?

2:18Robert HurlbutWell, in my experience in working with a company, I did work with one company where basically they brought me in as a security champion among others. Their perspective and my perspective was somebody who has typically some kind of development background, has an interest in security, application security, and ideally they're working with development teams, project managers, others, and they're working with those teams as somebody who has an eye for security issues, can bring attention to those issues to the team and can kind of be a sounding board, if you will, for certain issues and so forth. And so that's, to me, that's that person that you designate, somebody who has that interest, and you call that person the security champion for the development team or multiple teams. Maybe they have more than one team that they're interacting with.

3:19Chris RomeoYeah, so I guess to build on top of your definition, so Based on my experience, I'm thinking of this more from the programmatic side of it. And so, I'm thinking of a security champion as being somebody that I'm going to go out and recruit, like somebody that's a developer in a particular business unit or on a particular product team, somebody that I'm going to go out and recruit, and I'm going to then teach them about security and prepare them to be able to implement product security. You know, kind of outside of a core central security team within a bigger organization. So, that's kind of what I think of when I think of security champions because I guess that's a little bit of kind of from my background. And I think this is good because you're coming from the perspective of somebody who actually was brought in as the security champion, and I'm coming from the perspective of somebody who was responsible for an entire organization of security champions. Right.

4:19Robert HurlbutRight. So, great perspectives. I like it.

4:22Chris RomeoYeah. So, I guess the connection between application security and security champions, I think this is something that a lot of people are going to be thinking about right now. They're going to be kind of wondering, like, you know, this is the Application Security Podcast, of course. What's the connection here between security champions? And when I think about that, I think about kind of the core pieces. Like, when you say you have an application security program, One of the things that I expect that I'm going to be hearing about very quickly after someone says they have an overall program is that they have some type of a secure development lifecycle. They have a process for how they approach security and how they build that security into everything that they do inside their organization. And so, when I think about the connection between AppSec and security champions, the AppSec program defines the process and the tools and how we're going to impact the the build pipelines with the right tools to ensure application security is being considered at every point. The champions are the hands and feet that are out distributed across the company who are using the tools, the processes, and borrowing from the expertise that exists inside the AppSec program team. Is that consistent with kind of how you think about this, or do you got other kind of perspectives?

5:40Robert HurlbutNo, that's pretty much it. I think that they are the ones who are understanding the SDLC or secure SDLC. They understand the various phases. They understand how to implement those, and they're helping those move along. They're helping each phase or each part of that in terms of a program. How do you make those happen? And so I think that Yeah, there's a good connection between the two as a person who's helping to facilitate, helping to make that be realized, and the program itself of application security through the Secure SDLC. I think it makes sense.

6:22Chris RomeoYeah, I think both of them, you know, they're definitely— it's a very tight partnership between AppSec and between a security champion program. And, you know, one of the other things that I think about when I say security champions here, I want to make sure I mention, is you'll see this actually referred to with a lot of different terms that all mean exactly the same thing. So, I've seen some organizations that refer to this group of security-passionate people as security advocates. I've seen it referred to as a security guild. So, I've seen, you know, there's a couple companies that have taken the approach of going old school with like craftsmanship and, you know, the idea of somebody who's kind of a journeyman who goes through and learns a number of lessons before they become a master or something. And so they use that guild. And then, but security champions is really the term that I'm seeing used generically in the industry to describe this type of a program.

7:21Robert HurlbutYeah, I've seen the same thing and I've heard all those other terms as well. But yeah, security champion, I think, is, is a good general term that applies essentially as if you think of the term champion itself, you know, somebody who out there kind of— I don't know, what do you call it— like a cheerleader of sorts. I mean, just, you know, hey, we definitely need to think about this. Security is important. Not only is it important, it needs to be something that we do. It's something that we don't just think about, but we actually have in place plans to make it happen, and here's how to make it happen, and so on. And so I like that that term champion. They champion the cause, if you will.

8:05Chris RomeoYeah, and I'll even build on top of that a little bit more now that you kind of— you're making me think of, you know, how do we break down the champion's role into a number of different kind of focus areas? And there definitely is a focus area of just delivery, on delivering on the product security goals set forth by a central group. using the processes and things. But there's definitely also that engaging part of the role where, you know, a successful security champion is going to go out and actually teach people within their group how to do something like threat modeling, for example.

8:44Robert HurlbutRight.

8:44Chris RomeoSomething we love here on the AppSec Podcast. We're huge threat modeling fans. Threat modeling is something that I've found in my experience is if you teach a small group of people that are distributed from across all the different groups in your organization how to do threat modeling, and then you send them out to teach other people around them. Threat modeling is one of those things that's better caught than taught, right?

9:09Robert HurlbutRight.

9:09Chris RomeoYou have to kind of do it. You have to get out there. And so, Security Champions is a great example of someplace where you can use a group of people there that are that are passionate about security, you can teach them threat modeling and you can say, hey, please now take this to all of the people in your group.

9:27Robert HurlbutRight. I agree. I agree. And in fact, that was one of the things I was thinking about as well, is teaching not just the principles, the plans, and so forth, but yeah, how do you do this and modeling it and helping others to do the same as they go back to their groups, go back to their teams. and do the same things in terms of security threat modeling, other kinds of security practices.

9:51Chris RomeoYeah. So, another question I guess we've been working towards here is, so who are the security champions in a given organization? And you kind of touched on that a little bit earlier, but from your perspective, who do you see as the security champions? Where are these people coming from?

10:06Robert HurlbutWell, as I mentioned, at least those that I've seen that that makes sense, and it can be anybody really, but the ones that I've seen to be really effective are those who may have some kind of development background or at least be aware of development and how it works and that tie in with application security, I think, is a good person that could qualify as a really good security champion.

10:34Chris RomeoYeah, and that would classify the large majority of the people that were part of the Security Champion program that I had a big impact on. I would say fit into that category of developers, technical backgrounds, good at coding, but maybe don't necessarily have the security foundation that they can then tap into when they're trying to do secure coding and they're trying to look at their code and find vulnerabilities or avoid vulnerabilities completely. But I also, in kind of coming from the institutional side of it, I guess, One of the other things that I did is I integrated other roles. First of all, any— and here's a recommendation for anybody out there who's listening to this and thinking, hey, maybe I should do one of these type of champion organizations. I always left it wide open to anybody. So, I don't care what somebody's job role was in the program that I was running. If, you know, we had people that came from engineering, we had people that came from IT, we had people that came from— even sometimes people from customer advocacy, you know, customer support people. And I never closed the door on anybody because I thought, you know what, if anybody inside this company wants to come to this meeting, wants to be part of this group because they love security, I'm not turning anybody away. So, that's, I guess, one of my lessons learned here is that while you're going to focus in, the developer is going to be your primary role that you're going after in the beginning, don't turn away other people. Provide avenues for developers, for testers, for product managers. If you work in a product company, who are the people that you need first to build security into your products? It's the product manager who is the one who controls the original spec that says we're gonna build something new, right? If you get them involved in your champion program, even at a cursory level, they can become an advocate for you as well.

12:32Robert HurlbutThat's a great point. Like I said, I focus more on development and those who have that background, but absolutely, I like that idea. Just open it up to whoever is interested because that's number one, I think, is do you have an interest in security? Do you want to learn more? And I think that's a good point. I like that.

12:51Chris RomeoAnd then, so I guess the next question to tackle here is when do you need security champions? My answer is 10 years ago, probably, maybe 15. It would be an ideal place to have started. So, you kind of have 2 options here. I'm not sure, but maybe that DeLorean thing from Back to the Future actually does exist. The flux capacitor is a real thing and you can travel back in time and kind of start a program inside of your company and then come back to the future and see how it worked out. Maybe that's actually not going to work out so well. So, basically, I mean, you need security champions. If you don't have them, you need them now. And hopefully, you're 5 or 10 years into the establishment of a program because security champions is a long-haul game. You're going to— if you create a program today, you can't call me up in 3 months and say, you know, this champion thing is not working out. We are not getting the impact. We're not getting the results we needed. You know, this is a multi-year effort to really get momentum. I mean, you'll get some momentum early on as you get some of those really passionate people that you just kind of grab onto and pull them into kind of a core group of security champions. You'll get some momentum with them, but it's going to be 1 or 2 years before you really get that momentum.

14:15Robert HurlbutYeah.

14:16Chris Romeosnowball kind of going down the side of the mountain that turns into a gigantic boulder and an avalanche. It's going to take you some time to get to the point where you're really seeing huge results that you can start to roll up to executive management.

14:30Robert HurlbutYeah, it is the long game, right? It's not something that we'll try for a couple of weeks and see where it goes, but it really is the long game. Once you start down this path, you need to keep at it. It needs to be a part, just like everything else that you're making a part of your organization in terms of security. When you're starting to implement a secure software development lifecycle, if you're starting to think about threat modeling, you're starting to think about all these things. Introducing security champion is a long game. It's something that you want to start, continue, build on, and keep it going. It's going to pay dividends, I think.

15:11Chris RomeoSo why do you think— I mean, why do people need security champions then, from your perspective?

15:16Robert HurlbutWell, I think the main reason is that in terms of development teams, there are a lot of things that they focus on. They focus on, of course, the requirements and all the things that go into those, functional and nonfunctional. Security can be considered by some as a nonfunctional requirement. With that in mind, it may or may not always get the attention it needs. Sometimes it could be primary depending on the kind of applications that they're building and trying to determine, okay, is this a critical thing? Do we need to think about security? Or it could be one of those backlog things. Well, we'll get to it eventually. We'll get to it. But we know, you and I and others who've had experience with this, is that building security in later is always a bad idea. Ideally, you want to think about this from the very beginning. And having someone who is there at the forefront thinking about that, bringing those security issues to the attention of the entire team very early on, is really important. But not everybody has the expertise. Not everybody knows all the security issues that they need to think about and so on. And so, on a security development team. And so, why do you need a security champion? Well, that's the person who can help bring the attention that's needed to this requirement early on and consistently throughout the project.

16:52Chris RomeoYeah, and definitely agree with your kind of answer to why. I guess my take on this is you need security champions because security champions breed security culture and security culture change. And so, when I think of security culture, I think about how, you know, how's a developer going to respond when they've got a particular security challenge in front of them? Meaning, they have option A, which is to take the insecure route and ship the code and meet their deadlines, or option B, where they can— where they're going to put up their hand and say, nope, I'm not shipping this because there's a security flaw here and I refuse to let it go out the door. So your security culture is defined by what does that developer do when nobody's looking over their shoulder, when they've got the ability to flat out just make that decision. And so security champions, the more security champions you get around, the stronger your security culture becomes, and that's a huge benefit for you as you get, as an organization, as you get more serious about security and you think about You know, how are we going to embed this security thing? Well, it happens with security culture change, and that happens by, once again, hands and feet on the ground through security champions. So, the big benefits from my perspective are you're going to get an organization that's focused on security, and in the long haul, that's going to result in less vulnerabilities in your product. Once again, caveat, asterisk, put that asterisk next to long haul, right? You're not going to, you know, in the first quarter after launching this, you're not going to see your vulnerability challenges and things and problems your tools are finding are not going to drop off by 50%. It's going to take some time. But as people start to understand more about security and they see that shared mission that you're instilling in them as a champion, they're going to start to make improvements. And you get to the end of year 1, end of year 2, you're going to start to be able to see things in the metrics that allow you to actually quantify the fact that security is getting better inside your organization.

18:57Robert HurlbutAbsolutely.

19:00Chris RomeoSo, I guess the last part, what we wanted to kind of enlighten our listeners on is if somebody happens to be listening here and they're thinking, hey, I want to set up a security champion program, where do they start? Is kind of the last question that we wanted to address here. And so, I'll give an example first, Robert. One of the things that I recommend to people is kind of the first way to get this thing going. And I'll let you come back with another one. But the first one that I always recommend to people, I think the most simple thing you can do after you say, hey, we're forming this group, is to set up a monthly training session. And what I like to see there in those monthly training sessions is I like to see a gathering time where you bring the champions together, you spend some amount of time briefing on maybe some piece of the secure development lifecycle. If this is a brand new thing for your teams, maybe you spend one 30-minute segment talking about threat modeling or security requirements or static analysis, dynamic analysis, code review. You pick whatever piece of your secure development lifecycle, but you kind of work your way through the secure development lifecycle. for 30 minutes. And then, you bring in— for another 30 minutes, you bring in either somebody from the internal team who is a passionate security person who's got some expertise and has something they want to share, or you go out and invite somebody from the external community and ask them to come in and do a 30-minute talk. And so, for the people right now who are gasping and saying, I could never get somebody to come in and talk to my Security Champion group for 30 minutes. Here's a little tidbit for you about people in the community, even such as myself. If you want me to come and fly to your offices and spend the day getting there to talk to your team for 30 minutes, the answer is probably no. If you want me to log in on a web conference and talk to your team for 30 minutes about something that I already love, the answer is almost always going to be yes because I'm only giving up 30 minutes of my time ultimately.

21:04Robert HurlbutYeah.

21:05Chris RomeoAs long as I don't have to travel to do that. So, what I'm saying here is you can— I mean, because I had all kinds of people come in when I ran the program at this other company I was with. I mean, I had Katie Mazouras come in when she was at Microsoft. I had Dave Kennedy come and do a talk. Jeff Williams came and did a talk, all because— and I didn't ask them to come and fly and spend a bunch of time to get to come and see us. I just asked them for 30 minutes on a web conference. And you know what? Most people will give you that 30 minutes. So, the monthly training session is a great way to start this process off and to get your folks kind of engaged and involved in being a part of something. And it gives you a measuring stick. It gives you a way to engage people in a monthly fashion. And it just provides that central connection point when you're kicking off a new program. Robert, what would you say? I mean, what would you tell people about starting a beginning kind of champion program?

22:00Robert HurlbutWell, I do like that training opportunity. Along with that is you want to identify either internal people who might be interested, who in particular have a more keen interest in security, want to— let's say they're developers and they want to go beyond what they're doing and think about application security, learn more about application security. If you can identify those people, that's great through those training opportunities. Or you look for outside. If you look around and look for somebody who maybe has some experience in application security or development with application security or wants to help with application security in a team, that might be another opportunity as well to bring in somebody from the outside. I think either way what you're doing is you're trying to identify good candidates and apply all the things we just mentioned before about who you're looking for, what you're looking for, and what you want them to do. Get that clear, I think, is one thing, is to also determine what will they be doing as a security champion and make that available to people to say, okay, we're looking for people who are interested in this kind of role. Is this something that you would be interested in? Here's what you're going to be doing. And I think it's a good way to find those people as well and start that program to be able to then carry it through.

23:30Chris RomeoYeah, it's another set of great points there because you mentioned role definition, which is something that I also recommend when people are starting off with a beginning program. Set up some goals and objectives. Set up, you know, what do you want people to do? What are they signing up to do? Because one of the things you'll find in most organizations is this is going to be a nights and weekends job for some people. So they love security so much that they want to invest their time in it, but they're not necessarily going to get a slice of 10 hours a week from their manager to focus on their security champion role. It's great if you can do that, if you can negotiate that into your original roles and your executive management says, yeah, okay, you know what, you could have 25% of all these people that you sign up. Great, more power to you. How many times have I seen that actually happen? I've heard of it happening once in all the different things I've looked at and stuff. And so, so a lot of times, so by, by, well, by defining that role well, you're giving people the opportunity to understand what am I signing up for, how am I going to balance it against my other commitments, and what are the rewards going to be. And so it's good, it's good. People should get rewarded for going above and beyond And, you know, participating and being a security person here in this world. So, you talked about the role definition. That's definitely huge. The recruitment side as well is big in the beginning of a program. So, going out and looking for the right people to get that core group. So, you're really looking for the people that are already passionate about security. You're just kind of—

25:05Robert HurlbutYeah.

25:05Chris Romeoyou're capturing and channeling their security efforts and having them be kind of beginning or core members of this champion program. Getting that strong foundation of people that already love security is a great draw for when you start inviting more other people kind of to come into it.

25:21Robert HurlbutYeah, I agree. Just thinking about how can we leverage what we have or bringing in, but also just giving some parameters that That way, people understand what's expected, where are we heading, and also success. You know, what are success parameters? How do we know that we've reached what we need to reach and where else do we need to go? And setting goals and things like that, I think, helps for the program as well.

25:48Chris RomeoYeah. And that's a whole other show that we'll address that sometime in the future in season 4 or 5. We'll come back and do another show short episode like this talking about metrics and all the things that go into— because I could literally talk about champion programs all day. It's something I enjoy, enjoy setting up and being a part of. So it's, it's— I've got a lot of, a lot of additional data. But I would say, Robert, for now, let's leave the listeners with kind of this basic introduction, and we can come back and dive deeper into this topic in a future episode. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @appsecpodcast or on the web at www.appsecpodcast.org.