---
title: "Alex Olsen -- Security champions, empowering developers, and AppSec training"
url: https://appsecpodcast.com/alex-olsen-security-champions-empowering-developers-and-appsec-training/
date: 2022-12-20
duration_seconds: 3556
guests: ["Alex Olsen"]
topics: ["Threat Modeling", "Building an AppSec Program", "Security Testing", "Security Culture"]
audio: https://www.buzzsprout.com/1730684/episodes/11860582-alex-olsen-security-champions-empowering-developers-and-appsec-training.mp3
video: https://www.youtube.com/watch?v=S_C20kwcLKc
transcript: true
---

# Alex Olsen -- Security champions, empowering developers, and AppSec training

*December 20, 2022 · 59 min*

with [Alex Olsen](https://appsecpodcast.com/guests/alex-olsen/)

on [Threat Modeling](https://appsecpodcast.com/topics/threat-modeling/), [Building an AppSec Program](https://appsecpodcast.com/topics/appsec-programs/), [Security Testing](https://appsecpodcast.com/topics/security-testing/), [Security Culture](https://appsecpodcast.com/topics/security-culture/)

[Audio](https://www.buzzsprout.com/1730684/episodes/11860582-alex-olsen-security-champions-empowering-developers-and-appsec-training.mp3) · [Video](https://www.youtube.com/watch?v=S_C20kwcLKc)

## Show notes

Alex leads the Cyber Security Consulting Group, part of Rakuten's Cyber Security Defense Department. The group's dedication is to providing global security services, including security architecture, DevSecOps tooling and integration services, delivery of technical training, and running Rakuten's Security Champion community. His focus is on empowering teams to improve security throughout the development lifecycle. Alex Olsen leads the Cybersecurity Consulting Group, part of Rakuten's Cybersecurity Defense Department. The group is dedicated to providing global security services, including security architecture, DevSecOps tooling and integration services, delivery of technical training, and the running of Rakuten's Security Champion Community. Alex joins us to talk about security champions, one of the topics that's near and dear to my heart.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
We help enterprises reduce vulnerabilities through application security education for developers and everyone in the SDLC.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Alex Olsen:  
→ [Rakuten: Democratizing AppSec](https://rakuten.today/blog/democratizing-appsec-enhancing-security-quality.html)  
→ [OWASP Global AppSec APAC](https://apac.globalappsec.org/)

Mentioned in this episode:  
→ [Rakuten: Democratizing AppSec](https://rakuten.today/blog/democratizing-appsec-enhancing-security-quality.html)  
→ [OWASP Global AppSec APAC](https://apac.globalappsec.org/)

Chapters:  
00:00 Meet Alex Olsen: Security champions, empowering developers, and AppSec training  
03:54 There was a term that was used, democratizing AppSec. And it  
07:00 Yeah. So, I guess what's the trade-off then when you— it  
10:21 Like, they are the police because they can stop deployments, they  
11:36 Yeah, so the kind of primary thing we want to talk  
15:03 What's been your response or your experience with that  
19:43 I'm listening to kind of— you're describing an onboarding process for  
22:21 We think about, like, scope, cost, and effort, levels of effort  
26:50 There's lots of excitement, right  
35:21 Tactically or specifically about individual champions now. What have you seen  
46:20 It was the return on investment. So, you know, what are  
53:59 From a key takeaway perspective now, so let's kind of summarize

## Transcript

*9,944 words · assemblyai*

**0:00 Chris Romeo:** Alex Olsen leads the Cybersecurity Consulting Group, part of Rakuten's Cybersecurity Defense Department. The group is dedicated to providing global security services, including security architecture, DevSecOps tooling and integration services, delivery of technical training, and the running of Rakuten's Security Champion Community. His focus is on empowering teams to improve security throughout the development lifecycle. Alex joins us to talk about security champions, one of the topics that's near and dear to my heart. We get into what is democratizing AppSec, what's the value of security governance and empowerment activities for champions and for the organization, how does scope, cost, and effort fit into this, and then what is the ROI on things like training and security champions? We hope you enjoy this conversation with Alex Olsen.

**0:58 Robert Hurlbut:** The Application Security Podcast is brought to you by Security Journey. We help enterprises reduce vulnerabilities through application security education for developers and everyone in the SDLC. Learn more at securityjourney.com.

**1:11 Chris Romeo:** Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I'm the Chief Security Officer at Security Journey and co-host of the podcast. I'm actually flying solo today. But I'm excited to be joined by Alex Olsen, and we're going to talk about something that's near and dear— talk about a couple things that are near and dear to my heart: security champions and security training. Alex, whenever we start an interview, we always dive directly into our guest's security origin story because our listeners, no matter where they are, are sitting on the literal edge of their seat waiting to hear you got into security. So how'd you get into security?

**1:53 Alex Olsen:** Well, thanks for the introduction. Um, yeah, my, uh, way in unfortunately is, is pretty typical and, and not too exciting. So, um, as, as many people in AppSec, uh, I started my career as a dev. So, you know, I had a, a dev job out of university, um, and then I, I did move into information security first Um, so took a slightly more like, yeah, hands-off looking at policy governance, helping, uh, organizations, um, try and, uh, get some of their things in order. So trying to bridge that gap between the technical world of security and, and the business need. Um, and then worked for a few different organizations, um, big and small. Um, I definitely, I've been part of a, a 2-man security team at a reasonably large organization. And, um, that's kind of a double-edged sword because because you learn a lot of things because you have to do everything, but it's also very stressful because everybody expects you to know everything and do everything. So I'm grateful that I put my time in, but I'm not sure I'd do it again. I definitely lose a few hairs in that kind of setting. And yeah, so since I was a dev, AppSec's always been my, I would say, strong points. Done a little bit of penetration testing, now part of a much larger security team, uh, at a tech company called Rakuten. And, um, yeah, basically looking after, um, the secure development lifecycle, security champions, cybersecurity training, um, and, and quite a few of the other services that, uh, that Rakuten basically provide, uh, as an internal service to, to development teams.

**3:41 Chris Romeo:** Okay, so I know that a few Rakuten people, including yourself, did talks at the OWASP Global AppSec APAC edition here a few months ago.

**3:53 Alex Olsen:** Yeah.

**3:54 Chris Romeo:** And there was a term that was used, democratizing AppSec. And it seemed like it was, you know, from the Rakuten blog post I saw that we'll put in the show notes, it was kind of how you interwove your 3 talks together under this heading. And so I've heard other people use this term democratizing AppSec, but I want to get your take on what does that mean from your perspective?

**4:17 Alex Olsen:** Yeah, so I suppose there's a few things that, that go into it. The first is, um, who's really responsible? Like, so if I have a development team and they want to build something and release it, they're responsible for the security of that. Um, now obviously we have internal, um, regulations that say, hey, you have to do X, Y, and Z. These are the minimum requirements, basically compliance. But ultimately, you know, if, if your service is breached or if something bad happens, the security team is not, it's not responsible. It's your service, you know, you, you have to look after it. And so I think given that that's the case, and probably rightly so, um, what we try and do is make sure that, um, services that, or that the business need, uh, not necessarily owned by, but, um, are, um, built around teams and, and what they need at, like, you know, let's say every step of the typical development lifecycle, um, that they have the tools and services they need, um, that are relevant to them and, uh, ones that they want to use rather than services that we want them to use. So Um, as an example, um, let's say a lot of teams come to us and say, hey, we want to do threat modeling, or we want to do, um, we want our architecture looked at. Um, so there's, there's kind of 2 ways we could go about this. We could be like, hey, okay, come to us and we'll give you a report, we'll do an architecture review. Or we can say, okay, we can do training and we can do an architecture review for you to get started, but But then, you know, every sprint you can revisit your threat model and then you can kind of keep it up to date. And basically that pushes the ownership of that service or that, um, uh, like security process onto development teams. And then they can take that, adapt it, make it what they need it to be. And I think for me, that's really, um, ties in very closely with the shift left philosophy. Um, and that's, that's basically what it is. So instead of us telling teams, you must do this, it's more like, okay, let's collaborate together, create a service that works for you and meets the organization's security requirements. And we'll help you do that. But really, uh, giving them a lead role in, in that, not just, you know, do this 'cause we said so. So, Yeah. 'cause we're not the police. And I, I think, um, that comes up. I say this a lot in meetings actually. We're not the police. We don't own the risk. So, you know, it's—

**6:59 Chris Romeo:** Yeah. So, I guess what's the trade-off then when you— it sounds like what you're describing here is, you know, developers and development teams own a particular service. They're given some span of control. So, in this concept of democratizing AppSec, do they still have to use a standard set of tools that you provide that are part of your SDL? Or can they just free— can they freelance everything along the way as long as they're within a certain kind of wide set of guardrails? Or how much freedom do they actually have?

**7:33 Alex Olsen:** So in terms of, um, like the development lifecycle, if they want to use a particular tool, um, they're welcome to do so. They don't have to use the tools that we provide. So, uh, we kind of have like a dual offering. Some services are fully managed, so you tell us, we'll scan your applications, we'll give you the results, and we'll, we'll automate that and do it on a schedule. Or we'll provide you with a nice containerized scanner that you can deploy, integrate into your pipeline. Um, because we have a lot of different development teams on different technology stacks, we basically kind of have to provide both of those services. But if a team comes to us and says, hey, you know, I really want to use X scanner instead of, instead of Y, and there's a good reason Um, there's— I mean, we, again, we're not the police. We can't really say no. And there are some minimum requirements. So for example, to launch a service, you have to pass, um, uh, what we call a pre-release, um, audit, which is basically a penetration test. So, um, you have to come and do that. There are some like, you know, non-argument things. But in terms of the development lifecycle, um, you know, we have waterfall teams, we have agile teams, um, we have, uh, teams following completely, you know, their own, um, style of development or development lifecycle. And some teams have 50 devs, some teams have 1, or some teams now have 0, unfortunately, but the application still exists. Um, and so yeah, um, I think we have a lot of flexibility in terms of, um, what you're doing during development, uh, until you get to those, um, later release stages where you need to pass your pen tests or you need to, um, do your patching and things like this. And in that case, we basically have my group and we kind of act as consultants. So, you know, we might not be able to turn something around within, you know, a day, but if a team comes to us and says, hey, we want to use this tool, we need some help integrating it though, they can put a ticket in and we'll support them with that. So, you know, we have architects, DevSecOps engineers, and security engineers kind of on hand to deal with those types of requests. But I would say 95% of the time we have enough coverage in terms of the tools. And the tools are quite mature now so that actually it's easier for them to kind of follow what we have. And that's the service we want to give. We want people to want to use our services. So that's like, you know, the ideal place to be.

**10:05 Robert Hurlbut:** Yeah.

**10:07 Chris Romeo:** And I love the idea that your statement is, we're not the police.

**10:11 Alex Olsen:** Yeah, exactly.

**10:14 Chris Romeo:** So many programs I see, they could say they're not the police, but effectively they are.

**10:20 Robert Hurlbut:** Yeah, yeah.

**10:21 Chris Romeo:** Like, they are the police because they can stop deployments, they can be a roadblock. And this idea, what I'm hearing in democratizing AppSec, is it's really about partnership, collaboration. It's not like it's a free environment where they can choose whether they want to do security, the right security things or not. They still have to do the things. You're just giving them room to do them creatively the way that they want to, as long as they meet that goal of passing that audit. Sounds like it's a better way to give developers more room for what?

**10:56 Alex Olsen:** Yeah, yeah. And I think there's a lot of passive benefits to that as well, because we have better working relationships with the teams. And again, there's no confusion around, you know, who owns what or who's responsible for for what? Because I, you know, I've been in, um, I was at a conference, uh, maybe a couple of years ago now, virtual conference, and somebody was saying, uh, as a member of the security team, if I pick up something on, on the web server, I'll shut the server down. I'm like, whoa, how? You shouldn't. That's not your server. You can't just go and shut things down. So, um, yeah, it's, it's really, yeah, really important to understand and meet each other halfway.

**11:35 Chris Romeo:** Yeah, so the kind of primary thing we want to talk about here is Security Champions. And I guess before we kind of dive into that, I'm curious, how did you get into Security Champions? Did you just kind of fall into it? Like they said, hey everybody, we need a volunteer, and everybody else took a step backwards and you were standing there in front of the group going, wait, what just happened? Or, you know, how did you get into or get get connected to this idea of champions?

**12:05 Alex Olsen:** Yeah, so, um, the security champions, um, it was actually an initiative that already somewhat existed before I joined, uh, current— my current organization. So, um, it went back, it was under a different name. It used to be called System Security Lead, and they had some responsibilities were a little bit different. Basically Um, because there are hundreds of development teams, you need somebody on each team to assign vulnerability tickets to. Otherwise they just sit there and nobody, uh, takes ownership of them. So I think that was basically the original use cases. Somebody needs to be responsible for, for findings and we need to know who to chase. Um, and this was, um, maybe 10 years ago. I've seen documentation going back for a very long time. So this kind of idea of, uh, security champions, maybe. existed a long time ago but wasn't, you know, very mature at the time. And then, um, let's say about 4 years ago, um, so I've been looking after cybersecurity training and expanding that. And I think security champions being, um, probably, um, they attend a lot of our training, come to a lot of, um, they test a lot of our tools. Naturally, I had a, a lot of, um, uh, day-to-day connection with the security champion community. And so I basically said, okay, let's, let's revamp, because, um, a lot of teams didn't have security champions. Some of them were assigned, but, you know, we were incommunicado, could never get a hold of them, um, and, and things like this. So, um, I think from the top there was like a, hey, can somebody, you know, do something and, and revamp this? And I was probably just in the best place to to take it and run with it. And we did a whole revamp and we changed the name as part of the, basically like the relaunch and the expansion. And at the time it was also, it was an empowerment activity. So it wasn't mandatory, not every team had to have a security champion, but yeah, I changed the group regulation to basically say, okay, every development team must have an assigned security champion because I think for it to work, Yeah. Um, especially in the way that, uh, we intended, where we have at least one connection to every development team. Um, we needed to make sure that, that, um, everybody was, um, every team was, was on board. And that's probably like the one governance part of our Security Champions program. We do a lot of empowerment stuff in that, hey, there's this training if you want to do it, or these, these extra tools that we're testing that you don't have to use, but they'll give you some benefits. Um, come to our threat modeling sessions if you want to. So the assignment is mandatory, but a lot of the other stuff we do is basically come if you want, but we're not going to force you.

**15:02 Chris Romeo:** What's been your response or your experience with that? Because, you know, when I've talked to folks about starting new champions programs and stuff, this is kind of the voluntold versus volunteer. stage. And I've seen in a lot of different security champions programs, I've seen them kind of broken into 3 phases with a number of iterations of these 3 phases. But it kind of starts with a small group of people who have a little security spark that are your initial champions. You go to the voluntold phase where it's like, hey, everybody needs— we need to have a champion from every unit of the business, whatever that unit is, product, business unit, whatever. And then, you reach this stage where people start to volunteer and they're raising their hand going, wow, I see the value in this champions program. And so, but I think of that voluntold portion as sometimes a struggle because I get people that don't really want to be there. They just, their leaders pointed at them and said, you're our champion. Go forth. And they're like, I don't care about security or whatever these people are doing. So, what's been your experience with that as far as people being voluntold? To be a champion?

**16:17 Alex Olsen:** Yeah, so I suppose that, um, in our case we have a mixture of— we do have a group of voluntold people who are basically assigned because it's a regulation, but a bunch of people who also, you know, enjoy and proactively contribute, um, as well. So, but going to the voluntold side, um, it really depends on— yeah, the, the classic answer— it depends on, um, your organization and culture and things like this. I am a really firm believer in that having good onboarding training and having good orientation and connecting with people on a, like, you know, face-to-face— I know that's not always possible, but as close as you can— will get you off to a really, really good start. The worst thing is, imagine if you're assigned a security champion, don't really know what your responsibilities are, you maybe join a webinar or some kind of lecture where you don't really want to ask too many questions, and then you get radio silence after that. I mean, you're never gonna really contribute, right? So you need to pull people into the community and make them part of it. And yeah, how we do that is we do have our orientation training, and then we do have like an open chat room where anybody can ask any questions. And, you know, when there's activity in there, people see that and they think, oh, it's okay, 'cause this person's asked, where are the latest crypto guidelines? Or this person's asked this question about this tool. And, you know, if you have that kind of open communication, that really helps. Something that I think a lot of people have struggled with in terms of launching a Security Champions program or initiative is that, you try and get a champion from every business unit, their requirements or their needs are very different. So we're very specific in that development teams and developers are our security champions, mostly because we have a separate information security team. They deal with policy and, and, and compliance and governance, and we deal with cybersecurity, so technical stuff. Um, so we're very lucky in that point of view where, um, I can spend all my time talking about cool technical things and really deep diving on scanners and taking devs' issues and helping them. But if I had somebody from accounting, for example, I'm not sure what I would tell them. I'd be like, try not to click on any phishing links. I mean, beyond that, I don't really know what to teach them. So I think our program is very focused, and I think that's a real benefit. or a really good thing. That's not to say that you can't have a diverse security champions program, or you can't have different streams, but I think it's difficult to make that work. Yeah.

**19:16 Chris Romeo:** Yeah, I've seen champions programs that focused on non-developers, but it was using a lot of the same strategies that we use to work with developers, but just different content, different kind of things we're asking them to do. So, I think the overarching structure works. It's really just different. You're just going to be having different activities for them to perform as they make their way through.

**19:43 Robert Hurlbut:** Yeah.

**19:43 Chris Romeo:** So, when I'm listening to kind of— you're describing an onboarding process for how you counter this idea of voluntold. And I think that's a really interesting way to approach it. I've never really thought about or instituted an onboarding process where I was specifically thinking about how am I going to draw these people into, you know, putting myself in their shoes for a second. Like you said, they're brand new. They get thrown into a webinar. They're like, what are these people talking about? This isn't— I don't want to be here.

**20:16 Robert Hurlbut:** Yeah.

**20:17 Chris Romeo:** How do you, you know, how do you engage them and bring them in? So, I'm going to add that to my tool belt, my Security Champions tool belt, this idea of onboarding and some different levels of being able to onboard there. I think that's, I think that's a very valuable step to connect them to the community and make them feel like this is, this is, I'm home. I'm not an outsider here. I'm part of this thing.

**20:38 Alex Olsen:** Yeah. And at the very least as well, uh, it gives them a good baseline in terms of clarifying, you know, somebody might've been like, hey, you're a security champion. And you as a dev might've been like, well, what do I have to do? And then your manager's like, I don't know. So it really clarifies kind of like, um, what's expected, you know, um, what services are available, what support's available, you know, where to go. And also puts a name to, to a face as well. So within— I mean, it's a big organization, so I don't— I try not to do it too much. I, when I get requests, I'm like, ah, you know, you need to fill in this form, you know, raise a ticket. But with our Security Champions community, we kind of make an exception to that and be like, you can direct message me anytime. You know, and, and, uh, the engineers who are basically, um, monitoring the chat or supporting, uh, the Security Champions program, um, again, kind of adopt that approach in that they're like the face of our organization. So you can direct message them and it's not a problem. We're never gonna, um, uh, be like, oh, just go away or, or, or pass you around to 5 different people.

**21:44 Chris Romeo:** Yeah.

**21:45 Alex Olsen:** So, um, so yeah, we try and, um, create that, um, approachable and, yeah, I think approachable feeling for devs and new security champions.

**21:58 Robert Hurlbut:** Keep security top of mind with continuous application security training for your developers. Security Journey offers bite-sized lessons with hands-on interactive training for all roles in the SDLC. Give your admins the ability to use pre-built or custom training paths with easy-to-use tracking and reporting. Visit securityjourney.com to see our solution today.

**22:20 Chris Romeo:** So, when we think about, like, scope, cost, and effort, levels of effort involved in a champions program, I think we've talked a little bit about scope already. We've kind of— that's kind of been interwoven into our conversation about, you know, the right distribution for champions. But what about from, you know, I think one thing that we never talk about in the world of champions is, like, What does it cost? What's the budget requirements? Because we tell people, and I tell people all the time, oh, you need to start a champions program. I never tell them how much to budget to do that and what the level of effort should be from a central security team's perspective. So, what are your thoughts on kind of cost and level of effort to make this thing successful? Yeah.

**23:07 Alex Olsen:** So, I mean, obviously, it depends on the scale. And you have items that are, some things just cost money. So it could be licenses, could be a training platform, could be if you're set up and let's say like your security champions are the one who's, you know, running vulnerability scanning, for example, you'd need a license for every security champion. So there are like items that are like that, that depends. But, So basically run, I mean, we have somewhere between 300, 300 and something security champions. I think it basically takes 2 full-time employees to run that community. Now that's kind of like spread over a few different people, but basically in terms of headcounts, 2 full-time people is a good estimate. Now whatever that costs within your region, at least that gives you kind of a, you know, a headcount, but this scales as well. So I suspect that actually running a Security Champions program that has 30 Security Champions is probably a full-time job if you're doing monthly update meetings, if you're doing the reports, if you're answering questions, if you're trying to improve services, if you're doing awareness, if you're doing all of the things that, you know, contribute to a, I'd say a fairly large-scale security champions program, but a lot of those things scale quite well. So we do our monthly update meeting, for example. Everybody comes along and we present on different topics— open vulnerabilities, changes to services, new training, things like that. The cost of doing that doesn't matter whether you have 300 people or 10. So a lot of stuff scales really well. So from that perspective, the cost is quite low. So I'd say between 1 and 2 full-time headcounts for probably— that is most security champion programs. If you're in a really large organization, 100,000 people, then who knows? It really depends on many things. But to give some scale, my organization is about 20,000 people and about 300 security champions. And, and yeah, it's about 2 people. Um, yeah, and I think some of the other, um, items are, uh, if we want to do, uh, new training, for example, if you have a dedicated training team in-house, uh, like, like we do, we build a lot of our own contents, uh, we use a lot of things that we've seen from within the organization to do that. Again, it just costs us time, but, um, If you're going to pay an expert to come in and deliver training, that's quite expensive. So if you're going to do like a week's training with 30 people, I think we've paid for external consultants maybe like $20,000 for about a week for 5 days and to do like hands-on training or something like this. So I can't give you like an exact, you know, how much does it cost, but I give you some ideas in terms of how much it costs in terms of time and then big ticket items if you can afford them.

**26:29 Chris Romeo:** Yeah, it's helpful to get that perspective. And so I think one of the things that a lot of people that are starting new champions programs struggle with is by the time you get to that 4th month of that monthly meeting, I think the first 3 months are easy because you got—

**26:50 Alex Olsen:** Yeah.

**26:50 Chris Romeo:** There's lots of excitement, right? You've got people. How do you keep the agenda both full and interesting in a monthly meeting a year after the Champions Program has launched? Because I think a lot of people struggle with that.

**27:07 Alex Olsen:** Yeah.

**27:07 Chris Romeo:** And that's why a lot of these programs We kind of alluded to it earlier, the fact that a lot of these programs, in my experience, the program I picked up at Cisco, I ran Cisco's Champions Program. I picked it up from— it was on life support. I mean, it was doing that little blip across the screen. Like, it was flatlining. I picked it up and this was the 3rd time it had gone through an iteration of people got really excited, people spent a lot of time on it, that person left, and The whole thing went down to a flatline. The second person picked it up. I was the third person. So, a lot of people are going through these same type of things. And so, from your perspective on that monthly meeting, how are you filling up that agenda, making it where people are like, I got to be there. This is must-see webinar time. I don't want to miss it.

**27:51 Alex Olsen:** Yeah. I think this is a really common issue, especially with the people that I've talked to. And actually, To be fair though, there's something to be said for, you know, if you are rotating through people and you do a big spike and you do some great things and then it levels off for a while and then you revisit and do some other good stuff, that can be a legit way to run a program. It's kind of quite different. But kind of circling back to your question, I think I mentioned this when I was giving my presentation at the last conference. Repeatable but not repetitive is really, really important because repeatable means it keeps costs down, keeps efforts down, and also means that you can delegate. So you have a mature process. I think treating it like a service where you need to build a process around it, something that works and is scalable and, and, you know, doesn't cost the earth in terms of hundreds of hours. That's just kind of like the business side of building a mature security champions program. You've got to treat it like you would any other service. And the repetitive side is kind of tricky. So again, this depends on your organization. So for me, I'm very lucky. We have a big security team and we have a number of sections in our team. So we have a section who deals with monitoring incident response. And we have a team dedicated to penetration testing. We have teams dedicated dedicated to, um, like DevSecOps, security engineering. Um, we have a team of architects. And so basically what I do is I rotate through those teams and then I say, okay, like, um, we do basically a quarterly rotation. So once a month, the SOC knows that, uh, on the 4th month of every quarter, so like April, for example, they know that they're expected to give, um, like an update on the current threat landscape, an update on incidents that they can share, um, insights and tips for developers. And Doing a presentation once per month or a talk or a deep dive on something is not that difficult. If you say, hey, once per quarter you need to present, there you go, they can do that. And then, you know, the next month, okay, pen testers, you need to report on findings or changes to the service or give developers some insight into, hey, how can you prepare so that, you know, you have a better pen test? Better being like in quotes, I mean less vulnerabilities or things like this. So the burden of that is suddenly much, much lower because the responsibilities spread much wider or much, much more widely. And nobody can say, ah, I didn't, I didn't realize, you know, I forgot I've got this meeting next week. You know, it comes up once per quarter and, you know, creating 30 minutes of content once per quarter, if you don't have something interesting to talk about for 30 minutes once per quarter, I, you know, maybe you're not doing, uh, uh, some very exciting things, if that makes sense. Um, yeah. That's, that's kind of unfair 'cause obviously some teams are very like, you know, they're just doing operations, but, um, you know, they can talk about projects and, or, and, and findings and, and, and feedback and all, all sorts of things. Um, so I lean on the fact that I have other teams who have bought into the Security Champions, uh, initiative. to basically support that content. And I suspect it's much like running a podcast where you reach out to different people or different people with different experiences, and that helps you create content so that you don't run out of things to talk about. And by the time a year passes for us, a lot has changed. We might have a whole brand new service. The threat landscape may be very different. Technology changes so fast. And also, we do sometimes have security champions themselves come and talk. They'll talk about, like, an experience that they've done or something that they've changed. A good example is, as part of my— one of my training sessions, we talk about auditing access permissions particularly for teams that are working with public cloud environments, making sure that they review their access controls. And one of our security champions basically took that, built a process within his team, automated a lot of it, and basically once per month, it grabs all of the permissions, produces a nice report. He runs through it with the manager and is like, yes, no, yes, no, yes, yes, this person left, remove all of these that are still there. And then did a talk at our security champion meeting, um, you know, a 15-20 minute talk about it. And now slowly other teams are also adopting the same, um, the same process. And the original process there is, yeah, it's a little bit hard-coded, a little bit janky. It works for his team, but, you know, they're obviously collaborating, taking that idea and now, and implementing that. And I think those are some of the things that, um, that we really lean on to make sure that it's not just the same content. And I think when I arrived and originally started to take over the program, it was just we'd have a monthly meeting and it'd be like, please don't forget to do your, you know, submit for penetration tests. Please don't forget to do this. And, you know, and everybody was obviously like, you know, just the numbers were dwindling and things like this.

**33:33 Robert Hurlbut:** Yeah.

**33:34 Alex Olsen:** So, um, so yeah, that's, that's basically how we've made it work and we get really, really high attendance every month, which is great to see. And some people type in the chat because we have hundreds of people attending. It is a bit more like webinar lecture style, but we do still get interaction. We always have a Q&A and we always get questions at the end, which is really, really good to see because if people don't ask questions, then they're probably busy checking their emails and stuff like that. But, you know, people are listening if they're asking questions.

**34:10 Chris Romeo:** Yeah. And I love that idea, the concept here. I think this is something that other people can take away and learn from this conversation. Look at different areas within the security team as potential speakers on a schedule to share the things that they're seeing. And And, you know, everybody's seeing something interesting once a quarter. You know, there's something they can talk about.

**34:36 Alex Olsen:** Yeah.

**34:36 Chris Romeo:** It may be something in the industry. You know, the incident response team may talk about something that's happening in the industry that isn't even impacting your own company, but it's something that's on their radar because they were at a conference and they heard somebody do a keynote on it, you know. So, I love that idea about, you know, kind of spreading the load across the security team and letting everybody come and And you end up with a champions organization that— a set of champions that really knows how security works inside the business because they know, oh, I heard Jane talk about this from this other part of this, you know, the risk team or something, telling us about this, how it all comes together. So, yeah, I think that's a really powerful concept there.

**35:19 Robert Hurlbut:** Yeah.

**35:20 Chris Romeo:** So, let's talk tactically or specifically about individual champions now. What have you seen? What kind of highlights do you have have about individual contributions you've seen? I know you just mentioned one where, you know, at the— where this individual was able to share how they were doing, you know, identity and access management at the cloud level, and it was a collaborative thing. But what other big contributions have you seen that have really made you proud of the champions that you're working with?

**35:49 Alex Olsen:** Yeah. So, one that really comes to mind is for training. So, I'm sure a lot of people can sympathize with this, where people don't get into security because they want to be public speaking, for example, or giving training or whatever. Obviously, a lot of engineers, you know, don't like being in the limelight, and that's okay because we need, you know, obviously people from lots of different backgrounds, lots of different skills. And actually, our security training has 2 challenges. One is Um, uh, having specialist knowledge. So obviously, uh, so I deliver a lot of security training in terms of application security, but for example, if it comes to Android, iOS, it's not my area of expertise. And so I don't really have any business teaching, um, iOS developers, Android developers how to do security. Um, I, I could probably tell them the basics and be like, you know, Watch out for this, or or or or use this tool to to scan. So what we started to do is basically partnering with security champions who were an expert in a certain area. So for example, one of our expert iOS developers, they would team up with one of our penetration testers or security auditors who who had experience pen testing mobile. applications, and together they would basically create the content. So the security champion in this case, having obviously a deep knowledge of the technology stack and having the penetration tester support them, or the security engineer support them in terms of what common threats they see, what kind of findings we have, and then them working together to, to build the training and then deliver it. That's really, really good. And not just from a collaboration and community, it's also kind of like free resource for my team that's responsible for cybersecurity training in that I'm, you know, I'm getting experts to come in and support our training activities. So this has been really, really valuable.

**38:12 Robert Hurlbut:** Yeah.

**38:13 Alex Olsen:** for us, especially for topics, like I say, that, um, we don't necessarily have a specialist engineer or somebody with, with really, really deep knowledge in, in that area. Um, we can collaborate with, um, those security champions or collaborate with the, with the organization. Um, and this is kind of our default training now. So our default training used to be, yeah, security engineers would deliver the training, but I think having, um, again, developers own the training and having us support them, uh, is going back to, again, democratizing, um, AppSec, shifting left, um, and getting people, making them feel really like they're, they're part of security and security is part of their work. It's not just something that we're, we're telling them to do. So, um, when their colleagues, they see, you know, ah, my, my colleague is, is delivering this security training, that's like, that's part of their world. It's not, it's not part of our world anymore, if that, if that makes sense. So this is probably one of the biggest things that, um, uh, I think, uh, that I started, uh, started doing, um, with our, with our Security Champions community. And, um, yeah, it's had really, really big impact and, uh, it's done wonders for, for our training catalog. Our training has, has expanded quite rapidly because Suddenly we have access to expertise that we just, we didn't have before, or would take us a really long time to research, build, learn, and deliver. So that's been really interesting.

**39:53 Chris Romeo:** Yeah, and that's, I think, is a very powerful connection between training programs and champions programs. Previously to Security Journey, I, like I mentioned, I was running Cisco's Security Champion Program, but I was also running the internal education program. And the champions, just like you're describing, were my feeders for my content, my subject matter experts on various topics, because I know what I know. I know what I'm good at. I can bring things together. But just like you're describing, I needed people that had subject matter expertise. We were doing content on hardware security. I don't know anything about hardware security. I know enough to be dangerous about hardware security, not enough to do a 15-minute segment on it. But I brought some people into the conversation and had them help. And so, I think there's got to be a good collaboration. And when I think training programs, the training program for me is bigger than the Champions Program.

**40:52 Robert Hurlbut:** Yeah.

**40:52 Chris Romeo:** But the Champions are a foundational piece, and they're also my target for, of pushing my envelope as high as I can to give them as much knowledge and experience and whatnot about security. So, I don't know if you've had this experience yet, but I've had some people that came through my Champions Program, went through the training programs, came through the Champions Programs, and then they said, I'm going to be a security person now. I'm not going to be a developer anymore. And they transitioned into AppSec. And like, for me—

**41:23 Alex Olsen:** Yeah.

**41:24 Chris Romeo:** That's something that, like, that's just, that's a happy moment. It makes me smile to think this program enabled some people to pick a different career path and go and chase a passion that they found that they had for this topic of security. And so, I think that's really the value of bringing these 2 things together.

**41:42 Alex Olsen:** Yeah, yeah, yeah, definitely. And we've found the same where a few people within the Security Champions community have been, you know, real advocates for security, um, and, and doing really, really great things. And, and some of them obviously, yeah, like you say, kind of considering more of a career in, in security now, or, or others are just being happy being kind of like, um, uh, almost like a security specialist, but still, you know, sitting in their current role. They're like the security guy on, on their team, which is which is really nice to see. Yeah.

**42:20 Chris Romeo:** Yeah. It's a fun part of our jobs to see that come together.

**42:25 Alex Olsen:** One last thing we do with our training, just on that. So when we onboard people, we go through like our orientation training, and we have some other stuff as well, and the welcome stuff. We also do run a Certified Security Champions training or set of training, which sounds, kind of sounds confusing. It's like, hey, you're a Security Champion, but you're not a Certified Security Champion. Naming conventions aside, um, this is because of our internal kind of like certifications. Um, we basically run, um, 5 days of, of training where we cover like a much deeper dive on threat modeling. So, um, uh, afterwards, Security Champions basically should be able to, you know, run a threat modeling session on their own, on their own. Um, we do a couple of days of DevSecOps security automation, but on the, on the sec side, obviously, like the dev and the ops side, not so much, but automating security tools and dealing with them. And then we do a day of, um, uh, secure development. And after that, um, they get an official, uh, like certification from, uh, from our HR team. Oh, uh, which is great because, um, uh, thinking about like the two-way street, obviously they're giving us time and effort and making their applications more secure. Um, if you give somebody a certification which they can take back to their manager and say, hey, I've achieved this, and they can put that onto their— either write that into their goal setting, objective setting, or, or have that as part of their evaluation, you're giving back to them as well. So, um, I think providing evidence of, um, uh, training attendance, training completion, um, contributions to the Security Champion um, program, um, people who, who deliver training for us, people who do talks, making sure that you communicate that back to their manager and making sure that the security champion can make the most of that in terms of, um, that evaluation, um, at work also helps as well. So wherever you have an opportunity to, to give them something, we should always try and, try and jump, uh, on that because I think a lot of times people contribute and then They're kind of like, you don't want any unsung heroes. You want lots of heroes who are—

**44:41 Robert Hurlbut:** Yeah, agree.

**44:42 Alex Olsen:** Sung about or talked about. And yeah, I often email managers or give feedback and say, hey, so-and-so's contributed to this. This has been really great. They did a talk here, or they've achieved this training, and their manager's then aware. And it's not coming from the security champion, from, you know, somebody in, in the cybersecurity team. So myself, I'm a manager. It's coming from like another, another manager who, um, I think that's, it's important to have that kind of recognition. Um, yeah. And that, that's, that's the word I was looking for is, is recognition is really, really important.

**45:18 Robert Hurlbut:** Yeah.

**45:18 Alex Olsen:** If you wanna keep people, um, uh, keep people on board. And to be fair, it's only fair, like giving people recognition for their work. That should be, should be default. So we shouldn't forget about that.

**45:30 Chris Romeo:** Yeah, I don't know, you might have noticed I've been sitting here smiling for the last 30 seconds or so. If you ever have a chance to hear one of the talks I do on Security Champions, it's going to be eerie for you because you and I think so much alike on these things. Like, I say this, like, I'm listening to you, I'm like, I say the same thing he's saying. Like, recognition, of course we email the managers and we do it. Like, so yeah, I mean, I'm just, I love it because it's, it's, I'm learning some, I've learned a number of different things from you that I haven't thought about, but There's also a lot of things that, like, you and I are aligned as far as how we're going to the business with these things and what we're prioritizing there. So, a couple more questions here. Probably the one we should start with if we had executives listening, you know, how to track ROI on things like training and security champions. That would have been the one question they would have been waiting for. That's the only one I showed up for.

**46:19 Alex Olsen:** Yeah.

**46:20 Chris Romeo:** It was the return on investment. So, you know, what are your thoughts on return on investment for champions and training?

**46:26 Alex Olsen:** Yeah, so the easy things to track, you know, if you're just launching a program or if you're in those early stages, and obviously tracking participation is good, creating those basic reports, making sure that you have that in place so you can say, hey, look, we are getting participation. You know, the things we're doing are working. But beyond that, you do need to think about, okay, what processes do I need to put in place? What data do I need to access? And really, there are obviously a lot of moving parts and a lot of things that impact these, but looking at time to remediation for vulnerabilities, looking at service usage for services that your team provide that are not mandatory, you might see like an uptick. So for example, we promote a scanner, we'll obviously look at, okay, how many tickets have come in regarding that. We even track our internal pages. So if somebody comes to our knowledge base, what are they looking at? How many page views are we getting? And when we launch new services or advertise new services, we look at, okay, what's the uptick in terms of scanner usage? What's the uptick in terms of people looking at our knowledge base and looking at those, those sections? Um, kind of a side note, um, all of the scanning services that we, um, we provide, um, took a little bit of work, but every time a developer uses one of our scanners, it basically checks in so we can see like usage over time. Um, it's very easy for our, for our managed scanners, but the ones that where we just provide a Docker image, um, it doesn't pull any data. It just says, hey, I'm scanning a service, you know, or I've been used a few times this month. So we can actually get some reasonable insight into what the usage of our open source or like packaged open source tools are. So those are some of the kind of things that you can put into a report and you can say, okay, over time we've seen a reduction in time to remediation, we've seen a reduction in open vulnerabilities, for example, or the average number of findings over time. And you obviously need to look at, okay, Security Champions is probably not the only thing that's happening. Um, but you can do some really interesting analysis, especially if you're having new teams and new services. Services over time with repeat testing tend to get better. Um, uh, as a, you know, you pen test once, you find a ton of stuff. You pen test twice, you find less. And then you, you kind of like, you know, um, less findings over time. So the time to, from getting to a, um, service that has, you know, lots of open vulnerabilities or is in a pretty immature state, and there's many ways to measure that, to time to being, let's say, compliant or ready for the release, um, over a number of years, you can track that. Um, I know that's quite difficult to do and you need a lot of data and you need a lot of insights, but that's kind of what you should be aiming for long term. And Don't wait 2 years to be like, ah, now we're gonna start looking at time to remediation, or now we're gonna start looking at scanner or service usage. Put it in early. And even if they don't give you the insights you need straight away, in a year's time, that data's gonna be really, really valuable. So I suppose collecting data, putting those processes in place is, it's really, really important. And you can do other things like tracking tickets, tracking questions, Especially if you have something like a chatbot or some kind of messaging service where you're answering questions, you can do some analysis on the number of questions you get. Previously, you might have had, I don't know, 50 inquiries a month about penetration testing. And, you know, you recognize, ah, we're getting asked the same questions again and again. You launch your Security Champions program, you do training with them, suddenly the inquiries drop. you can attribute that to your Security Champions program. That's gonna save you time, save you money, unless again, you have a chatbot and that's handling it all, then obviously you're not saving time. But that's kind of like a basic example. So a lot of this really depends on your setup, depends on your organization, depends what services you're providing. But there are many, many ways to basically track how mature are our target— how our target audience is, and what's the impact over time of our Security Champions program as it changes, as it grows. And again, you can look at specifics if you want to. So if you are pushing for teams to implement a certain tool, you can then look at that data. But you've got to start collecting the data early. Don't roll out the tool and then be like, ah, we need this, all this data later on, and be like, ah, we don't have it, so that sucks, so we can't prove the return on investment, for example. Um, a really good one to look at is a lot of the time, um, we talk about, uh, it's cheaper to find a vulnerability during the development stage rather than finding it as you're trying to release or, or after release. Now that sounds obvious to us, but, um, if you do have, uh, something like Jira, um, where, uh, you're tracking, um, uh, all of the, um, uh, all of your backlog and everything, you can look at, okay, how long does an item in the backlog take once it's— if it's discovered at a late stage of the development process rather than something that's brought up during Uh, during development, and you can do comparisons on those things. And then you can look at how your Security Champions program is impacting the overall, um, uh, uh, where vulnerabilities are, are being discovered, um, basically. So are more of them being discovered earlier in the development lifecycle rather than later on? So you're kind of looking at 2 different points and then And then comparing the two. Um, so there's no easy way to kind of like track ROI, but there is a lot of data that you can use to, to support your, your argument. And if you are using Jira and tickets very closely and tracking hours and things like that, it's quite easy to, um, to pull out, okay, this has saved us like, you know, on average 10 hours per, you know, um, per sprint or something like this per, per engineer.

**53:14 Robert Hurlbut:** Yeah.

**53:14 Alex Olsen:** You can start to, to get that data, but it does take a bit of work.

**53:19 Chris Romeo:** Yeah. Wow. I feel like you've given us a $100,000 consulting engagement here on building security champions. And, you know, I'm going to go back and actually listen to my own interview once we release this because you've given— there's so many things that you've shared in this interview about how to do champions successfully.

**53:44 Robert Hurlbut:** Yeah.

**53:44 Chris Romeo:** And, you know, strategies and things like that. So, this has been super valuable. And I look forward to doing a follow-on conversation where we can dive deeper onto the training side because I'd love to get that appreciation for that as well.

**53:57 Robert Hurlbut:** Yeah, definitely.

**53:58 Chris Romeo:** So, from a key takeaway perspective now, so let's kind of summarize for our audience here. You know, what do you think are some key takeaways that you would offer to our audience? Are there things that you would suggest they do? Or, you know, how would you leave this conversation with them?

**54:14 Alex Olsen:** So I think from me, I would say, having learned from a lot of mistakes, some of the key things that if you're thinking about, you want your Security Champions program to be successful, orientation and training, you've got to have that on point. That really, really helps. Otherwise, confused people won't turn up. Yeah, making people part of your community. Our monthly meeting, I think once I revamped it, became very successful. And I think the shared responsibility for that and the variety of topics is really, really important. Recognition, which I don't think I mentioned before. Obviously, we did talk about recognizing people who had done the training and contributed to the program, but our Security Champions is an official title. So if you look somebody up in our directory, you will also see them. They will have an official title of Security Champion, um, as well. So, so that, that helps. Um, and I think avoiding— so for me, some, some of the harder lessons we had: newsletters didn't really work for us. Um, that was kind of a complete waste of time. Um, Security Champion Awards is a controversial one, um, because recognition is very important, but when you have, you know, let's say 10 people who all do amazing things year on year, it's hard to be diverse in terms of your awards.

**55:45 Robert Hurlbut:** Yeah.

**55:46 Alex Olsen:** So think very carefully about when you start to do rewards— awards. If you want to keep it running long term, you're going to have to figure out— there's always going to be some like A-star students or people who are like really, really good, and obviously you want to reward them, but, um, at the cost of everybody else, it's— it can be difficult. So just, just think carefully. Um, and I think, uh, make sure your security champions, things that they do are shared within the community. So the training, um, that our security champions contribute to, um, the new processes in terms of like, like we, uh, I spoke about before, Um, uh, where we, we look at access control processes and, and things like that. And that's become more widely adopted. Um, and collect data, collect tons and tons of data and, and make sure that you can, when you're ready to do the analysis, that you're not just, you're not just stuck and then be like, oh, we need to wait 6 months before, before we have something meaningful. Um, try and, try and get in early. Even if it's just basic stats, basic usage stats, basic attendance. If you're running a monthly meeting, make sure you keep those numbers because if they trend upwards or trend downwards, that's going to tell you something, right? So you can act on this information. And even for me now, I know that running a security champion meeting on a Monday afternoon, on a Friday afternoon is not a good idea. And there are some other things, some regional differences where we needed to figure out time zones and and other kind of large tech meetings that happened within our organization. So even just getting the timing right was a real something that we couldn't do without the data. So yeah, it's really, really important to track those things.

**57:37 Chris Romeo:** Yeah. Alex, thank you so much for sharing this very wide amount of, like, large amount of information about your experiences. I think it's It's so powerful when we can share those stories with other people. And my hope is that what you've shared today is going to encourage somebody out there right now who's like, I have to start this Champions Program and I don't know what I'm going to do. And they're listening to this and go, I've got ideas now. I know Alex said that if I do this or this worked and this didn't. So, I think this is really powerful. So, definitely look forward to a follow-on conversation.

**58:14 Robert Hurlbut:** Yeah.

**58:15 Chris Romeo:** on this topic and many others, probably for most of the day. But because our listeners have to go back to work or drive in their car or whatever they're doing right now, we'll stop now. And we will definitely pick this up in a part 2 at some point in the future. So, Alex, thanks for being a part of the podcast.

**58:31 Alex Olsen:** Great. Thank you very much. Yeah, I really hope that some people learned from our mistakes as well as things that worked out. So, it's great. Thank you very much for having me on the show.

**58:44 Chris Romeo:** Definitely.

**58:46 Robert Hurlbut:** Thank you. None of the top 50 university programs teach secure coding in their curriculum. At Security Journey, we help enterprises reduce vulnerabilities through application security education for developers and everyone in the SDLC. With over 400 up-to-date lessons created by industry-leading security experts and a programmatic approach that creates security champions, our program has increased AppSec knowledge as much as 80% Visit securityjourney.com to try our training today.

---

Source: https://appsecpodcast.com/alex-olsen-security-champions-empowering-developers-and-appsec-training/