---
title: "Milan Williams -- AppSec Metrics"
url: https://appsecpodcast.com/milan-williams-appsec-metrics/
date: 2025-01-14
duration_seconds: 2176
season: 12
episode: 2
guests: ["Milan Williams"]
topics: ["Building an AppSec Program", "Vulnerabilities and Exploits"]
audio: https://www.buzzsprout.com/1730684/episodes/16431938-milan-williams-appsec-metrics.mp3
video: https://www.youtube.com/watch?v=YF2Y7wCZc-M
transcript: true
---

# Milan Williams -- AppSec Metrics

*January 14, 2025 · 36 min · Season 12, episode 2*

with [Milan Williams](https://appsecpodcast.com/guests/milan-williams/)

on [Building an AppSec Program](https://appsecpodcast.com/topics/appsec-programs/), [Vulnerabilities and Exploits](https://appsecpodcast.com/topics/vulnerabilities/)

[Audio](https://www.buzzsprout.com/1730684/episodes/16431938-milan-williams-appsec-metrics.mp3) · [Video](https://www.youtube.com/watch?v=YF2Y7wCZc-M)

## Show notes

Milan Williams discusses the importance of application security metrics and how to make them both meaningful and actionable. She explains that metrics are crucial for tracking progress in what can often feel like an overwhelming security landscape, and they're valuable for career advancement and securing resources. We discuss metrics categories and several specific metrics that are good to track. Milan shares important principles on the importance of making metrics actionable through storytelling and relating security impacts to real-world consequences for users. Milan Williams is a senior product manager at Semgrep, where she helps security engineers and developers work together to ship secure software. She recently graduated from Harvard University with degrees in computer science and physics.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
We provide application security training for not just your developers, but for all roles in your SDLC.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Milan Williams:  
→ [Quiet Influence](https://jenniferkahnweiler.com/best-introvert-book/quiet-influence/)  
→ [Semgrep](https://semgrep.dev/)

Mentioned in this episode:  
→ [Quiet Influence](https://jenniferkahnweiler.com/best-introvert-book/quiet-influence/)  
→ [Semgrep](https://semgrep.dev/)

Chapters:  
00:00 Meet Milan Williams: AppSec Metrics  
02:58 I like that idea. So, all right, Robert, where are we  
04:45 Do you think people associate metrics with boring  
07:13 Yeah. It's all about OKRs, objectives and key results. And I  
10:33 If you've got that perspective, because it's one of those things  
12:00 For the metrics framework, could you walk us through that a  
15:19 Then, is there, are there additional metrics in the framework  
16:48 Before we go to any of the remaining categories, metrics, so  
22:55 Yeah. So, I took a turn there in the middle. We  
25:54 If it's simple and it's easily, easy to calculate, whichever thing  
27:05 Because that's the one that it's easy to create metrics, but  
30:41 All right, Lon, we have 3 questions to ask about in  
32:23 Very cool. And the final question is around book recommendations. What's  
34:19 Very cool. So, Milan, what do you want to leave our

## Transcript

*6,407 words · assemblyai*

**0:00 Chris Romeo:** Milan Williams is a senior product manager at Semgrep, where she helps security engineers and developers work together to ship secure software. She recently graduated from Harvard University with degrees in computer science and physics. In her free time, you can find her running in San Francisco's Golden Gate Park or enjoying local theater. Milan joins us to discuss all things AppSec metrics.

**0:23 Milan Williams:** She's created a framework that you can apply The Application Security Podcast is brought to you by Security Journey. We provide application security training for not just your developers, but for all roles in your SDLC. Learn more at securityjourney.com.

**0:38 Chris Romeo:** Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am the CEO of Devichi. and also co-host of said podcast. As always, joined by my partner in crime, Robert Hurlbut. Hey, Robert.

**1:03 Robert Hurlbut:** Hey, Chris. Yeah, Robert Hurlbut, Principal Application Security Architect and Threat Modeling Lead at Acquia. And pretty excited about this topic today as we talk to our guests. Very interesting, metrics.

**1:16 Chris Romeo:** Yeah, we never, it's one of those topics, I think this might be the first time we've spoken about metrics.

**1:21 Robert Hurlbut:** I think so.

**1:22 Chris Romeo:** Milan, no pressure, but I got a lot of questions. So, we are super excited to be joined by Milan Williams. And Milan, we always like to jump right into security origin story. How did you get involved in the world of security?

**1:38 Milan Williams:** Yes, absolutely. So, my security origin story started at Semgrep. I'm currently a product manager there where I'm responsible for our SAST product, static application security testing, where I focus on figuring out what we need to secure when it comes to people's first-party code, the code that your engineers write in-house. And so, it's through kind of this opportunity that I've gotten introduced to security and have really, honestly, fallen in love with it over the past almost 4 years now. It's just really a group of people that I find I just have a lot in common with, honestly, and it's really just a mission that Um, I just get excited about waking up every day. There's a lot of really good people that are trying to tackle a really, really difficult problem, um, and are trying to do their best. And so, um, yeah, I've just been excited about trying to help people, uh, through this role.

**2:31 Chris Romeo:** Cool. I've never heard that term first party.

**2:36 Milan Williams:** Huh.

**2:37 Chris Romeo:** That's cool. I like that. I'm gonna, I, I will attribute it to you 2 more times and then I will pretend like I came up with it, which is kind of what I do. Um, no, I mean, we always talk about third-party code is like, just part of our AppSec vocabulary. We just, it rolls off the tongue without even thinking about it. Third party, third party, third party.

**2:54 Robert Hurlbut:** Or fourth party.

**2:55 Chris Romeo:** First party code.

**2:56 Robert Hurlbut:** Yeah.

**2:57 Chris Romeo:** I like that idea. So, all right, Robert, where are we going with this?

**3:01 Robert Hurlbut:** All right. Well, like we said, we're going to dive into metrics today. And so, Milan, we have a couple of parts to this question when I ask you. So, first of all, why are metrics important? And then, as a follow-up, How can we get over the stigma that metrics are boring? But first, why are metrics important?

**3:22 Milan Williams:** Absolutely. So, I have a lot of thoughts about why I think metrics are so important when it comes to security as a whole. I think with any kind of big, oftentimes daunting or intractable problem, it's really difficult to feel like you're making progress. There's so many vulnerabilities that get reported every day. There's so many new issues. It can feel like you're just drowning in a sea of alerts and notifications and incidents. And so, I think, honestly, for kind of our own sanity, metrics are a really great way to make sure you feel like you're making progress, quarter over quarter, month over month. And that's why I feel like metrics can be so important. There's also a very clear career importance to metrics. I've worked now with a number of application security professionals, where, you know, having those metrics come peer review time can be really great for advocating for yourself and making sure that you get the visibility, not only for yourself, but for your team, make sure you're getting the resources and budget that you need. And oftentimes, a lot of business leaders will speak in terms of metrics, that's the language they understand. And so, that's why I feel like metrics are just really so important. It's a skill that, yeah, not a lot of us like, they can be pretty boring a lot of times, but when you do take the time to understand them, to quantify them, to put them in terms of the language that other people around you need, they can be really helpful for your team and your career.

**4:44 Chris Romeo:** Why do you think people associate metrics with boring?

**4:50 Milan Williams:** Yeah, I think all of us have had some kind of metrics presentation or, like, dashboard review where it just feels like a sea of numbers and your eyes kind of glaze over. You're not really processing what all of those numbers really mean in context. And so, I think if you just get a list of numbers without understanding what they mean or what progress looks like, you can just feel like this is boring, this isn't relevant to me, I don't understand what this all means. And so, I think one of the things that can kind of change metrics from being really a lot more interesting is really about providing the context. I think when I talk later about what I think good metrics look like. It's all about being able to tell that story. And what that means is being able to say, like, what is the change over time? A big dashboard that says, like, 10,000 vulnerabilities is not, it's gonna be really hard for a lot of people to understand what that means. But if you have a story that says, like, hey, we started at a million vulnerabilities and we've got it down to 10,000, like, all of a sudden, I'm engaged, I'm interested, like, how did you do that? There's questions that come up naturally when you start to talk about in terms of change over time.

**5:59 Chris Romeo:** Yeah.

**5:59 Milan Williams:** I think that's the biggest way you can start to get metrics out of just the, the big scary number and into something that's a lot more interesting, even with people that aren't close to security.

**6:08 Chris Romeo:** Yeah, I'm, as a small business owner and a few times a small business owner, metrics are crucial to the business side of the house where we have to measure productivity through metrics. We have to measure the state of the business. How much money are we making? How much money are we spending? Like, what's the, what's the difference there? So I don't think metrics are boring. I mean, Robert might, but I actually, he lit up when he was introducing the topic. So I used, and I'm a big fan of Measure What Matters by John Doerr. That's, that's kind of the system that I ended up using that that, you know, was kind of battle-tested at Google and a lot of different places. So, have you ever done it? Have you ever seen Measure What Matters by John Doerr?

**7:00 Milan Williams:** No, I personally haven't, but the, definitely it resonates with, you know, you want to focus on a few things and kind of few core principles when you're defining what your metrics look like. Yeah.

**7:12 Chris Romeo:** Yeah. It's all about OKRs, objectives and key results. And I just found the simplicity of it was, and we can dive more into that too as we go forward. So, based on this, you created this list of AppSec-specific metrics, and I'm curious, what was the source for that? Was it just, you know, came to you in a dream or something? I don't know.

**7:38 Milan Williams:** That would be kind of the deal. Yeah. I wish my dreams were that helpful. No, no. It came from a lot of conversations with people, honestly. I just kept speaking with people at BSides, at RSA, at all of these conferences, and I kept hearing over and over again that people were having a really difficult time justifying to other people in the business, like, hey, I know that this problem is huge. I can see this. It keeps coming up over and over again, but I just can't seem to get buy-in, or I can't seem to get support from anywhere else in my organization, and I'm so frustrated. And as part of my job, I speak with a lot of different customers from like really small startups to like very large enterprises. And to see something come up over and over again, I was like, there's gotta be a solution here. Like, we've gotta figure out some way. The industry just seems like we're really ripe for an opportunity to try to define this really hard and intractable problem. And so, it started with just interviewing different application security leaders, CISOs, and general, like, IT professionals to understand, like, what are the things that you care about? What are you measured on? What do you like about that? What don't you? And what that eventually came into was a list of over 40 different interviews with these different professionals that I combined into a set of, like, what could become a set of distilled kind of gold standard metrics that you can use for your application security program and start to—

**9:05 Chris Romeo:** Yeah.

**9:06 Milan Williams:** use that as a way to get started, even if it's not the thing that you use forever. Organizations grow and change. Oftentimes, just having a place to start can be really helpful to customize. So, that's where it kind of, this kind of interest of mine came out of, just a lot of conversation with people who all had the same problem.

**9:22 Chris Romeo:** You mentioned CISOs and AppSec leaders, who in general tend to be, I would say, more business savvy. because they're in more business conversations. What was the distribution between kind of leadership and folks that are in the trenches doing AppSec?

**9:45 Milan Williams:** Yeah, it's definitely a mix. At the larger enterprises, focused on like AppSec managers, CISOs that are like managing entire program, but at least 10 of them were definitely like one-person, one or two-person shops where There's one person who's both the manager and the practitioner of their application security team. So, it varied. I tried to get, SemGroup has a lot of different types of customer sizes, so kind of reflected that was the people that I talked to. So, wanted to get definitely some perspectives of people that are hands-on, because if folks that are hands-on don't buy the metrics or don't believe in them, it doesn't make sense. It's not like a very good metric at all.

**10:22 Chris Romeo:** Yeah, and that really sets up the metrics framework. to be supportive of small startups to large enterprises.

**10:32 Milan Williams:** Yeah.

**10:33 Chris Romeo:** If you've got that perspective, because it's one of those things, like metrics are one of those things that we often focus on in the large enterprise. We think a large enterprise when we talk about metrics, but I can see how even the smallest startup can, by instituting metrics early, that's how you get from one AppSec person to 2 AppSec people is by saying, let's look at the scope of the problem and how big the challenge is that we have. And back to your earlier point, like, if you don't have data, what do you— how do you make that argument? Well, we need another AppSec person. Okay, what's your proof? Well, I have a feeling. I have a feeling, a gut feeling that we need more people here. But data is really what makes that equation for you though, right?

**11:19 Milan Williams:** Absolutely. It's an incredibly powerful tool to be able to, yeah, to grow your team and even justify prioritization. Even if you're not trying to grow your team, I know budgets are tight right now on a lot of AppSec teams, but even just being able to prioritize your work and justify why you're investing in certain areas, metrics can be a really great way to say, like, this is why we're focusing on SQL injections this month, because it's the number one most common vulnerability class in our code base, in our production repos. Having that data to back up what you're working on is like, oh yeah, that makes sense. That's why you're doing this investment, and that's why you're doing this sprint. It can kind of stave off some of those questions around like, what is security doing?

**12:00 Robert Hurlbut:** For the metrics framework, could you walk us through that a little bit about how it's put together and approach and so forth?

**12:08 Milan Williams:** Yeah, absolutely. So, the way I was thinking about this, metrics framework is really in kind of 2 pieces. One set of metrics that are what I call pulse metrics, which focus on things that are happening now, like how many vulnerabilities are in production, what are the kind of urgent things I need to address, set of metrics that are really focused, that are good for when you are coming into a new organization, coming into a new team, or just really trying to diagnose a larger issue. And then a second set of metrics, what I call like path metrics, which are more about charting a long-term path. So, once you've identified, say, hey, there's a lot of SQL injections that are in production right now, I identified that through my Pulse metrics, what's the path to actually burning that down over time? How are we gonna pay down that backlog? What are the guardrails that we're gonna put in place to make sure that we don't keep shipping these kinds of queries into production? So, that framework, I think, has been really useful to kind of ground any metrics, but really can kind of help put things into perspective for different audiences. Again, even folks who have no concept of security, they can understand, this is what we're doing today, this is what we're trying to do tomorrow. That structure has resonated well.

**13:22 Chris Romeo:** Okay. So, we, you mentioned kind of the vulnerabilities, vulnerability metrics, and then the, you call them the path. metrics, which were like to focus on something like SQL injection and how we're going the path down. So, are those the 2 primary categories, or are there additional categories?

**13:44 Milan Williams:** Yeah, so within, like, the kind of today-style metrics, things that are happening now, the key ones that I've found that have been very effective is, first and foremost, just tracking your vulnerability backlog, like how many things are in production, right now, and specifically tracking that, as I mentioned, the change over time. I think being able to, again, tell that story between this is how many things are in production, it's been going up consistently over the past month or so, that has been a very, very powerful metric to be able to communicate. A second one that's similarly related to pulse of how well we're doing today is being able to group these vulnerabilities not only by this is how much stuff we have in production, but being able to say, this is the vulnerability classes, this is how many SQL injections, this is how many cross-site scripting vulnerabilities. Those 2 together, how many vulnerabilities we have and what are the most common classes, helps paint a really good story for an investment or say like, hey, if we wanna tackle both, again, SQL injections and cross-site scripting in the same quarter, we need another AppSec person. There's no way we're gonna be able to solve this problem at the same time. given the people we have, and given the amount of growth that we have. And so, I think while they're honestly pretty simple, I think even just talking about them, we get what they are. It's shocking like how few teams are able to instrument them or put them into place. And so, just those 2 alone, I found can be very, very helpful to start the conversation. Okay.

**15:18 Chris Romeo:** And then, is there, are there additional metrics in the framework?

**15:23 Milan Williams:** Yeah, so on the preventative side, so thinking about once you've identified what are the common trends within your application security program, the second piece is really about tracking your developer engagement and your kind of prevention. So the metrics that I recommend tracking at that point are really around twofold. Like one, how well are you able to prevent things from going into production? So you may know SQL injections, you've now categorized all of your SQL injection vulnerabilities, you can detect them through some kind of mechanism in your organization. You wanna be able to see how many of those are you actually preventing from getting out to production. Ideally, your production backlog isn't just increasing over time, but you have some kind of mechanisms in place with a tool or somewhere else to be able to detect and say, hey, these are the amount of things that we prevented, whether it's secrets, whether it's other kinds of vulnerabilities, and eventually increase that over time. That's really the sign of a successful kind of shift-lefted security program. And then similar, you wanna track your developer engagement. How many things are being ignored? How many things are getting actually fixed? And how many things are being left to stay open? And with those 2 metrics, both like how many things were prevented and how many things got fixed, both of those help you figure out how well are we actually making sure that we're moving the security posture forward overall?

**16:48 Chris Romeo:** Before we go to any of the remaining categories, metrics, so the developer engagement one, I'm curious about your thoughts in regards to, I see how metrics could be used to create a negative culture, especially in working with developers, So, if those, like, what, I guess, any thoughts on that? And I'm thinking about, like, the company that says, we're gonna use these AppSec metrics to figure out what developers don't need to be writing code here anymore because they're the worst offenders of SQL injection. So, we've been, we use that as an example, right? So, like, what are your thoughts on how you keep a, how you keep the metrics culturally positive versus negative?

**17:39 Milan Williams:** Yeah. Absolutely, absolutely. There's, I think the biggest thing when rolling out these kinds of metrics is being really honest that it's gonna be a partnership between the engineering team and the security team.

**17:54 Chris Romeo:** Mm-hmm.

**17:55 Milan Williams:** No one's trying to just, here's a bunch of vulnerabilities, throw them over the fence, this is now your problem to deal with. It's really gonna be a good, healthy joint effort between the two. But I think it does obviously create, you know, highlight spots of where there's been gaps in the existing security program. And so, that's a natural thing that will come when you start to detect things like these metrics in more detail. But I think the way to do the messaging that I've done with folks in the past is really around, hey, we're going to do a campaign together to tackle this big security vulnerability. This is not pointing fingers at anyone who didn't know this before. We haven't talked, maybe we haven't talked about this before, maybe we didn't make this as big of a priority, but we are now, and here's materials and guidance for how to remediate these kinds of vulnerabilities going forward. And hey, if it's still an issue at that point, you know, if you're still committing secrets directly into the codebase 6 months after we rolled out our program, you know, that's, you know, something we want to address. But I think coming at it from a very collaborative perspective of we're trying to solve this problem together, makes it a lot clearer that we're not trying to point blame at anyone. Security problems are hard, are really, really hard to, to nail down. And so, it's going to take everybody to really work together to make it happen.

**19:09 Chris Romeo:** Yeah. Yeah, I like that as a, as a general philosophy. Like, I've never been a proponent of security being kind of the, like, things like metrics being used to, in a negative, what I would think of as a negative way to, uh, and, and I was in the training space for a long time and people always clamored for some type of a developer score that they could use.

**19:41 Milan Williams:** Oh.

**19:41 Chris Romeo:** And I didn't, I just didn't like it because it's like, I'm, I'm a teacher at heart. Like, my goal is to get people to learn about these things so that they'll change their approaches, and they'll build better software because developers are engineers, and engineers in general want to build better things. Like, I've never met a developer/engineer who was like, I want to make something that's worse. Yeah, absolutely not. Like, they want to create better things. They don't want to create buggy, insecure code. But, but it's, but you can also create a culture where there is an advantage to not having security things be found. And like, we don't ever want to have that as a culture. Like, we want to be, we got to know, if we don't know about it, we can't fix it. So, we have to have an environment where people are like, hey, this is, this is friendly. I can talk to security.

**20:38 Milan Williams:** Yes.

**20:38 Chris Romeo:** And they're not going to, they're not going to you know, have me written up by HR or, you know, negatively impact my job, my review, my yearly review, because I tried to do the right thing, but it just didn't work out.

**20:52 Milan Williams:** Absolutely. And I think there's a lot of opportunities, honestly, I've seen with metrics where you're able to highlight wins, like teams that do really well. That can be a huge motivator for engineering teams, for development teams, to be able to say, hey, We've burned down our backlog by this much this month. Uh, there's a great opportunity to highlight folks as well, um, when you're able to track things more closely. So, but definitely all about a collaborative approach and complete blameless culture. We're just trying to make the software better together.

**21:22 Robert Hurlbut:** Yeah.

**21:22 Chris Romeo:** Yeah. And competition's good. Now we can use metrics to drive competition up. In my world, competition's healthy.

**21:30 Milan Williams:** Yes.

**21:31 Chris Romeo:** Because I don't do it from, there's not a negative, but like, I remember there was one day at a previous technology company I worked at called Cisco, where we shared some metrics. We had collected a number of different programmatic-style product/application security-style metrics, and we created dashboards, and then we shared them with the senior leadership. And senior leaders are competitive people. That's how they got to where they are.

**22:01 Milan Williams:** Yes.

**22:02 Chris Romeo:** And there were a number of operations directors, who got phone calls after that meeting and were just told, I don't care how you do it, fix it so that I'm not on the bottom of the list. And so, in that case, that was a, that was a healthy competition.

**22:16 Milan Williams:** Absolutely.

**22:18 Chris Romeo:** But, yeah, we want to make sure comp— we want to make sure metrics are a positive thing, because it's all about culture. It's a culture. This whole thing about security is a culture game.

**22:26 Milan Williams:** Absolutely. Absolutely. I'm huge on— one of the other things, going back to how to make metrics not boring, is gamification, trying to Make it really easy to understand. That's why I think simple metrics are good, because people can understand what they are and understand what they need to do to drive them up or drive them down. And yeah, I think that's a great example, you know, categorized by different teams, try to put different teams against each other and see what happens. You know, those are really great ways to just drive adoption.

**22:55 Chris Romeo:** Yeah. So, I took a turn there in the middle. We were in the middle of walking through the metrics framework. So, We talked about kind of vulnerability metrics. We talked about path metrics. I talked about developer engagement. I like what you were saying about the kind of, I guess, almost pipeline metrics to make sure that things aren't sneaking through and whatnot. So, what else do we have then in our metrics framework? What other components?

**23:26 Milan Williams:** Yeah. The last one that I always think is important, but is part of just making sure that folks track, is just your fix rate. It's, again, pretty simple. Again, it's— but I think it can be a really powerful one. I know we focus a lot on all these alerts, all of these different notifications that come in throughout all of our different tooling that we have, but really focusing on what has actually got fixed this month, how did we actually move security forward? I think that's a very, very powerful metric to be able to show progress, again, celebrate wins, and it can be another very powerful one to justify increase in the team or budget, whatever you're looking for. The last one that I include as part of the framework is around compliance. While it's kind of a necessary part of a lot of security programs, is really making sure you're tracking your SLAs. So if you have, Typical teams that I interact with either have 30, 60, or 90-day SLAs for different vulnerability classes, depending on the severity of the vulnerability. And so, making sure that you stay within your compliance requirements, if you do have them, or anything that you've promised your customers, it's a good baseline one to have. The last thing you want is for a customer to come to you and say, hey, why hasn't this been fixed? You're out of your SLA compliance. That's a whole a whole problem. So, just making sure those are tracked early and often is a good one.

**24:54 Chris Romeo:** Well, yeah, yeah. This is a solid collection of different pieces that I could see how these can very easily be implemented. And I love the fact that you hit on simplicity because that's how I've survived my entire career. 'Cause if you make it too complicated, I can't figure it out. I don't understand what— it's all about that. Let's keep it simple. We'll go with the silly version. Keep it silly or keep it simple silly.

**25:27 Robert Hurlbut:** Yeah.

**25:28 Chris Romeo:** We're kid for, you know, in case those people have kids listening in the car, you know, we want them to be future application security professionals. But yeah, simplicity is key because if you make it, it's so easy to say, when something's challen— or something's complicated, it's so easy to say, eh, that's too complicated.

**25:49 Robert Hurlbut:** Right.

**25:50 Chris Romeo:** For everybody in the equation, including me, I can say that's just too complicated.

**25:53 Milan Williams:** Yeah.

**25:54 Chris Romeo:** But if it's simple and it's easily, easy to calculate, whichever thing you described there, I don't have to be a PhD in mathematics to figure out how to, how to put this into a system. Yes. And read it out. So, yeah. So, I mean, that's, so that's by design though, right? You—

**26:10 Milan Williams:** Yes.

**26:10 Chris Romeo:** Simplicity was one of your core principles, I'm guessing, in this process.

**26:14 Milan Williams:** Yes. It's all about— and I think the reason why that matters so much is because in talking to a lot of security folks, it's one of the— maybe surprisingly, or known to folks in the industry, is it's an organization that interfaces with almost every other aspect of the business. You talk with legal, you talk with product, you talk with engineering, you talk with executives. And so, having metrics that can speak to all those different languages is really important. So, that's why I think simplicity is so important. Making sure, one, it's easy to track, easy to implement, but also just easy to understand. Again, if someone isn't in security on a day-to-day basis, they can still get what you've been doing and what you've been working on and can support your team.

**26:55 Chris Romeo:** All right, so here's the million-dollar question, Milan.

**26:59 Milan Williams:** Yes.

**27:00 Chris Romeo:** How are we going to make these things actionable?

**27:03 Milan Williams:** Yeah.

**27:04 Chris Romeo:** Because that's the one that it's easy to create metrics, but you've already shared a couple of different examples of, You know, having that positive program, let's get a program together this month where we're focusing on squashing SQL injection. What else can we do to make these actionable so that they don't just become the numbers that get calculated about what we're doing?

**27:27 Milan Williams:** Yes. Yeah, I think this is where I think some, again, very simple fundamental aspects, but one of the things that has been incredibly powerful for my customers that I've worked with is really just having a centralized place to view all of them. You can have a list of 5, 6 metrics, keep it again really simple. I think that makes sure that everybody is aligned on what the metrics are, what our goals are we're trying to move towards. I think it just keeps it very, very easy to understand what we're trying to do. If you have a clear baseline and a clear goal, the team can work their way to figure out how to make that goal happen. The other thing that comes along with metrics that I mentioned at the beginning, but it's just around storytelling. It's maybe an underrated skill that I think is actually really, really crucial in security, but really being able to paint the picture of like why it matters, why security matters in your organization. That story's gonna be different for everyone, but— Yeah. For example, one common one that I worked on with someone, a customer of ours in fintech, was working around how can we talk about why this matters to our customer, framing how a vulnerability can impact their day, putting things in terms of not only what this vulnerability is, but says, what would happen to their financial information? How does this impact their future financial prospects? how do these vulnerabilities impact their day-to-day life? I think it puts it in a really human level that people can understand. And again, that's going to be different for every organization. But if you think about the user of like, why does this metric matter? Why do we want to protect them from whatever it is? That can be a really powerful kind of human motivator for people to understand why these metrics are important.

**29:18 Chris Romeo:** Yeah, I love the idea of storytelling. Such a powerful thing. And we as human beings are storytellers by nature. Like, that's the things you remember are stories that you can go to a full-day training and the one thing you'll remember is the story the instructor said about this or that. You won't remember like, well, what facts did they share with you?

**29:44 Milan Williams:** I don't know.

**29:45 Chris Romeo:** There were some facts. I'm sure there were facts presented on the slides, but if there's a story, that's what sticks with you. And that's a great point about using storytelling in metrics to be able to To make, so people can, can understand the context. They can understand what, what actually happened. Like, we should, you should be a lot more excited. Let me explain to you why you should be a lot more excited about this, because this is actually a very powerful movement we've had. And it's, it's about layering on the context and, and telling the story.

**30:14 Milan Williams:** Yes, absolutely. There's, it's just a great way to get a lot of people, people involved. Anybody can listen to a story and can understand. start to get bought in and get excited and want to help you, honestly. I know for me personally, that's always a huge motivator.

**30:28 Chris Romeo:** Yeah.

**30:30 Robert Hurlbut:** All right.

**30:31 Chris Romeo:** Well, we've reached the lightning round where Robert shines all the time, but this is where he especially shines is during the lightning round.

**30:40 Robert Hurlbut:** All right, Lon, we have 3 questions to ask about in our lightning round. First is, what's your most controversial opinion on application security, and why do you hold this view?

**30:52 Milan Williams:** Yeah, my most controversial opinion, I think, about AppSec is really about, I'm a huge believer in prevention. I think a lot of application security, we can talk a lot about what is in production now, what is burning today, which is important, but I think we undervalue the importance of setting ourselves up for long-term success. We do a lot of firefighting. a lot of just trying to address the here and now. And I think we could save ourselves and save our engineering team a lot of time by setting up systems to be proactive instead of a lot more reactive. Examples of this look like investing in training, investing in really clear areas to make it clear for how to fix these kinds of vulnerabilities, or tooling to make sure you can fix things before they hit production. I think, Yeah, I think in general, that's my like most controversial opinion in AppSec, just really an overinvestment in the proactive side of things to decrease the load later down the line.

**31:55 Chris Romeo:** Okay, thank you.

**31:58 Robert Hurlbut:** So our second one is, what would it say if you could display a single message on a billboard at the RSA or Black Hat conference?

**32:06 Milan Williams:** Yeah, we've been talking about metrics, so I think it would have to say something around metrics. Maybe there's some kind of slogan around like track your metrics, something like that. Make sure you're tracking those. I think that would be my slogan.

**32:22 Robert Hurlbut:** Very cool. And the final question is around book recommendations. What's your top book recommendation? But also, it doesn't have to be technical, but we want to know why do you find it valuable?

**32:36 Milan Williams:** Yeah. I recently, semi-recently read this book called Quiet. I believe it's like an introvert's guide to being in the workplace, something like that. And it's really good. I think it's by Susan Cain. And essentially what she does is she talks through how to take advantage of introverts that work on your teams or that you work with. I'm an introvert by nature, and And so, uh, I found it a really powerful book, um, especially I work with a lot of engineering teams and we're all kind of introverted by nature. And so, I just found it a really reflective book to think about, like, what are my strengths? What are the unique ways that I can contribute to a team that can be really extroverted? Um, so yeah, that would be my, my book recommendation of descent.

**33:22 Chris Romeo:** That's interesting. I'm gonna add that to my holiday reading list.

**33:26 Milan Williams:** Yeah.

**33:26 Chris Romeo:** 'Cause that, that sounds something like something that'd be very practical.

**33:30 Robert Hurlbut:** Yeah.

**33:30 Chris Romeo:** To, because I, I don't know, I kind of switch between being an extrovert and an introvert depending on the time of day and other things. But yeah, it's, if you could just understand how to, I like that, that idea of how do you unlock some of the capabilities that exist in your team that if you're not reaching out to introverts, they can They can have the best ideas, but you're not getting them because they're not gonna be like, let me talk, let me talk. Like some of us are.

**34:02 Milan Williams:** Yeah, it's a great book. I think she just gave a lot of good tactics and also for introverts to figure out like how you can better, you know, tap into the parts of yourself that you may not usually do, especially in a business context, things that you wanna work on or be better at. So I thought it was a really great read.

**34:18 Chris Romeo:** Very cool. So, Milan, what do you want to leave our audience with, from as far as a key takeaway, perhaps a call to action? What would you, how would you land the plane here?

**34:30 Milan Williams:** Yes, I think my number one key takeaway is if you don't have a metrics program today for your application security program, today's the best day to get started with one. Keep it simple, don't make things super complicated. Use stories as a way to get buy-in and get people excited about the metrics that you do present. And then celebrate the wins. I'm sure there are things, once you start to get these spun up, you'll start to see a big change in your security program, and make sure to be loud about it. Use them as a way to advocate for yourself and for your team to make sure you expand your team or expand your budget, whatever you need to achieve your goals as a security professional.

**35:09 Chris Romeo:** Very, very cool. Well, Milan, thank you for sharing your wisdom and insight here from this whole metrics project. And I think it was a great, collection of metrics that people can use. And definitely looking forward to having you back on the show again as a guest, because I want to, I want to dive into product management.

**35:31 Milan Williams:** Yeah, I'm happy to.

**35:32 Chris Romeo:** Which is not an area that we spend a lot of time as security professionals understanding and kind of using as a resource. And so, I'd love to have another conversation in the future just to, just to go down that whole realm of, because we've, we've, we've, we did an interview with Tony Quadros about the AppSec salesperson.

**35:57 Milan Williams:** Yeah.

**35:57 Chris Romeo:** Which was good because it was, it was, it was, and I, I learned a lot from, you know, because how salespeople in the industry think and, and whatnot, how they interact with security. So we'll do that again in the future. We'll have you come on and just talk. We'll talk product management.

**36:12 Milan Williams:** Happy to. Happy to. Thank you for having me.

**36:14 Chris Romeo:** Yeah. Thank you.

---

Source: https://appsecpodcast.com/milan-williams-appsec-metrics/