Milan Williams -- AppSec Metrics

0:00Chris RomeoMilan Williams is a senior product manager at Semgrep, where she helps security engineers and developers work together to ship secure software. She recently graduated from Harvard University with degrees in computer science and physics. In her free time, you can find her running in San Francisco's Golden Gate Park or enjoying local theater. Milan joins us to discuss all things AppSec metrics.

0:23Milan WilliamsShe's created a framework that you can apply The Application Security Podcast is brought to you by Security Journey. We provide application security training for not just your developers, but for all roles in your SDLC. Learn more at securityjourney.com.

0:38Chris RomeoHey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am the CEO of Devichi. and also co-host of said podcast. As always, joined by my partner in crime, Robert Hurlbut. Hey, Robert.

1:03Robert HurlbutHey, Chris. Yeah, Robert Hurlbut, Principal Application Security Architect and Threat Modeling Lead at Acquia. And pretty excited about this topic today as we talk to our guests. Very interesting, metrics.

1:16Chris RomeoYeah, we never, it's one of those topics, I think this might be the first time we've spoken about metrics.

1:21Robert HurlbutI think so.

1:22Chris RomeoMilan, no pressure, but I got a lot of questions. So, we are super excited to be joined by Milan Williams. And Milan, we always like to jump right into security origin story. How did you get involved in the world of security?

1:38Milan WilliamsYes, absolutely. So, my security origin story started at Semgrep. I'm currently a product manager there where I'm responsible for our SAST product, static application security testing, where I focus on figuring out what we need to secure when it comes to people's first-party code, the code that your engineers write in-house. And so, it's through kind of this opportunity that I've gotten introduced to security and have really, honestly, fallen in love with it over the past almost 4 years now. It's just really a group of people that I find I just have a lot in common with, honestly, and it's really just a mission that Um, I just get excited about waking up every day. There's a lot of really good people that are trying to tackle a really, really difficult problem, um, and are trying to do their best. And so, um, yeah, I've just been excited about trying to help people, uh, through this role.

2:31Chris RomeoCool. I've never heard that term first party.

2:36Milan WilliamsHuh.

2:37Chris RomeoThat's cool. I like that. I'm gonna, I, I will attribute it to you 2 more times and then I will pretend like I came up with it, which is kind of what I do. Um, no, I mean, we always talk about third-party code is like, just part of our AppSec vocabulary. We just, it rolls off the tongue without even thinking about it. Third party, third party, third party.

2:54Robert HurlbutOr fourth party.

2:55Chris RomeoFirst party code.

2:56Robert HurlbutYeah.

2:57Chris RomeoI like that idea. So, all right, Robert, where are we going with this?

3:01Robert HurlbutAll right. Well, like we said, we're going to dive into metrics today. And so, Milan, we have a couple of parts to this question when I ask you. So, first of all, why are metrics important? And then, as a follow-up, How can we get over the stigma that metrics are boring? But first, why are metrics important?

3:22Milan WilliamsAbsolutely. So, I have a lot of thoughts about why I think metrics are so important when it comes to security as a whole. I think with any kind of big, oftentimes daunting or intractable problem, it's really difficult to feel like you're making progress. There's so many vulnerabilities that get reported every day. There's so many new issues. It can feel like you're just drowning in a sea of alerts and notifications and incidents. And so, I think, honestly, for kind of our own sanity, metrics are a really great way to make sure you feel like you're making progress, quarter over quarter, month over month. And that's why I feel like metrics can be so important. There's also a very clear career importance to metrics. I've worked now with a number of application security professionals, where, you know, having those metrics come peer review time can be really great for advocating for yourself and making sure that you get the visibility, not only for yourself, but for your team, make sure you're getting the resources and budget that you need. And oftentimes, a lot of business leaders will speak in terms of metrics, that's the language they understand. And so, that's why I feel like metrics are just really so important. It's a skill that, yeah, not a lot of us like, they can be pretty boring a lot of times, but when you do take the time to understand them, to quantify them, to put them in terms of the language that other people around you need, they can be really helpful for your team and your career.

4:44Chris RomeoWhy do you think people associate metrics with boring?

4:50Milan WilliamsYeah, I think all of us have had some kind of metrics presentation or, like, dashboard review where it just feels like a sea of numbers and your eyes kind of glaze over. You're not really processing what all of those numbers really mean in context. And so, I think if you just get a list of numbers without understanding what they mean or what progress looks like, you can just feel like this is boring, this isn't relevant to me, I don't understand what this all means. And so, I think one of the things that can kind of change metrics from being really a lot more interesting is really about providing the context. I think when I talk later about what I think good metrics look like. It's all about being able to tell that story. And what that means is being able to say, like, what is the change over time? A big dashboard that says, like, 10,000 vulnerabilities is not, it's gonna be really hard for a lot of people to understand what that means. But if you have a story that says, like, hey, we started at a million vulnerabilities and we've got it down to 10,000, like, all of a sudden, I'm engaged, I'm interested, like, how did you do that? There's questions that come up naturally when you start to talk about in terms of change over time.

5:59Chris RomeoYeah.

5:59Milan WilliamsI think that's the biggest way you can start to get metrics out of just the, the big scary number and into something that's a lot more interesting, even with people that aren't close to security.

6:08Chris RomeoYeah, I'm, as a small business owner and a few times a small business owner, metrics are crucial to the business side of the house where we have to measure productivity through metrics. We have to measure the state of the business. How much money are we making? How much money are we spending? Like, what's the, what's the difference there? So I don't think metrics are boring. I mean, Robert might, but I actually, he lit up when he was introducing the topic. So I used, and I'm a big fan of Measure What Matters by John Doerr. That's, that's kind of the system that I ended up using that that, you know, was kind of battle-tested at Google and a lot of different places. So, have you ever done it? Have you ever seen Measure What Matters by John Doerr?

7:00Milan WilliamsNo, I personally haven't, but the, definitely it resonates with, you know, you want to focus on a few things and kind of few core principles when you're defining what your metrics look like. Yeah.

7:12Chris RomeoYeah. It's all about OKRs, objectives and key results. And I just found the simplicity of it was, and we can dive more into that too as we go forward. So, based on this, you created this list of AppSec-specific metrics, and I'm curious, what was the source for that? Was it just, you know, came to you in a dream or something? I don't know.

7:38Milan WilliamsThat would be kind of the deal. Yeah. I wish my dreams were that helpful. No, no. It came from a lot of conversations with people, honestly. I just kept speaking with people at BSides, at RSA, at all of these conferences, and I kept hearing over and over again that people were having a really difficult time justifying to other people in the business, like, hey, I know that this problem is huge. I can see this. It keeps coming up over and over again, but I just can't seem to get buy-in, or I can't seem to get support from anywhere else in my organization, and I'm so frustrated. And as part of my job, I speak with a lot of different customers from like really small startups to like very large enterprises. And to see something come up over and over again, I was like, there's gotta be a solution here. Like, we've gotta figure out some way. The industry just seems like we're really ripe for an opportunity to try to define this really hard and intractable problem. And so, it started with just interviewing different application security leaders, CISOs, and general, like, IT professionals to understand, like, what are the things that you care about? What are you measured on? What do you like about that? What don't you? And what that eventually came into was a list of over 40 different interviews with these different professionals that I combined into a set of, like, what could become a set of distilled kind of gold standard metrics that you can use for your application security program and start to—

9:05Chris RomeoYeah.

9:06Milan Williamsuse that as a way to get started, even if it's not the thing that you use forever. Organizations grow and change. Oftentimes, just having a place to start can be really helpful to customize. So, that's where it kind of, this kind of interest of mine came out of, just a lot of conversation with people who all had the same problem.

9:22Chris RomeoYou mentioned CISOs and AppSec leaders, who in general tend to be, I would say, more business savvy. because they're in more business conversations. What was the distribution between kind of leadership and folks that are in the trenches doing AppSec?

9:45Milan WilliamsYeah, it's definitely a mix. At the larger enterprises, focused on like AppSec managers, CISOs that are like managing entire program, but at least 10 of them were definitely like one-person, one or two-person shops where There's one person who's both the manager and the practitioner of their application security team. So, it varied. I tried to get, SemGroup has a lot of different types of customer sizes, so kind of reflected that was the people that I talked to. So, wanted to get definitely some perspectives of people that are hands-on, because if folks that are hands-on don't buy the metrics or don't believe in them, it doesn't make sense. It's not like a very good metric at all.

10:22Chris RomeoYeah, and that really sets up the metrics framework. to be supportive of small startups to large enterprises.

10:32Milan WilliamsYeah.

10:33Chris RomeoIf you've got that perspective, because it's one of those things, like metrics are one of those things that we often focus on in the large enterprise. We think a large enterprise when we talk about metrics, but I can see how even the smallest startup can, by instituting metrics early, that's how you get from one AppSec person to 2 AppSec people is by saying, let's look at the scope of the problem and how big the challenge is that we have. And back to your earlier point, like, if you don't have data, what do you— how do you make that argument? Well, we need another AppSec person. Okay, what's your proof? Well, I have a feeling. I have a feeling, a gut feeling that we need more people here. But data is really what makes that equation for you though, right?

11:19Milan WilliamsAbsolutely. It's an incredibly powerful tool to be able to, yeah, to grow your team and even justify prioritization. Even if you're not trying to grow your team, I know budgets are tight right now on a lot of AppSec teams, but even just being able to prioritize your work and justify why you're investing in certain areas, metrics can be a really great way to say, like, this is why we're focusing on SQL injections this month, because it's the number one most common vulnerability class in our code base, in our production repos. Having that data to back up what you're working on is like, oh yeah, that makes sense. That's why you're doing this investment, and that's why you're doing this sprint. It can kind of stave off some of those questions around like, what is security doing?

12:00Robert HurlbutFor the metrics framework, could you walk us through that a little bit about how it's put together and approach and so forth?

12:08Milan WilliamsYeah, absolutely. So, the way I was thinking about this, metrics framework is really in kind of 2 pieces. One set of metrics that are what I call pulse metrics, which focus on things that are happening now, like how many vulnerabilities are in production, what are the kind of urgent things I need to address, set of metrics that are really focused, that are good for when you are coming into a new organization, coming into a new team, or just really trying to diagnose a larger issue. And then a second set of metrics, what I call like path metrics, which are more about charting a long-term path. So, once you've identified, say, hey, there's a lot of SQL injections that are in production right now, I identified that through my Pulse metrics, what's the path to actually burning that down over time? How are we gonna pay down that backlog? What are the guardrails that we're gonna put in place to make sure that we don't keep shipping these kinds of queries into production? So, that framework, I think, has been really useful to kind of ground any metrics, but really can kind of help put things into perspective for different audiences. Again, even folks who have no concept of security, they can understand, this is what we're doing today, this is what we're trying to do tomorrow. That structure has resonated well.

13:22Chris RomeoOkay. So, we, you mentioned kind of the vulnerabilities, vulnerability metrics, and then the, you call them the path. metrics, which were like to focus on something like SQL injection and how we're going the path down. So, are those the 2 primary categories, or are there additional categories?

13:44Milan WilliamsYeah, so within, like, the kind of today-style metrics, things that are happening now, the key ones that I've found that have been very effective is, first and foremost, just tracking your vulnerability backlog, like how many things are in production, right now, and specifically tracking that, as I mentioned, the change over time. I think being able to, again, tell that story between this is how many things are in production, it's been going up consistently over the past month or so, that has been a very, very powerful metric to be able to communicate. A second one that's similarly related to pulse of how well we're doing today is being able to group these vulnerabilities not only by this is how much stuff we have in production, but being able to say, this is the vulnerability classes, this is how many SQL injections, this is how many cross-site scripting vulnerabilities. Those 2 together, how many vulnerabilities we have and what are the most common classes, helps paint a really good story for an investment or say like, hey, if we wanna tackle both, again, SQL injections and cross-site scripting in the same quarter, we need another AppSec person. There's no way we're gonna be able to solve this problem at the same time. given the people we have, and given the amount of growth that we have. And so, I think while they're honestly pretty simple, I think even just talking about them, we get what they are. It's shocking like how few teams are able to instrument them or put them into place. And so, just those 2 alone, I found can be very, very helpful to start the conversation. Okay.

15:18Chris RomeoAnd then, is there, are there additional metrics in the framework?

15:23Milan WilliamsYeah, so on the preventative side, so thinking about once you've identified what are the common trends within your application security program, the second piece is really about tracking your developer engagement and your kind of prevention. So the metrics that I recommend tracking at that point are really around twofold. Like one, how well are you able to prevent things from going into production? So you may know SQL injections, you've now categorized all of your SQL injection vulnerabilities, you can detect them through some kind of mechanism in your organization. You wanna be able to see how many of those are you actually preventing from getting out to production. Ideally, your production backlog isn't just increasing over time, but you have some kind of mechanisms in place with a tool or somewhere else to be able to detect and say, hey, these are the amount of things that we prevented, whether it's secrets, whether it's other kinds of vulnerabilities, and eventually increase that over time. That's really the sign of a successful kind of shift-lefted security program. And then similar, you wanna track your developer engagement. How many things are being ignored? How many things are getting actually fixed? And how many things are being left to stay open? And with those 2 metrics, both like how many things were prevented and how many things got fixed, both of those help you figure out how well are we actually making sure that we're moving the security posture forward overall?

16:48Chris RomeoBefore we go to any of the remaining categories, metrics, so the developer engagement one, I'm curious about your thoughts in regards to, I see how metrics could be used to create a negative culture, especially in working with developers, So, if those, like, what, I guess, any thoughts on that? And I'm thinking about, like, the company that says, we're gonna use these AppSec metrics to figure out what developers don't need to be writing code here anymore because they're the worst offenders of SQL injection. So, we've been, we use that as an example, right? So, like, what are your thoughts on how you keep a, how you keep the metrics culturally positive versus negative?

17:39Milan WilliamsYeah. Absolutely, absolutely. There's, I think the biggest thing when rolling out these kinds of metrics is being really honest that it's gonna be a partnership between the engineering team and the security team.

17:54Chris RomeoMm-hmm.

17:55Milan WilliamsNo one's trying to just, here's a bunch of vulnerabilities, throw them over the fence, this is now your problem to deal with. It's really gonna be a good, healthy joint effort between the two. But I think it does obviously create, you know, highlight spots of where there's been gaps in the existing security program. And so, that's a natural thing that will come when you start to detect things like these metrics in more detail. But I think the way to do the messaging that I've done with folks in the past is really around, hey, we're going to do a campaign together to tackle this big security vulnerability. This is not pointing fingers at anyone who didn't know this before. We haven't talked, maybe we haven't talked about this before, maybe we didn't make this as big of a priority, but we are now, and here's materials and guidance for how to remediate these kinds of vulnerabilities going forward. And hey, if it's still an issue at that point, you know, if you're still committing secrets directly into the codebase 6 months after we rolled out our program, you know, that's, you know, something we want to address. But I think coming at it from a very collaborative perspective of we're trying to solve this problem together, makes it a lot clearer that we're not trying to point blame at anyone. Security problems are hard, are really, really hard to, to nail down. And so, it's going to take everybody to really work together to make it happen.

19:09Chris RomeoYeah. Yeah, I like that as a, as a general philosophy. Like, I've never been a proponent of security being kind of the, like, things like metrics being used to, in a negative, what I would think of as a negative way to, uh, and, and I was in the training space for a long time and people always clamored for some type of a developer score that they could use.

19:41Milan WilliamsOh.

19:41Chris RomeoAnd I didn't, I just didn't like it because it's like, I'm, I'm a teacher at heart. Like, my goal is to get people to learn about these things so that they'll change their approaches, and they'll build better software because developers are engineers, and engineers in general want to build better things. Like, I've never met a developer/engineer who was like, I want to make something that's worse. Yeah, absolutely not. Like, they want to create better things. They don't want to create buggy, insecure code. But, but it's, but you can also create a culture where there is an advantage to not having security things be found. And like, we don't ever want to have that as a culture. Like, we want to be, we got to know, if we don't know about it, we can't fix it. So, we have to have an environment where people are like, hey, this is, this is friendly. I can talk to security.

20:38Milan WilliamsYes.

20:38Chris RomeoAnd they're not going to, they're not going to you know, have me written up by HR or, you know, negatively impact my job, my review, my yearly review, because I tried to do the right thing, but it just didn't work out.

20:52Milan WilliamsAbsolutely. And I think there's a lot of opportunities, honestly, I've seen with metrics where you're able to highlight wins, like teams that do really well. That can be a huge motivator for engineering teams, for development teams, to be able to say, hey, We've burned down our backlog by this much this month. Uh, there's a great opportunity to highlight folks as well, um, when you're able to track things more closely. So, but definitely all about a collaborative approach and complete blameless culture. We're just trying to make the software better together.

21:22Robert HurlbutYeah.

21:22Chris RomeoYeah. And competition's good. Now we can use metrics to drive competition up. In my world, competition's healthy.

21:30Milan WilliamsYes.

21:31Chris RomeoBecause I don't do it from, there's not a negative, but like, I remember there was one day at a previous technology company I worked at called Cisco, where we shared some metrics. We had collected a number of different programmatic-style product/application security-style metrics, and we created dashboards, and then we shared them with the senior leadership. And senior leaders are competitive people. That's how they got to where they are.

22:01Milan WilliamsYes.

22:02Chris RomeoAnd there were a number of operations directors, who got phone calls after that meeting and were just told, I don't care how you do it, fix it so that I'm not on the bottom of the list. And so, in that case, that was a, that was a healthy competition.

22:16Milan WilliamsAbsolutely.

22:18Chris RomeoBut, yeah, we want to make sure comp— we want to make sure metrics are a positive thing, because it's all about culture. It's a culture. This whole thing about security is a culture game.

22:26Milan WilliamsAbsolutely. Absolutely. I'm huge on— one of the other things, going back to how to make metrics not boring, is gamification, trying to Make it really easy to understand. That's why I think simple metrics are good, because people can understand what they are and understand what they need to do to drive them up or drive them down. And yeah, I think that's a great example, you know, categorized by different teams, try to put different teams against each other and see what happens. You know, those are really great ways to just drive adoption.

22:55Chris RomeoYeah. So, I took a turn there in the middle. We were in the middle of walking through the metrics framework. So, We talked about kind of vulnerability metrics. We talked about path metrics. I talked about developer engagement. I like what you were saying about the kind of, I guess, almost pipeline metrics to make sure that things aren't sneaking through and whatnot. So, what else do we have then in our metrics framework? What other components?

23:26Milan WilliamsYeah. The last one that I always think is important, but is part of just making sure that folks track, is just your fix rate. It's, again, pretty simple. Again, it's— but I think it can be a really powerful one. I know we focus a lot on all these alerts, all of these different notifications that come in throughout all of our different tooling that we have, but really focusing on what has actually got fixed this month, how did we actually move security forward? I think that's a very, very powerful metric to be able to show progress, again, celebrate wins, and it can be another very powerful one to justify increase in the team or budget, whatever you're looking for. The last one that I include as part of the framework is around compliance. While it's kind of a necessary part of a lot of security programs, is really making sure you're tracking your SLAs. So if you have, Typical teams that I interact with either have 30, 60, or 90-day SLAs for different vulnerability classes, depending on the severity of the vulnerability. And so, making sure that you stay within your compliance requirements, if you do have them, or anything that you've promised your customers, it's a good baseline one to have. The last thing you want is for a customer to come to you and say, hey, why hasn't this been fixed? You're out of your SLA compliance. That's a whole a whole problem. So, just making sure those are tracked early and often is a good one.

24:54Chris RomeoWell, yeah, yeah. This is a solid collection of different pieces that I could see how these can very easily be implemented. And I love the fact that you hit on simplicity because that's how I've survived my entire career. 'Cause if you make it too complicated, I can't figure it out. I don't understand what— it's all about that. Let's keep it simple. We'll go with the silly version. Keep it silly or keep it simple silly.

25:27Robert HurlbutYeah.

25:28Chris RomeoWe're kid for, you know, in case those people have kids listening in the car, you know, we want them to be future application security professionals. But yeah, simplicity is key because if you make it, it's so easy to say, when something's challen— or something's complicated, it's so easy to say, eh, that's too complicated.

25:49Robert HurlbutRight.

25:50Chris RomeoFor everybody in the equation, including me, I can say that's just too complicated.

25:53Milan WilliamsYeah.

25:54Chris RomeoBut if it's simple and it's easily, easy to calculate, whichever thing you described there, I don't have to be a PhD in mathematics to figure out how to, how to put this into a system. Yes. And read it out. So, yeah. So, I mean, that's, so that's by design though, right? You—

26:10Milan WilliamsYes.

26:10Chris RomeoSimplicity was one of your core principles, I'm guessing, in this process.

26:14Milan WilliamsYes. It's all about— and I think the reason why that matters so much is because in talking to a lot of security folks, it's one of the— maybe surprisingly, or known to folks in the industry, is it's an organization that interfaces with almost every other aspect of the business. You talk with legal, you talk with product, you talk with engineering, you talk with executives. And so, having metrics that can speak to all those different languages is really important. So, that's why I think simplicity is so important. Making sure, one, it's easy to track, easy to implement, but also just easy to understand. Again, if someone isn't in security on a day-to-day basis, they can still get what you've been doing and what you've been working on and can support your team.

26:55Chris RomeoAll right, so here's the million-dollar question, Milan.

26:59Milan WilliamsYes.

27:00Chris RomeoHow are we going to make these things actionable?

27:03Milan WilliamsYeah.

27:04Chris RomeoBecause that's the one that it's easy to create metrics, but you've already shared a couple of different examples of, You know, having that positive program, let's get a program together this month where we're focusing on squashing SQL injection. What else can we do to make these actionable so that they don't just become the numbers that get calculated about what we're doing?

27:27Milan WilliamsYes. Yeah, I think this is where I think some, again, very simple fundamental aspects, but one of the things that has been incredibly powerful for my customers that I've worked with is really just having a centralized place to view all of them. You can have a list of 5, 6 metrics, keep it again really simple. I think that makes sure that everybody is aligned on what the metrics are, what our goals are we're trying to move towards. I think it just keeps it very, very easy to understand what we're trying to do. If you have a clear baseline and a clear goal, the team can work their way to figure out how to make that goal happen. The other thing that comes along with metrics that I mentioned at the beginning, but it's just around storytelling. It's maybe an underrated skill that I think is actually really, really crucial in security, but really being able to paint the picture of like why it matters, why security matters in your organization. That story's gonna be different for everyone, but— Yeah. For example, one common one that I worked on with someone, a customer of ours in fintech, was working around how can we talk about why this matters to our customer, framing how a vulnerability can impact their day, putting things in terms of not only what this vulnerability is, but says, what would happen to their financial information? How does this impact their future financial prospects? how do these vulnerabilities impact their day-to-day life? I think it puts it in a really human level that people can understand. And again, that's going to be different for every organization. But if you think about the user of like, why does this metric matter? Why do we want to protect them from whatever it is? That can be a really powerful kind of human motivator for people to understand why these metrics are important.

29:18Chris RomeoYeah, I love the idea of storytelling. Such a powerful thing. And we as human beings are storytellers by nature. Like, that's the things you remember are stories that you can go to a full-day training and the one thing you'll remember is the story the instructor said about this or that. You won't remember like, well, what facts did they share with you?

29:44Milan WilliamsI don't know.

29:45Chris RomeoThere were some facts. I'm sure there were facts presented on the slides, but if there's a story, that's what sticks with you. And that's a great point about using storytelling in metrics to be able to To make, so people can, can understand the context. They can understand what, what actually happened. Like, we should, you should be a lot more excited. Let me explain to you why you should be a lot more excited about this, because this is actually a very powerful movement we've had. And it's, it's about layering on the context and, and telling the story.

30:14Milan WilliamsYes, absolutely. There's, it's just a great way to get a lot of people, people involved. Anybody can listen to a story and can understand. start to get bought in and get excited and want to help you, honestly. I know for me personally, that's always a huge motivator.

30:28Chris RomeoYeah.

30:30Robert HurlbutAll right.

30:31Chris RomeoWell, we've reached the lightning round where Robert shines all the time, but this is where he especially shines is during the lightning round.

30:40Robert HurlbutAll right, Lon, we have 3 questions to ask about in our lightning round. First is, what's your most controversial opinion on application security, and why do you hold this view?

30:52Milan WilliamsYeah, my most controversial opinion, I think, about AppSec is really about, I'm a huge believer in prevention. I think a lot of application security, we can talk a lot about what is in production now, what is burning today, which is important, but I think we undervalue the importance of setting ourselves up for long-term success. We do a lot of firefighting. a lot of just trying to address the here and now. And I think we could save ourselves and save our engineering team a lot of time by setting up systems to be proactive instead of a lot more reactive. Examples of this look like investing in training, investing in really clear areas to make it clear for how to fix these kinds of vulnerabilities, or tooling to make sure you can fix things before they hit production. I think, Yeah, I think in general, that's my like most controversial opinion in AppSec, just really an overinvestment in the proactive side of things to decrease the load later down the line.

31:55Chris RomeoOkay, thank you.

31:58Robert HurlbutSo our second one is, what would it say if you could display a single message on a billboard at the RSA or Black Hat conference?

32:06Milan WilliamsYeah, we've been talking about metrics, so I think it would have to say something around metrics. Maybe there's some kind of slogan around like track your metrics, something like that. Make sure you're tracking those. I think that would be my slogan.

32:22Robert HurlbutVery cool. And the final question is around book recommendations. What's your top book recommendation? But also, it doesn't have to be technical, but we want to know why do you find it valuable?

32:36Milan WilliamsYeah. I recently, semi-recently read this book called Quiet. I believe it's like an introvert's guide to being in the workplace, something like that. And it's really good. I think it's by Susan Cain. And essentially what she does is she talks through how to take advantage of introverts that work on your teams or that you work with. I'm an introvert by nature, and And so, uh, I found it a really powerful book, um, especially I work with a lot of engineering teams and we're all kind of introverted by nature. And so, I just found it a really reflective book to think about, like, what are my strengths? What are the unique ways that I can contribute to a team that can be really extroverted? Um, so yeah, that would be my, my book recommendation of descent.

33:22Chris RomeoThat's interesting. I'm gonna add that to my holiday reading list.

33:26Milan WilliamsYeah.

33:26Chris Romeo'Cause that, that sounds something like something that'd be very practical.

33:30Robert HurlbutYeah.

33:30Chris RomeoTo, because I, I don't know, I kind of switch between being an extrovert and an introvert depending on the time of day and other things. But yeah, it's, if you could just understand how to, I like that, that idea of how do you unlock some of the capabilities that exist in your team that if you're not reaching out to introverts, they can They can have the best ideas, but you're not getting them because they're not gonna be like, let me talk, let me talk. Like some of us are.

34:02Milan WilliamsYeah, it's a great book. I think she just gave a lot of good tactics and also for introverts to figure out like how you can better, you know, tap into the parts of yourself that you may not usually do, especially in a business context, things that you wanna work on or be better at. So I thought it was a really great read.

34:18Chris RomeoVery cool. So, Milan, what do you want to leave our audience with, from as far as a key takeaway, perhaps a call to action? What would you, how would you land the plane here?

34:30Milan WilliamsYes, I think my number one key takeaway is if you don't have a metrics program today for your application security program, today's the best day to get started with one. Keep it simple, don't make things super complicated. Use stories as a way to get buy-in and get people excited about the metrics that you do present. And then celebrate the wins. I'm sure there are things, once you start to get these spun up, you'll start to see a big change in your security program, and make sure to be loud about it. Use them as a way to advocate for yourself and for your team to make sure you expand your team or expand your budget, whatever you need to achieve your goals as a security professional.

35:09Chris RomeoVery, very cool. Well, Milan, thank you for sharing your wisdom and insight here from this whole metrics project. And I think it was a great, collection of metrics that people can use. And definitely looking forward to having you back on the show again as a guest, because I want to, I want to dive into product management.

35:31Milan WilliamsYeah, I'm happy to.

35:32Chris RomeoWhich is not an area that we spend a lot of time as security professionals understanding and kind of using as a resource. And so, I'd love to have another conversation in the future just to, just to go down that whole realm of, because we've, we've, we've, we did an interview with Tony Quadros about the AppSec salesperson.

35:57Milan WilliamsYeah.

35:57Chris RomeoWhich was good because it was, it was, it was, and I, I learned a lot from, you know, because how salespeople in the industry think and, and whatnot, how they interact with security. So we'll do that again in the future. We'll have you come on and just talk. We'll talk product management.

36:12Milan WilliamsHappy to. Happy to. Thank you for having me.

36:14Chris RomeoYeah. Thank you.