Aram Hovsepyan -- Your Security Dashboard is Lying to You: The Science of Metrics

0:00Chris RomeoLet's cut through the noise of flashy charts and bogus numbers. Today's episode, Your Security Dashboard Is Lying to You. We're exposing the myths behind common security metrics, why total vulnerability counts and risk scores can mislead, how bad math warps decisions are, and what you should measure instead. With the Goal-Question-Metric framework as our guide, we'll explore how to build dashboards that reflect risk and readiness. Let's dig into the truth behind the metrics.

0:27Aram HovsepyanThe Application Security Podcast is brought to you by Security Journey. Security Journey is an enterprise-class solution with lessons that are built on learning science principles to deliver long-term, measurable results. Learn more at securityjourney.com.

0:40Chris RomeoHey folks, welcome to another episode of the Application Security Podcast. My name is Chris Romeo. I am a VP of DaVinci at Security Compass. And as always, joined by my co-host slash co-conspirator, perhaps. I don't know. Robert Hurlbut.

1:09Robert HurlbutHey, Chris. Yeah, it's Robert Hurlbut. And I'm a principal product security architect and threat modeling trainer at Torion. And glad to be here as well again to, to chat and see what's going on in application security.

1:25Chris RomeoYeah, in the world of security dashboards and metrics. Well, let's jump right in. So, Aram, we're excited to have you here and want to start by hearing your security origin story. If there was a comic book for your career, what would I find if I read episode— or episode, it's not an episode, it's not a TV show, it's a comic book. If I read the first release of this comic book?

1:52Aram HovsepyanI'll go. Yeah. Well, I would say you would first find me on the COVID of the comic book wearing 3 hats because that's sort of how I go. And obviously I wear them figuratively, but hey, it's a comic book, so why not put me look a bit comic-ish? I started my career as a researcher at the University of Leuven in Belgium. Belgium, and after finishing my regular studies, I was like, hey, being student is fun, so why stop? So I I jumped into a PhD track, and I started working at a research lab that had a lot of apps going on. It's the Distrinet Lab. I'm not sure if you are familiar, but oh yeah, many people in the OWASP community also coming from the Distrinet Lab.

2:43Chris RomeoKim Butz is a is a good friend of ours in the threat modeling community, so I think that's where she yeah. So, you kind of grew up in, uh, in privacy, right? In, uh, at the lab there.

2:54Aram HovsepyanSo, actually, and you're bringing up Kim, one of the research topics I was working on was actually the framework called Linden and privacy threat modeling, which was back in the days, it was a very obscure thing. And security was already somewhat hard to explain to normal people, but I guess The hacking things helped a little bit, but then privacy was off the charts. But then luckily GDPR came and then privacy went from niche academic thing to a corporate panic button. And Linden got into the spotlight and the methodology is brilliant, but I think it's still ahead of its time because it was designed for companies to build privacy in their products and Honestly, I don't know many companies who really, really want to do that because they are sort of into selling your data business rather than making things privacy-friendly or privacy, having the privacy property by design.

3:57Chris RomeoYeah.

3:58Aram HovsepyanI've done other research, but I'm not going to go into that because Lindon is the most famous one from the works I've done. The rest is really, really obscure. And when we come back to one of the research I did, because it's related to metrics. And then my next hat would be the entrepreneurial hat, because around the same time I was doing research, I started doing hobby projects as a consultant, which was, by the way, very much discouraged by the universities. And thankfully though, my PhD advisor, Wouter, was an entrepreneur. So he helped me navigate around the dinosaurs. And what started as a side hustle became sort of a boutique consultancy, and now it's a product company called Codific. And we are the makers, amongst others, of the SAMI tool. And then finally, my last hat, which is the best hat I would say, is the OWASP hat. And I like to think of it as the vigilante hat where I help the world. fight cybercrime and make applications more secure by being an OWASP SAM core team member. And the story there is, while at the company, and in general, I often get the question, like, is your product secure? Which is honestly, like, it's the tech world version of, what are your weaknesses, on a typical job interview. It's a good question, but then it's sort of a trap at the same time. And then one day, I stumbled upon OWASP SAM, which sort of provides an excellent way to framework to answer that question in a systematic way, rather than just saying, trust me, bro, we've pen tested it, it's secure. And funnily enough, the project lead, one of the project leads was Bart, who was also from this unit. So I knew Bart, he was an ex-colleague of mine, and Seba, who is part of OWASP, built-in OWASP chapter, whom I also knew. So since, ever since I've joined the project, I'm living the dream.

5:59Robert HurlbutYeah.

6:01Chris RomeoCool.

6:01Aram HovsepyanAnd that is more or less my origin in the world of security.

6:05Chris RomeoI always love that question. What are your weaknesses? And I think the best answer that I ever heard, I think, was, I think it was Michael Scott on The Office that, you know, I, what did he say? I volunteer too much. I sing in the shower. And sometimes, sometimes I hit an employee with my car, but That was a different segment. So, Robert, why don't you kick us off? Let's get into this metrics topic.

6:30Robert HurlbutAbsolutely. Hey, Oren. Recently, you spoke at the OWASP AppSec EU in Barcelona, and your topic was, Your Security Dashboard Is Lying to You. I'm curious, what inspired that title, and what lies are most harmful?

6:46Aram HovsepyanYes, that's a great question. And I'm gonna start from where I came to that topic at all, which was, like, a total surprise. Actually, one day, Seba, who was the chapter leader, sorry, he's involved in Belgian OWASP, and he's the project lead, co-lead of OWASP SAM. He asked me, hey, Aron, why don't you do a training on security metrics? And I was like, but I don't know. Well, I don't know much about security metrics. And then I was like, isn't that like something easy? You just pull the metrics from tools and you put it in dashboards. And he was like, yeah, yeah, just give a talk on that. And then I went into a mode of like, hmm, this is not the right, something is wrong here. So I went into the research, complete researcher mode, and I started reading about metrics in general. And then I figured out that, hey, I've actually worked on this while I was a researcher, because I was looking at into defect metrics. which is a huge topic. And security metrics are sort of— so, security defects are a type of defects, not the same thing, I know, but they're close. And there is a ton of research on that. And at that moment, I was like, okay, there's a lot of stuff here. And I was like, hey, what we are doing with security metrics doesn't really, really make sense because there are a ton of issues there. And typically, when you think about dashboards, they are filled with that things are low, medium, high vulnerability counts, their CVSS scores, and maybe some additional heuristic, which combines all of them into a nice single risk score. Unfortunately, if you unpack all of that, you end up with basic key qualities of what is a good metric as one of the problems, and then these qualities don't really match the reality that you see in a dashboard. So, it is likely that the dashboards are representing information which is not reflecting the reality of the security posture that is going behind that certain application.

8:57Robert HurlbutYes.

9:00Aram HovsepyanAnd what was the second question? There were, I think there were 2.

9:05Chris RomeoYeah, I think you kind of covered some of it, what lies are most harmful, but I think you, unless you want to address something specific to that topic.

9:12Aram HovsepyanYeah, I would say that eventually then you start optimizing your organizational AppSec program around making sure that these metrics are green. Let's call them green. I don't know what the numbers are, that we don't have any high criticals. The mediums are also solved somewhere there, and then we pull out all of that to green. But then that green doesn't really correlate in scientific terms with actual security, and it doesn't really correlate with breaches, potential breaches, or risk. Eventually, you're— we are interested in risk.

9:44Chris RomeoSo, if you're— if, if we're talking about metrics in general and for, for application security and how they're used in most organizations, are you saying that they're flawed in how people are doing this today? Like, would you go as far as to say that? I will, but, you know, would you agree with that based on the research that you've done and what you've seen?

10:08Aram HovsepyanI'm gonna say it depends, 'cause that's always a good answer to give, but then I'm gonna say, at least based on my experience, what I've seen, and actually, I have been involved as a consultant as well, AppSec consultant in various organizations, smaller and large, including Fortune 500 companies, I would say that these metrics are not necessarily lying, but then the question is, if we revert back a couple of steps, would the question, an excellent question would be, what are we trying to achieve? What are our goals? And then if we try to go, if we try to look at this issue from a top-down fashion, if we try to look at it from the goal, which would be for security, it's likely to be, We want to reduce our risk, our risk of getting breached. And if we start going from there to the metrics, we're gonna end up with a bunch of problems. It is, and these problems are potential issues. I believe they're real issues, but it could be that in a specific scenario, they are still good approximations, all these metrics.

11:16Chris RomeoIt is.

11:16Aram HovsepyanHowever, there are all these problems, there are these conclusions that you're reaching, hey, we are green, we are good. So false sense of security based on metrics, which have a lot of validity issues when we have to support these decision-making process.

11:35Chris RomeoSo I guess the million-dollar, million-euro question here is, if all of my metrics are green across my dashboard, am I secure?

11:52Aram HovsepyanI'm gonna again pick the safer answer, which would be it depends.

11:57Chris RomeoDepends on the quality of the metrics that I put in.

12:02Aram HovsepyanYeah, and it depends what is your risk appetite. It depends what is your application doing. Are you building a nuclear power plant with access to— I would not imagine what— I have no idea why somebody would do that, that externally facing, internet-facing nuclear power plant. Or a banking app, let's call it banking, that's more realistic, a banking application, which is externally facing, perhaps you wanna be paranoid there, but having all metrics green, sorry, all vulnerability counts, 'cause that's not all the metrics that you could have, green, that is, I would not say that this application is secure or as secure as it can be.

12:42Chris RomeoYeah.

12:46Aram HovsepyanI would say that having red dashboards would indicate a problem. So, if you have a ton of high and criticals, I would not be comfortable living in that situation. It, in my opinion, it actually would approximate technical debt rather than security risk in the first place.

13:04Robert HurlbutOkay.

13:04Aram HovsepyanSo, why are you having all these vulnerable libraries and findings from tools, and you're not fixing them or not triaging them, at least there is something wrong there. But on the other hand, if they're all green, I would not necessarily jump to the conclusion saying, hey, this is great. We're happy here.

13:23Robert HurlbutYeah.

13:23Chris RomeoSo metrics drive human behavior in an organizational context. So, right? Isn't that the— that's kind of like, why do we have metrics? We have metrics to report for executives so that they can cheer and say that they're the best out of 10 different departments that they're the best. That's one reason. The other thing is to drive some type of action by the human beings who are building something here. And so I'm kind of, as I'm listening to you describe kind of this high level, kind of starting to put this together for us, it seems to me that it's the human behavior, like good metrics, drive human behavior towards making things more secure and, and considering privacy? Is that the ultimate goal of what metrics are for, or is there something else?

14:18Aram HovsepyanI, I would rather reverse that and start with a, with a goal, which again comes from the top down. So an organization needs to figure out, hey, what do we want to achieve here? What are our goals? And then once we know what our goals are, we could then start selecting. I'm now jumping a step over because there is a great framework called Question Metric Framework. But let's say you have your goals and then we jump to metrics. Then we could go and select the metrics that are suitable in order to help us answer the questions behind the goals. And then I'm like, there is a level missing there, obviously.

15:03Robert HurlbutYeah.

15:03Aram HovsepyanAnd that's the goal-question-metric framework, which I would like to bring up, which is the systematic approach to metrics. So instead of, and this is what we are doing in the industry right now, I have yet to come across an organization that's not doing that. Instead of picking metrics that are available in the tools, which I honestly a year ago, I would also do that. I was like, yeah, metrics, we just pick some metrics that look good and we just put them in nice dashboards. Isn't that metrics? Well, actually not. You should start with your goals and we start with goals because Every organization is unique. Your risk appetite is unique, and there is no way that you can go bottom up because the number of metrics you could potentially select are are pretty large. It's not just the vulnerability tools or ASPM tools that bring you these metrics. There are a ton of tools that can give you a lot of metrics. You need to start with goals, and a goal of one organization and another organization are. likely to be different, although they're both probably focused on risk. And then once we have these goals, the next level is actually formulating questions to measure progress towards that goal, to measure, to see how well we are doing in terms of a specific goal and to measure the progress towards that goal. And only then we pick metrics to answer these questions in a quantitative way. And this is the surprising part, they can also be qualitative metrics. So they don't really need to be always numbers. They could be something like a SAM assessment. You bring in an external consultant, he dives deep into your practices and processes and says, here is a list of things that I've observed that are not great. I would still consider that a measurement and hence a metric.

16:48Chris RomeoNow, this goal question metric, is this something that you created or is this kind of a, from your research background?

16:58Aram HovsepyanOh, no, totally not. So this has been there for 30 years. This is a framework from 30 years ago, which was focused on defect management, which is, in my opinion, security defects are a subset of defects. And there are also alternatives to goal-question-metric framework, but typically all alternatives end up with a triad. So you have like 3 layers where eventually the metrics are your lowest layer where you pick. To go up in the abstraction level where you end up with goals. So it's the work of a researcher called Vasily, Viktor Vasily, and some other authors. And it was designed and it was, they are actually sort of proving that if you have to use metrics, you have to go in a top-down fashion. Otherwise you are going to end up, otherwise it's just not going to work, the bottom-up approach. And a nice analogy would be if you would try to optimize certain blood markers like your sugar level or your cholesterol level without having any goal, and maybe you don't have a problem at all, and there is absolutely no need to optimize this in any direction. So you always need to start from a specific objective that your organization is setting. and then work towards figuring out how do I measure that in order to improve that or to reduce that, whatever qualitative objective you have. And that's the way it goes. And then you go back to answering the questions that you formulated for the goals and then looking at the progress of the goals.

18:38Robert HurlbutLet's dive into talking about some security metric. I think I think you have 3 different aspects. So there's precision, reliability, and accuracy. What's the difference between those and why would it matter or why does it matter?

18:55Aram HovsepyanYeah, and that's a nice next segue to the next topic because it's, so the goal-question-metric, we have our goals, we have our questions and the metrics. And then the question is, what is a good metric? And then a good metric has 3 qualities. qualities. The first one is reliability, which is about if we calculate that measure, if we do that measurement several times or by several people, that we need to end up with the same value. So, if you step on a scale, your weight, you step 3 times on your scales, your weight should be exactly the same, or let's say with a margin of error 1%, 2%, probably that would be acceptable. And on the other hand, if we think about, for instance, CVSS scores, I'm not sure if many people know this, but there is about 40% chance that we come up with completely different CVSS scores. So if 2 people do that, although it's like you need to pick, like, for people who are not familiar with CVSS scores, eventually it boils down to many qualitative, many labels. Well, they are qualitative labels, like how complex is the attack, and it's low, medium, high. What is the impact to confidentiality, low, medium, high, and then you have a bunch of similar qualitative labels, which form a vector, and the CVSS score ends up transforming that into a numeric score for easier representation. But then if 2 different people do this labeling of the same vulnerability, there is a 40% chance they're going to end up with a different score. So, the reliability there is maybe not terrible, but it's also not 100%. We're not going to end up with the exact same score. And that is reliability. Precision is about the smallest unit of measurement. So, and in the context of CVSS, it's probably less relevant. But if we look at security awareness tests, for instance, a test that ends up with a score of this person is, yes, he passes the test, or no, he doesn't pass the test, versus another test that says this person has a score between 0 and 100, the second test would have more precision. It's a number between 0 and 100. You have like basically more buckets. So you end up with more precision. And in that case specifically, it brings you more information. So your metric is going to be actionable. And that's a very important quality for a metric. It needs to be actionable. If you have a metric that says, We were 40 and tomorrow we are 30 and the day after we are 50. I'm now thinking about the SPM risk scores. I'm not sure how actionable that is. And then the third quality of a good metric is actually not really accuracy, although I know in my presentations I refer to it as accuracy. It's validity of a metric. And then we have 3 types of validity and accuracy is one of them. It's actually the correlation with the outcome. How well does my metric measure the reality? You could also call it accuracy, but that is what we call criterion validity. And then we have 2 more validities. So we have content validity, which measures how well does the metric cover the problem, the outcome domain. And to give an example, if we have, if we look at, let's say, number of hours of training, that has less content validity versus another metric, which would be percentage of top 10 risks covered throughout the training. So number of hours of training, it doesn't really cover all the risks, let's say. While if you know that this other training, this other metric covers the top 10% of all risks, it has a much better content validity. And then finally, the construct validity, which is about how well does my metric reflect in the real world. And this one is very, very hard to explain typically. This is why I would resort to the IQ tests are notorious for this. So, although you might pass the test with high marks, the question is, how well does an IQ test reflect on the intellect of this person, the real intellect? Or in security, a great example is if you pass a security awareness test or an attack simulation test with flying marks, the question is, How well would you perform in real-life situation with an attack that you've never seen before? So a simulation attack is probably something you've seen in training. A construct validity is about how well does that score reflect a real attack. And when you pick your metrics, you will eventually be facing the questions of how reliable is this metric, how precise is this metric, and what is the validity of this metric when we want to go back and answer the question that to support this is a certain decision-making, which comes from the goal. Now, normally I do these explanations with slides. I'm not sure how, how well I did so far.

24:26Chris RomeoNo, no, it makes, it makes sense. I'm, I'm just, I'm kind of processing it as well, 'cause there is, what I'm realizing is a good metric does have some complexity. So it's, it's not something that we would say is simple. To reflect, but the categories definitely make sense about, you know, how reliable the metric is. So that, I mean, I didn't realize that about CBSS. I guess now that I think about it, it makes sense because there are a lot of subjective decisions that you can make along the way that will be influenced by the experience of the person who's making the decision about that.

25:05Aram HovsepyanYep, exactly.

25:06Chris RomeoI see how that reliability could be, could be, we could come up with different values just because I've worked a bunch of incidents and I see something and I'm like, ooh, this is like what I've seen before in the past. That's critical. Somebody else says, ah, that seems like it's kind of a medium level. I don't know that I've ever seen that happen. So our experiences reflect through the reliability, you know, so an unreliable metric is something that uses our experiences. versus data or science to get a 0 or 1 answer.

25:36Aram HovsepyanYeah, exactly. And if we think about the CVSS again, if we look at the vulnerability count, it's not only the CVSS, which is a reliability, which has some reliability issues. It's not terrible though. I would say that the reliability of CVSS is pretty good. But then we have a problem, a lot of issues with validity because Many of these vulnerabilities, CVSS scores, are not really exploitable. Some of them might not lead to a breach, although they are exploitable. And the number of these issues, the question is, they don't really tackle the construct validity. They cannot really answer how secure a certain application is, in my opinion. So, in terms of validity, using the number of vulnerabilities would not be ideal. I'm not saying, by the way, that we should not use this, obviously, but then the point here is to know these qualities, that there are some issues to the metric, and then perhaps look into other categories of metrics that could complement it.

26:43Chris RomeoYeah, I've always, I find myself historically arguing against total vulnerabilities, as a quality, as a good metric because it can be kind of distorted. And so the example I love to share, it goes back in way back into the, to long ago in history, is I spent some time working at Cisco and there was a researcher named Felix Lindner, Lindner, Lindner, who went by the handle FX. And one year, FX decided that he was going to research Cisco products. And so guess what? At the end of the year, we had a pile of vulnerabilities that he had found in the product. And now that the times have changed some, I get that. But if we would have just used that year's metrics against the previous year, we would have, we would have said, This is terrible. Like things are falling apart, but they weren't falling apart. We had a world-class researcher who was focused on Cisco products for a time period who found a lot of things that needed to be fixed and were real problems and real vulnerabilities. But if we would have used that metric and said, we're only using the metric, we would have said this year was a complete disaster compared to last year. When in fact, it really wasn't. It was, it was a partnership. that was happening as well. So, so yeah, so that's, that just, that was just, that just made me think about kind of how metrics can be, can be kind of misconstrued, I guess, depending on some of the circumstances.

28:28Aram HovsepyanThat makes sense.

28:30Chris RomeoLet me ask this one, Robert, and then you can ask the final one because I'm curious about the overlooked metrics. So what are, what are people, what are over— so Overlooked metrics to me sound like something that people don't use, but they should because it would get a lot of value for them. Like, what are some examples of that? You know, I know you mentioned the security awareness one that was kind of a mixed bag. I guess it could have been a good or bad. But what are some things that people just aren't aware of that they need to be adding to their metrics stack?

28:59Aram HovsepyanYeah. I'm gonna give you some metrics which I've never seen really being used, but at least, except for our own firm. Here are great metrics. Number of ESBS requirements that you've implemented in each of your applications, let's say relevant, or you can, if that sounds crazy much, there are, I think, the version 5 of OWASP ESBS has 370 plus.

29:30Chris RomeoWow.

29:32Aram Hovsepyanrequirements, which is crazy if you ask a team, hey, you need to now implement that because we're tracking that metric, they're going to go crazy. You could focus on level 1, which are relevant, and you might end up with 20, 30. And then you could see if you could even ask your teams, hey, who can implement more of these? And maybe they already have some of them implemented, so they don't really have to implement them because that is the metric. Related to that, Aside from simply implementing the requirements, making sure that the security requirements are tackled, you could go a step further and ask teams to write unit and integration tests for those requirements, which are then automated in your CI/CD pipelines. Excellent metric, because now you have like how many security controls requirements are in our system, in our product, and then how many of them are automated. I would say any security unit tests or integration tests that you have in the application, you could measure the amount of them because how I strongly— why am I saying this? I strongly believe that these reflect on real risk and risk mitigating measures because then you're baking controls in your software systems. So, you could claim that after introducing these controls, which are by definition risk reducing, risk mitigating measures, we claim that our risk went down. And the more of these controls you have, the lower your risk. I'm simplifying things. And many of the things that, by the way, I've mentioned today, I have butchered a couple of things here and there. Otherwise, it would've made the explanation way more complex. An interesting one would be, and I know a lot of threat modeling, you have a lot of threat modeling related episodes, and both of you guys are are part of the community, number of successful threat modeling sessions per year that you do. I'm just throwing it here. I haven't, I thought about it a couple of days ago, but I would throw in that as a good metric, because, but then the success, the definition of a successful could be tricky. What is a successful? I would say that if your threat modeling session should come up at least with one threat, Otherwise, it's questionable how successful it is unless you've been doing it for so many times that you cannot come up with any new threats. So number of threats is new threats.

31:52Chris RomeoYeah, but it's also a cultural move. I think of those, and I love that metric, and I've used, I've applied that concept to other scenarios as well because it's a movement metric. So I don't think of it as a quality. Like, I'm never gonna try to measure, especially in the early days of rolling out threat modeling in an organization, I'm never gonna measure the quality because I'm gonna come up with You know, bad grades for everybody, right? Because we know that the more threat modeling, you have to do threat modeling to get better at threat modeling. The more you do, the better you get. So I love that. It's just a quantitative measurement that says, hey, we did this many sessions this quarter. And then our, our goal becomes, maybe our goal in that goal-question-metric framework, our part of our, one of our goals for the future is we want to double the number of threat modeling sessions we have because we see the value of it.

32:43Aram HovsepyanYes.

32:43Chris RomeoYes. And then we can measure. So, yeah, so I'm in line with that. I love that idea because it's not qualitative. It's about how do we just, how do we get people moving in the direction that we want to, which is really the goal of our AppSec programs. We want developers to do security activities, and we want to ingrain that in their process and the way they think. For me, that's the true success metric of a threat modeling program is, do developers just do it as part of the standard process? If that happens, we've won, and we're done. can move on and go somewhere else.

33:11Aram HovsepyanExactly. Completely agree.

33:13Robert HurlbutYeah. So sort of coming into the end here, one question we've been wondering about is if you— we've seen a lot of security dashboards, but if you were redesigning a dashboard from scratch, what would you prioritize showing for someone who's looking at one?

33:38Aram HovsepyanThat's a great question, and it's a difficult question, I gotta say. But I would say that your dashboard has to have metrics that answer meaningful questions and not convenient ones that you picked from the tooling, so that you clearly have an idea what you're trying to improve. And for many, it will be the idea of reducing risk. Now, I'm not saying that ASPM tools with their risk score are a bad thing, but then that shouldn't be the only thing on your dashboard, perhaps, if you like that, if that's what you are preferring. But that could, if that's the only thing that you have, that could create an illusion of a secure AppSec program. Well, it represents one specific type of metric amongst probably hundreds of potential metrics that you could pick from. And your dashboard should reveal the truth of what's going on in terms, also in terms of your process perhaps, rather than just your output and these scanners that look into your code, because that's just one side of the story. And a typical AppSec program is about people, processes, and tools. And for me, many of the current dashboards are heavily focused on tools rather than people and processes.

35:01Robert HurlbutAll right.

35:04Chris RomeoWell, I got one more question because I've been kicking this metric around for years and years. And what I'm about to describe to you is something that I think is a good metric, but I want to, I want you to test it against all the research you've done into metrics, because I'm afraid maybe I've been leading people astray for all these years. I like the metric of mean time to resolution for a defect. And specifically in the world that I've lived in, it's been more from a security defect perspective. But I like the idea of mean time to resolution because for me, it takes out some of the noise in measuring, because I'm always trying to ask the question, are— well, look, I'm reading right into the framework here. But the question I'm always asking is, are we getting better? Are we better this quarter at fixing the security-identified defects, or defects that are, whether it came from a tool or from a researcher or whatever? Are we getting, are we able to fix them faster? Because that's what, that's a metric that I think I can demonstrate. The program is getting better. We're getting more efficient. Our education is paying off, all these other things. So, curious, Aram, to understand like, what is your perspective on my metric that I've considered to be a North Star?

36:27Aram HovsepyanChris, it's funny. I like that metric and I had, I was gonna actually give it in my examples, but I've sort of forgotten about it. And yep, I definitely, absolutely. You, the only caveat here, caveat, caveat.

36:42Chris RomeoCaveat. Caveat. Yeah.

36:43Aram HovsepyanCaveat. Yeah. Is how do you define, it might be tricky to measure it right, because then what is resolution? Is it I pushed it to fix? Is it I've deployed it? Is it in production? So that needs to be clear definition. So something I also didn't really discuss is your metrics have to be very objectively and easy to measure. So if you change the person who's measuring that same thing, that the mean time to resolution, we should end up with the same number.

37:13Chris RomeoYeah.

37:14Aram HovsepyanSo there should be clear guidelines and documentation how to measure that metric. That's the only thing I would throw in here. But yes, and it is related to the fact if you have urgent issues that are discovered by anything, let's say by pentesters or HackerOne, 'cause that is sort of more reliable than just looking at CVSS scores, you need to fix that as soon as possible and get it into production. And the question is, how long does that process take? And indeed, it's an interesting metric, and it also looks into how fast your whole process is set up to go, to accelerate that fix. But then on the other hand, I would not look into every issue. You might have issues which are low priority, and nobody really wants to fix them. And that could be part of your part that you should set up a clear question in a way and fine-tune the metrics so it measures the issues that you really want to fix fast or within a certain SLA. But yes, either.

38:20Chris RomeoSo not the lows.

38:22Robert HurlbutYeah.

38:22Chris RomeoSo great. I have not been letting— leading people astray for all these years. So it's good confirmation for me. So based on everyone who said I was, I actually wasn't. Nobody was. But what, um, Aron, what do you think of as kind of like a key takeaway? What, what can our audience— what do you want our audience to do as a result of this? Should they just go delete their dashboards and just say we're starting from scratch? Like, what, what do you want them to do as a result of this conversation?

38:48Aram HovsepyanDefinitely not delete them because many people will lose sleep over that. I would—

38:54Chris RomeoOh, they'll lose jobs too. They'll be losing employment.

38:57Aram HovsepyanSo I think that even though many dashboards could be telling some lies, but there will be also certain elements of truth in there. So I would like people to go and critically look into the metrics they're measuring with the scientific lens, with these criteria that I've mentioned, and answer for themselves, hey, what are the potential problems with these metrics? Are these good metrics? And perhaps drop some metrics and definitely look into adding others that would make more sense to answer meaningful questions and clear, clearly defined objectives.

39:34Chris RomeoThis is, this has been excellent. I've learned a lot about metrics and metrics is one of those things where it's easy as a practitioner to kind of brush them off as they're somebody else's problem, like metrics are. And And, but this has been very eye-opening for me. I've never heard of the goal question metric framework before, but I'm going to take that forward and I'm going to apply it to other things that I do in the future. So that's been very beneficial. So, Aram, thank you for sharing this expertise, teaching us and also teaching our listeners.

40:09Robert HurlbutThank you. No problem.

40:09Chris RomeoSo yeah, we really appreciate you sharing this experience with us because you've obviously done a lot of research and I love the fact that you came from a research background. So you're bringing in just a different way of looking at these types of problems than those of us who kind of don't have a research. We haven't ever had to research something scientifically and defend it. So that, to me, that means a lot because you've had to, I know in that world, you don't just say stuff. You don't just research stuff and say it. You research it, you defend, you say it, and then you defend it. And other people come at you and try to say, they try to prove you're wrong. Like, that's part of the fun, right? In that game. So thanks for being a part of the Application Security Podcast.

40:48Aram HovsepyanHappy to do it, Chris and Robert. It was awesome.