Sarah-jane Madden -- Threat Modeling to established teams

0:00Chris RomeoSarah-Jane Madden is Chief Information Security Officer of Sensing Technology Group, part of Fortive. She has over 20 years of software experience from the most formal environments to let's fix it in production type teams. She's been a longtime advocate of deliberate application security as a partnership with product management and believes security does not have to be an overhead. Sarah-Jane joins us to discuss her talk at OWASP Dublin, Far from Greenfields: Introducing Threat Modeling to Established Teams. She shares lessons learned from her 3-year journey and is transparent with the mistakes she made along the way. We hope you enjoy this conversation with Sarah Jane Madden.

0:44The Application Security Podcast is brought to you by Security Journey. Arm your developers with engaging secure coding lessons developed with input from leading application security experts. Security Journey offers responsive developer training plans that integrate with your existing AppSec testing tools to identify and address vulnerabilities in your own code. Visit securityjourney.com to try our training today.

1:10Chris RomeoHey, folks. Welcome to another episode of the Application Security Podcast. This is Chris Romeo. I'm the CEO of Curve Ventures, and I'm flying solo today. Super excited to talk with Sarah-Jane Madden, and the topic is going to be around threat modeling, but I'm going to wait, and we're going to dive into Sarah-Jane's security origin story because I know our guests or our listeners are always sitting on the edge of their seats, hopefully not while driving a car, but they're sitting on the edge of their seats waiting to hear how people got into security. So, Sarah-Jane, just take us into that story, please, and enlighten us.

2:05Yeah, sure, Chris.

2:07Sarah-jane MaddenSo, I sort of was co-opted into security. I started in technology 20-plus years ago. I spent a large part of my career in banking. And as it was at the time, a technologist was just a technologist. You got moved around. Hence, I spent time on the infrastructure side, and I spent time in testing, spent time in software development, which was highly frustrating at the time, but bears fruits at this stage of my career. But I spent 5 or 6 years of that on an infrastructure team. in the treasury function. So, you know, very much high availability, you know, security was important, etc. But I was far and away the most junior member of that small team, and nobody really wanted to do security. So, it got dumped on the junior, and I had some fun experiences there where, you know, the time I took the manual I was given for hardening an HPU Xbox, applied all the controls, And then it wouldn't boot. This very high-cost system, because I had, you know, not questioned what I was doing, applied absolutely everything, and we in fact had to get the manufacturer back in to reset the machine. So lessons learned the hard way. But from there, I kind of went to— I suppose I sort of like this area, and I could see the value to it, and to go way back to when I was born, my dad and a lot of my family were Gardaí here in Ireland, so in the police force. So I only realized then when I started doing security work in the bank that I had kind of a natural inclination towards not being malicious but thinking what the malicious actions could be. And I just— the bank had a model at the time of kind of distributed systems. So again, like Security Champions' idea, So, I spent a long time in that before deciding to sort of formalize my education, go back and do the master's in security and forensics in DCU. And then after I left the bank, I just, everywhere I went, sort of continued to evangelize security. And then when I moved with my husband to the United States, he was seconded and I went, sure, let's go, and working for Accruent.

4:28Chris RomeoOkay.

4:29Sarah-jane MaddenWhich is a software company that was then acquired by Fortive, who I work for now. I got the opportunity to help build out the security program for the first time. And there— now I know that's an opportunity that lots of people who spend time in security go, I'd love to try that, rather than inheriting somebody else's program. I actually got the flip side of it. So I had the formal— everything was very rigid in the bank, but I've kind of very early on got a chance to build out a program from scratch. It's harder than you think. And I sometimes, when people say that thing of, oh, I'd love to build it from scratch, I go, careful what you wish for. And there's a lot involved in rolling out a program from there. And yeah, I've just continued kind of evolving in that role then within Croon, which got acquired by Fortive. And now I'm CISO of the Sensing Technologies Group, which is another subdivision of Fortive. And we continue to sort of mature the program there. But yeah, I kind of came from the ground up as an individual contributor. Very cool.

5:33Chris RomeoYeah, very cool. So, when I think about bank security, everything I've ever heard about banks and security is that the organizations are super risk-averse. It's really difficult to do things. It's much more constrained than doing security like in a tech company. And so, I'm curious, how much of that philosophy came with you to building your own program? Did you build a bank-style program in a tech company or did you, like, how did that come together for you?

6:08Sarah-jane MaddenNo, I would say influenced by it. You know, there's a lot of, this is what good, looks like, but there were more conditional clauses, I suppose, in what I built out in the tech company, because in banking, a lot of it, as you say, is quite rigid. It's, you know, audit is our friend no matter where we go. I think that's a mistake we make. They help us find the gaps. You know, they may be part of your team, or maybe it depends on your structure, but the concept of audit is our friend. Now, in the bank, you're very much aligned to audits, and there's financial penalties to letting anything slip. And the upside to it is, I will say that banks, though they may not be known for being the most generous organizations— thinking Gringotts in popular culture— but in my experience, and, you know, of a large circle of friends, I suppose, who worked in banking at the time, and having seen other industries now, they're not tight when it comes to spending on security. And that was a huge advantage to get to work in that environment. If you made the case for it and it was needed, it got bought, you know, and I'm not sure we quite have that. I mean, I've worked in healthcare as well, and, you know, for a large private hospital group, and it's not the same approach. So, it's kind of a damning indictment of society, perhaps, that we value money more than things we should value. But, it's one area where it's much smoother. If you have a genuine case, you'll get the funding that you need. But when you— yes, a lot of it is very rigid. It must be done this way. It's kind of single-threaded through things. So, I just found you had to take that and maybe not dilute it, but just make it a little bit more flexible and make it able to move a bit faster as well. Almost agilize your program.

8:01Chris RomeoYeah. So, I got a chance to catch your talk at OWASP Dublin. Far from Greenfields: Introducing Threat Modeling to Established Teams. And I found myself sitting in the audience going, I really wish I could ask her a whole bunch of questions right now, in the midst of the talk. And so, as soon as I got back home, I was like, I'm reaching out because I want to do an interview because I want to be able to ask questions in the stream of this happening. It's really rude if you do that at a conference, I found, if you just sit there and ask questions in the front row, you know, while the person's talking. So, but I'm super excited to just kind of dive deeper into what you were talking about and ask some other questions about it. And so, kind of from, I guess, a place to start here, from your perspective, what are the unique challenges of introducing threat modeling to well-established software teams? And maybe give us some context about— you were telling me a little bit before we started about, you know, everybody thinks of threat modeling as always thought about as like, it's almost like the perfect state. Everybody assumes the perfect state for it. And I'd love to get your perspective on that.

9:03Sarah-jane MaddenYeah, so, and in some ways, and I'll come back to this, you do kind of need to go to perfect state in a very artificial way. But, you know, everybody thinks they understand what threat modelling is, and we do, okay? We do it from the moment we get up in the morning, whether you're a tech person or not. Subconsciously, you threat model. You think, you know, what am I trying to do? What could possibly go wrong? What am I going to do about it? Whether it's pouring your breakfast cereal or, you know, going out on a motorway. And the problem with that when you bring it into teams, and it is a more You know, I'm definitely not going to sit here and tell you how to threat model. There's much more qualified people who could do that. But when you bring it into teams, that ready acceptance of, oh, I know what it's about, can be a bit of a challenge because everybody just goes every which way doing it. And now couple that with what I found was that all— there's lots of material out there, and encourage anybody who's interested in threat modeling, and that should be everybody, to go, you know, leverage all that material. It's really good stuff, and the community has been very good in giving a lot of that, you know, pro bono out there, so you can get started and get to know what you're talking about without spending a penny, just your own time. The problem is, for simplicity and to make the ideas nice and clear, everything is based on, you know, the notion of greenfield sites, of, you know, or maybe startups or fresh ideas, etc. The reality is that many of us, I won't say most of us, but a lot of us were not working in greenfield sites. I was working at the time with very successful software teams where the software was older than some of the developers now, or the product was older. I'm not saying the actual given code they were working on, but the project, that's the true measure of a successful piece of software is if people are still using it and it's critical to their business 2 decades on. That's brilliant. Okay, then we've made something that really makes a difference. And when you go to then try and retrofit, hey, here's how you threat model, start with your design, because it's all very much— it's very much shift left— is you're suddenly looking whether you've modernized the software or you are dealing with a monolith is kind of immaterial. You're still dealing with a bit of a mammoth task. You may have, you know, you may have good practices like places like the bank where you literally everything has to be diagrammed and whatever. Or, you know, a lot of this software can come from the oldest startups in town, you know, and it means that maybe there isn't the same rigor there. And if you go to sit down and try to threat model without that, without your, you know, a diagram and a good understanding of the system, you're wasting your time. You know, you'll you might get some, you know, mad ideas. Oh, this could happen, but you're not really getting the value of kind of the systemic approach that threat modeling gives you. And that's, you know, where do you stop or where do you start with a team that has a large application like this? And that's what I found the books kind of didn't tell us. It was all like, so you've got this perfect world, you've got this well-understood product, you've got your documentation, you know, a lot of them, you know, you could go Google there how to start threat modeling and somebody say take your diagram, you know, here's where you insert the kind of oops moment and go what happens if you don't have one, you know. So asking a team to go back and, for example, diagram 20 years of software development, it's not practical. And so that's where we had to kind of modify how we approach this. Made lots of mistakes along the way. I'm not— I probably didn't have the realization originally of that everything I'd read, everything I heard didn't apply, but sure learnt it along the way. And you have to break it down for more established teams. You have to say, what fits now? What can we do? And what can we— then what can we continue to do? That's really, really important. What can we continue to do? And you'll eventually— And stop looking over your shoulder at the perfect greenfield site over there, you know, or that you heard about. And don't let perfection get in the way of progress, as they say.

13:22Chris RomeoYeah, yeah. And that's, you know, what you're describing here, I think, is something that, to your point, like a lot of people are struggling with this because they're not in startups where they're building a brand new— well, we're building a product, we drew a picture on a napkin, and we haven't even really coded it yet. So, we could do threat modeling, and we can really influence the architecture and the design where it's going. Like, I think I would say most of the people are locked into, hey, we've got this piece of software that, you know, has existed for a period of time. And, you know, I kind of grew up in threat modeling at Cisco, trying to figure out how to get different engineering teams at Cisco to do it. And I was in the same boat you were. Like, everything was existing. Like, it was running software on metal boxes around the world. Like, there was no, like, we're starting from scratch and let's try to figure this thing out. So, I'm curious though, like, Do you remember the first team that you went to to try to do threat modeling? Like, I'm just trying to understand how these unique challenges kind of ran into you. Was it like a bus kind of hitting you? Or, you know, what was the— like, so what was that first team experience like?

14:33Sarah-jane MaddenSo, where I was at the time, we actually had kind of an overhead architecture team. It was sort of the best and the brightest. And I decided that was a good intro. But I brought threat modeling. I was the security architect. at the time, and I brought threat modeling to them as a concept with the hope that they would go evangelize it on the teams they sort of served. And the problem with that was you're putting a bunch of very clever people in the room who got it, and we use sample diagrams and all the rest, and that was fine, but it didn't solve the problem of, you know, how do we actually bring this back to the teams? Initially picked teams that, frankly, weren't— so, these ideal teams, the architects were working very much in this sort of R&D space, didn't really produce anything that the teams that were really, you know, at the throttle pushing out tickets, you know, the whole time, didn't really get anything gainful for us there. I then decided that, with our engineering management, who were unbelievably patient and supportive through this whole process, and that's key, is getting that buy-in. We then decided, hey, we're just going to get all the teams to do it, and everybody's going to do the same thing. We're going to get these bright ideas. And that was chaos, okay?

16:00Mm-hmm.

16:00Sarah-jane MaddenBecause we could not keep up with them. It was, you know, small security team structure. We were literally one page ahead on the manual of what they were. And as I spoke about before, what happened was developers do what developers are excellent at doing. And, you know, when there's a gap, they solution for themselves. Okay. Sounds great. Okay. But in this scenario, what it meant was they were just making up their own rules and it was all over the place. So, that was— I had to kind of issue stop, stop, stop on that because we weren't really getting anything. Then, we'd grown through acquisition. Turned out there was a team we had all along who were doing threat modeling, and they just called it something different. And I decided, right, I'd sit by them, get what they were doing, and this would be so easy, just apply it to everybody else. The problem was that had brought me back a bit towards the greenfields as well. I sat in with them and realized they didn't even know how they'd gotten started. They were so long doing it that they had probably started when it was easier. They had all their diagrams and they just needed to— with each release, they were iterating, going, What's changed? You know, and they were doing it, they were doing it in quite a formal way, and because they did— what they did know was that it had originally come out of a contractual requirement, which they no longer had, but they had decided this is probably a good thing, we'll keep doing it. But they, they weren't really an established team introduced, they were an established team who'd done it all along. They were kind of that ideal world. So in the end, the, you know, the team where we actually struck gold were the teams that were moving at a very fast pace. And I'd avoided kind of interrupting them initially with a process that wasn't perfect, but they were the ones who were churning tickets very fast, very highly used products, very engaged customers with lots of change requests, etc. And we set the scope. That was the different thing. We just said, can you just please try this on what you're working on now? And they hadn't got the time to go making a cottage industry out of it. They said, sure, fine, we'll give it a go. And they were actually the best team to just keep going. They didn't sit there kind of waxing lyrical about it or anything like that. They just got it going. But setting the scope was key with them because they were working again on 2 decades of software, 10-year-old software, etc. And just setting it down to, okay guys and girls, what are you working on this sprint or this release, etc.? And let's just— you're going to miss things, okay? There is a broader system there, but little by little you build up the whole threat model and you're getting some of the threats out of the way. And as you evolve the model, we found that Again, ideal world would be to stop everything, sit down, investigate the system. But no, just that's setting the scope and little by little on busy teams. Annoying busy teams actually turned out to be a successful strategy.

19:10Chris RomeoA couple of questions that came to me as I was listening here. You mentioned that engineering management and getting their buy-in was actually something— it sounded like, I think you said it was relatively easy or you made a comment like that. Tell us a little bit more about that. You know, was there any— were they putting up any fight against threat modeling? Were they kind of like, we see the value, let's do it? Like, did you have to sell them on that process, or how'd that come together?

19:41Sarah-jane MaddenYeah, I did have to sell it. I will say it was relatively easy. There were some key members within the engineering leadership who got it instantly. There were definitely some skeptics, but they were sort of used to going, okay, let's, let's, let's give this a go. I'd done so much research, I was very convinced of the value of this, and I think that's kind of the first hurdle really to get over. So I'm, I'm no salesperson nor ever will be, but I, in that term, because I was so convinced of it, they were convinced too. Having background as a developer, so knowing what I was asking the teams to go and do, I think was a key thing as well. Going— I wasn't just coming in as a security person going, whatever you people do, you're going to do this as well. So you have to bear in mind, I was kind of— I was asking to introduce another obstacle to releasing a piece of software. You know, it was another thing to be done. I'm a very firm believer though that security is a quality measure. And, you know, the teams I was with at the time were very quality-focused, and that's— I came at it from that point of view. It was a slight mistake though on my part initially that I didn't understand the why. I did understand at that high level, at that almost greenfields, this is good practice, almost academic level of why we were doing it. And I convinced them on that basis, but When it started to go wrong and I sort of needed to convince again and I needed to convince kind of tiers of engineering, I found going back to Jira, you know, and I absolutely stumbled around in the dark in Jira. I'm no Jira guru, but I had the thought of, okay, how can I show something if I could find a ticket that wouldn't have happened that we could have reasonably predicted through threat modeling? You know, and we had various security tags and whatnot in Jira. And I went through it and I actually found multiple instead. And that's— and that helped me identify teams that were kind of ripe for threat modeling because I was able to say, hey, you know, in the past 6 months you've had this, this, this, and this all after production, you know, and they— that pain was always fresh with them then.

22:05Chris RomeoYeah.

22:06Sarah-jane Maddenyou know, we possibly— I never made any promises, you know, but we, you know, we possibly could have spotted this with this new practice. So you're kind of coming at it from 2 sides. Engineering leadership are well-informed, intelligent people who got the notion of threat modelling and why it was good. And then, you know, really kind of bringing it down to brass tacks with the, you know, the engineering individual team managers and some of the lead engineers and going, remember ticket 12345? Yeah, if we'd looked at this, we might have spotted it. Do you want to look at your next release that way? So, sort of, I suppose, data-driven. It might be a little bit of an elaborate label to put on it, but yes, I went poking in Jira and got some evidence of it.

22:51Chris RomeoAnd then you also mentioned kind of when you first were the developers were doing threat modeling, they started to do things kind of their own way, they started to kind of— I'm just curious, like, what were some of the methodologies that they came up with in those scenarios?

23:09Sarah-jane MaddenAnd it's funny, actually, because being a software company, as, you know, Accruent at the time, and I expected them to want to be a bit looser with us. And some of them kind of went super bank style. They were like really heavy and really rigid about what they did. And they were just, they were really invested and they went off and educated themselves. But what they did was, you know, in one instance in particular, I remember one team, it was like they'd gotten everybody's methodology from the internet and merged it all together so that you had to do all the things. And basically there was never going to be another single piece of software released on that team if they kept following it that way. They'd given themselves so many tasks. And because they were aiming for perfection, nothing was ever going to get out the door again. Very rigid. Lots of— everybody who'd put a template up there to help them, they'd adopted that. There was going to be so much paperwork, it was ludicrous. There were other teams who— like, threat modeling is that bit of it, just like development, there's science and art. Okay, so you don't want to curb the enthusiasm, you know, you don't want to—

24:19Chris RomeoNo.

24:19Sarah-jane Maddensquash the creativity. And some teams had looked at it and in a very logical way, you know, we had talked about tools, but if you're missing the point if you're just using tools to do threat modeling, and they'd gone, well, if I just draw the diagram, shove it through the tools, see what comes out and resolve that, done, you know, job's a good one. And this was prior to ChatGPT or anything like that, applying intelligence to it. So we had different extremes. We had teams that had just they'd magically reduce threat modeling on, you know, an older product, you know, a more mature product down to a 10-minute exercise. And when you dug in, it was like they were just shoving a diagram through Microsoft Threat Modeling tool, see what came out, marking most of it as a false positive, and away they went. And then others, it was overly heavy. It got a little bit tribal as well, is the problem when you've got a large number of teams. You know, I know many of your listeners may be, you know, focused on a small number of teams. We're talking about sort of enterprise software where there's, you know, X number of products with X number of teams. So it's kind of large-scale, and they were all developing it their own way and getting very married to it was the problem, whereas you want shared knowledge. And there's, there's an advantage in standardizing this or kind of any other practice But when you've got large teams, because one, it allows for developer sort of portability, you know, it gives them opportunities to switch to other teams if things are being done mostly the same way. There'll always be tech stack and there'll be little aberrations there. But as well, in terms of support, both supporting each other and as a smaller security team trying to support this, you know, this one-to-many with developers, if you're all doing it different ways, trying to follow all these different ways just doesn't work. But you're also able to build up within the developers themselves communities of practice, you know, for exchanging ideas and that when you've got something of a common baseline. But yeah, originally people got, yeah, very almost religious about their way of doing it, whether it's right or wrong.

26:29Chris RomeoYeah, and you mentioned the word tribal. And so, were there factions that developed? Like, certain— some teams followed the lead of other teams, and so there was the group of teams that were doing the minimum amount of effort, and then there was a grouping of teams that were doing the, let's put all threat modeling together and do every methodology.

26:50Sarah-jane MaddenYeah. And then there were some clever teams, actually. There's a third group who were floating, waiting to see which one would come out with the least work and they could just adopt it. You know, they were sort of pushing it off sprint by sprint, kind of going, yeah, let's see how this works out and we'll just take whoever's were sort of waiting. I sort of half admire those actually, even though they weren't contributing to forwarding it. But I was like, I like your tactic, it's kind of clever. So yeah, some people put a huge amount, and to be fair, people got so invested that some people put a huge amount of like their own time into it. But we needed to kind of keep it in check because it wasn't necessarily producing results.

27:28Chris RomeoYeah.

27:28Sarah-jane MaddenBut that always happens when you kind of stay in silos and turn in on yourself.

27:34Chris RomeoYeah. And that's a good— it's just another reminder that the enterprise is a different world than the startup, than even the medium-sized business. Oh, well, we have 2 products and 3 software teams. Okay. You don't have enough people to cause a giant fight about how we do this.

27:50Sarah-jane MaddenLike, you're talking about hundreds of developers. It's different. Yeah.

27:52Chris RomeoYeah. And it's just a different level of scale, but it's good to talk about it too because this is the type of stuff that a lot of times goes undiscussed. Like, people don't, they don't talk about the challenges. And so, you sharing this story is going to help some other folks who are likely in the same spot you were a couple of years ago, trying to figure out how am I going to do this across all of these disparate teams that, you know, want— are not happy with the fact they have to do more work along the way. So, let's transition and talk about some of the mistakes that you made because I remember in your talk that was There were some good examples and stories from that that really resonated with me, and I'd love for you to share those with the audience.

28:33Sarah-jane MaddenYeah. Well, I suppose I've already— I'm very quick to admit mistakes. I've already alluded to some of those grand ideas of, this is all going to be fine if I just let loose on hundreds of developers with minimum supervision or guidance. You know, it's great to have faith in your colleagues. They're all clever people. But if you're trying to introduce a new idea, you need to be responsible. for it and where it goes. So, the whole thing of introducing it to everybody was definitely a mistake. And the mistake of thinking I could leverage the team that were already doing it, that was in some ways a lower impact mistake. I spun a few cycles of my own on that one. But I think that's the lesson there, though, and this is something that, you know, of our company, that we have a whole business system around it, is experiment. Okay, your results of your experiment— a failure as such isn't a failure. You just learned, you know, as they famously say, a way that doesn't work. And that's what I was learning there was— but it was also learning that it was possible on big systems to do this, you know. So there were— there was good came from that, even though what I went in to do, which was like steal their homework and bring it to the other teams, didn't work, I learned lots of other things from them. So, even if you feel like you're facing failure, I'd say sit back, get a cup of coffee, and go, what have I learned from this to move forward? And help it formulate your next experiment. And, you know, that's how we then did approach the kind of super busy teams at that point. It was with the confidence of our mistakes.

30:19Chris RomeoMm-hmm.

30:22Sarah-jane MaddenNow, each time, human nature, think we'd learned from all the mistakes, you know, and we hadn't. There's always more to learn. And look, I'm going to be very honest and say we're still evolving on this journey, and I don't ever expect us not to be. That's, you know, and that's in no way being defeatist, that's being kind of optimistic about it. I think we'll always be able to evolve this process. Threat modeling is of such value that, you know, it would be worthwhile to continue to evolve it. And when it becomes part of what you do as well, that happens naturally. The push isn't as hard as it was at the start. So, that's my word of encouragement to anybody who's kind of hitting some bumps in the road rolling it out initially or trying to get some traction. It's worth it, and it will get smoother, and it will get easier. Other mistakes we made, I suppose, going to when COVID hit, we were already very much we had remote teams, very geographically dispersed. It's, you know, Fortive is very much a global company. That said, teams were inclined to be in the same regions, and some teams did transition well. Okay, they went from the whiteboard to Zoom, Teams, whatever, and, you know, we let them sort of pick their own tools. That wasn't a mistake, I have to say. You know, keep an eye on it, but whatever worked for them. Trying to— we went through a very short period of trying to— we were only trying to advise, trying to tell them what tools they should use. They had much better ideas as to what they should use to do it, particularly the teams who were starting to get into the rhythm of it. And we— I didn't keep a close enough eye on things once it was up and running. When you're looking after a whole security program, you kind of move on to the next thing that's going on and underestimated the impact of the change kind of with the pandemic. So people had gone into their own bubbles, you know, literally and figuratively, and people were now— there was less face-to-face even with regions. Our teams were kind of being moved around the place, and everything seemed possible with the remote. It was like, oh, everybody's remote now, so it's perfect. But it was things like threat modeling that started to suffer. And I didn't really notice that, to be honest. I thought it was up running, tickets were getting marked off. It was like I was reporting, yep, they're still doing it, whatever. And it was only when I got a reach out from a product manager— and partnership with product managers is key because they can control I should say manage, but whichever word you choose to use, uh, developers' time, you know. So we've gotten buy-in from, from them on the basis of that whole thing of looking at the Jira tickets going, you wouldn't have had these interruptions had you threat modeled. So they were, they were fairly well on board. But I got a reach out from a, a particular product manager who said, our last 3 threat modeling sessions have produced no tickets. And now, in hindsight, I think he may have been hoping I say, "Okay, you're done. Your software is perfect." You know, but he didn't say that explicitly, and I don't always get the nuances to the subtleties of what people are looking for. So hey, c'est la vie. And I decided I said, "Great, thanks for letting me know. I'll dig in a little bit." And when I did a little bit of snooping, as security people do, you know, I was like. Hey, when were those meetings and where were they and who went, etc.? Because people were living in this difficult reality, some of the teams had reduced threat modeling to kind of a looks good to me exercise in that, let's say, Chris, you were working the tickets and with no malicious intent, but it was just easier on your time zone. You did, you know, you did some diagramming, You proposed a design, you did the threat modelling, and you sent it async to me. I picked it up when I came on board, when I finished doing homeschooling or whatever else the new stresses of life were for me. I had no actual interaction with you. We didn't have that wider circle of, you know, different pairs of eyes, and I just went, ah yeah, that's grand, Chris, probably. And that's they'd taken the edges off the threat modeling process. So somebody was diagramming it, somebody was sticking it through a tool, and somebody else was having a look.

34:55Chris RomeoSo the collaboration was—

34:57Sarah-jane MaddenThe collaboration had just been watered down. That wasn't true of all teams, but it was definitely true of teams that were more physically separated, that this was happening because it was just hard to get time together. But what that showed that I think we could bring forward, you know, pandemic aside, because remote teams have huge advantages, but was that you really do need to get together for threat modeling to kind of address creator blindness. Creator blindness is one of the best things that threat modeling addresses because nobody designs a system to be vulnerable, you know, or if they do, they really need to take a hard look at themselves as to what their motivation is. But nobody typically goes out to do that, just like nobody designs any book into their system, any, you know, quality issue. So you need somebody, you need other eyes to come and look at it, and you need people to ask questions that they don't know the answers to. You know, that's— you have to encourage that safe space thing, kind of. You don't— this is not about me saying, oh, I see what's wrong and here's how to fix it. You can ask the question, go, sorry, how does that get from there to there, and what's to stop me doing X to it, and you miss all that when you reduce it to a quick diagram, shove it through a tool, and send it on to somebody else. But we caught it because of that, whatever the intention of the product manager was, he became something of the hero that we could have lost a lot of ground during that time, and we didn't. We had the opportunity to reset. We got back in, we did working sessions with the teams. We then found there were several teams that were kind of falling into this pattern, and it was very important to make sure that they were working sessions, that they weren't meetings. They weren't check-in, hey, here comes security because you've done a bad job, we're going to make sure you're all, you know, doing things as they should. It shouldn't be a checkbox exercise. So we come in, sometimes just our job to ask the stupid questions.

37:03Chris RomeoYeah.

37:03Sarah-jane MaddenYou know, and get that conversation flowing again and make sure people are, as you said, make sure the collaboration is there. They know the system way better than us. I'm highly opposed to this notion that you can hire security unicorns to drop in on a product that, you know, engineers have been working on day and night for X number of years and they're just going to magically— of course, they can bring expertise, but magically take all the security issues away and resolve them better than you can do.

37:31Yeah.

37:32Sarah-jane MaddenAnd that's something—

37:33Chris RomeoI'm with you there.

37:34Sarah-jane MaddenYeah.

37:34Chris RomeoI don't think that's a— I've never seen it. People can dream about it, but I've never seen— like, whenever I go to work on a new threat model with a team, like, I do a lot of the same things you're discussing here. Like, I ask questions. Sometimes I know the answer. Many times I don't know the answer. Many times I don't know if the answer is even realistic or if it's a crazy question that I'm asking, but I'm throwing it on the table just to get conversation going. And that's—

38:00Sarah-jane Maddenit's—

38:00Chris RomeoI don't care what the actual answer is. I'm just trying to get people that are being quiet to start saying, you know, they might be like, oh, that's not how it works at all. Oh, well then I just got a reaction from you, which means you're now in the game. And that was kind of my approach in doing it. So, if I draw out a couple of, I guess, best practices based on what I heard you say about remote threat modeling, one of them was let engineers— let them choose the tooling. that they're going to use, you know, for collaboration or whatever instead of trying to mandate, oh, you have to do it in a Teams meeting in this way. Let them choose. Make them working sessions. They're not meetings; they're working sessions. So, words mean things. And by changing the wording of what you're calling it can change the perspective. And then, yeah, ask lots of questions to help them unlock where they are. Any other best practices that you would throw in from your own perspective?

38:54Sarah-jane MaddenI'd go back to the tooling thing and say, if that all sounds a little bit loosey-goosey for somebody out there. Mandate the output. So this is what you've got to achieve. How you achieve it with, you know, make sure you're licensed for whatever you're using and have budget or whatever, but how you achieve it is up to you. But, and just mandate the output, and that gives teams something to— threat modeling is something that can get out of control very quickly, so give people some fixed points.

39:21Okay.

39:22Sarah-jane MaddenAnd both in terms of Mandating the output, setting the scope, exactly how much of, you know, particularly with these established teams, how much of this do you expect them to threat model at the time? And, you know, the short answer there really is whatever makes sense. But that can be a harder thing to establish than it sounds. But yeah, they're the main things. And just, I would say for anybody who's— look, this applies to engineers as well, software engineers as well. But particularly if you kind of have a responsibility on the security side for leading this, is keep an eye on it. Okay, make sure it's still achieving what it's meant to achieve. You do need to, you know, monitor and report. As a former boss of mine used to say the whole time, it is up to you to monitor and report. And advise. Okay, you don't know their product better than they do, but you do bring insights from the security world that While they're learning the latest framework, you should be educating yourself, be it on Twitter or whatever else, as to what's going on, what the threats are, and you should be bringing that to the conversation where you can.

40:32Chris RomeoSo, as we kind of come to the end of our conversation here, are there any— I guess, what would be your key takeaways or perhaps a call to action? Is there something you want to encourage our audience to go ahead and do as a result of what we discussed today?

40:52Sarah-jane MaddenYeah, I suppose be not afraid. It's, it's, it can be overwhelming, particularly when you're not from this ideal world of, you know, a fresh product. Don't, don't back off. Okay, just cut it into digestible slices. What can you achieve? bit by bit. And don't feel— you may get some pushback of, this is not for us, we're not that type of team. And that can even be after, you know, I was given the opportunity to speak at all-hands type events, engineering all-hands, and introduce the concept. And it was post that, the people go off and they do their own research, and they come back, you know, those who came back with the conclusion of, this isn't for us, this is for newer teams, you know. So don't let that put you off. It's just a different version of it, and it is achievable, and it is worth it, you know. So keep going and keep talking to each other about it. Not necessarily going to get an ideal world, but again, progress— progress is always preferable to no progress.

41:59Chris RomeoYeah, definitely. And sharing stories and experiences like this is a really powerful thing, and more people can do this, sharing what they've experienced. And one of the things I really did appreciate about your talk was you were just real about, here's what we did, here's where things didn't work out. And I learned more from hearing what people have tried and what the results were than by having somebody give me a lecture on, you know, here is the opportune way of doing threat modeling, for example. Like, it's that real world because, you know, we all work in messy organizations. that are not perfect and there are going to be struggles and mistakes are going to be made. And it's always good to learn from other folks as well. So, Sarah-Jane, thank you so much for sharing these experiences with us and with the audience. And look forward to another conversation in the future about whatever you might want to talk about.

42:52Sarah-jane MaddenThank you so much. I hope we're going to hear from more people with their experience. I'd love to learn from other people. And we've started the conversation maybe about other best practices or other just tips and tricks of how maybe they're progressing in established organizations. That would be kind of cool.

43:06Chris RomeoVery much so. I agree. Thank you so much.