MO Sadek -- Building an AppSec Program from Scratch

0:00Chris RomeoMo Sadek is a security transformation leader, passionate about staying ahead of emerging threats. He's built and fine-tuned vulnerability programs, customized solutions through hands-on coding, and driven security-first approaches in AI initiatives. Known for translating complex security metrics into actionable insights, Mo excels at uniting engineers, executives, and cross-functional teams. His commitment goes beyond technology. He thrives on fostering relationships and embedding security in the every corner of the business. Mo joins us to discuss how he built an AppSec program from scratch, partnering with other teams for success, and working on different teams to get to the root of how security needs to work.

0:43MO SadekThe Application Security Podcast is brought to you by Security Journey. We help enterprises reduce vulnerabilities through application security education for developers and everyone in the SDLC. Learn more at securityjourney.com.

0:55Chris RomeoHey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I'm the CEO of DaVinci, and I'm also co-host of this podcast, joined by my good friend as always, Robert Hurlbut. Hey, Robert.

1:17Robert HurlbutHey, Chris. Yeah, Robert Hurlbut, Principal Application Security Architect and Threat Modeling Lead at Acquia. And great to be back here again to talk about OPSEC.

1:26Chris RomeoYeah. Other than it's cold now, I feel like the last time we talked about OPSEC, it was warm, at least where I live.

1:32Robert HurlbutA little warmer. Yeah.

1:33Chris RomeoI can see the snow in your background there from the window where you live. There's probably feet of snow at this point, but luckily I live in the South where snow is a scary thing to everybody. But all right, we're happy to be joined by Mo Sanik. Mo, we're going to go right to security origin story. So the way I like to set this up, if there was a comic book about your security career, what would I find in episode 1 in that origin story?

2:03MO SadekSo it was kind of a fun thing. So episode 1 would probably be me with my clarinet not knowing what to do because I didn't get into my first school of choice for music. And I ended up getting into that school on something else with the pursuit of like me still wanting to do music, right? So I go in and I take music business classes and I'm still doing all the playing and I'm taking the performance and theory and I'm realizing, man, I— do not want to take 21 credits of like music classes and not know where to end up. So I end up hopping through a bunch of different majors, pre-law, chemistry, biology. I went into the business major and I was just like failing at most of these things quite literally. My GPA hit lows that I don't really want to talk about anymore. And luckily no one reviews that. So it got to a point where I go, I take a class in computer science, and lo and behold, I'm still not doing great, and things aren't clicking for me at first. And I talked to one of my professors, and I'm more interested in kind of like the hacking scene, you know, having grown up on movies like The Matrix and WarGames and stuff. And it turns out this professor was a CEH, and he was doing kind of like these small security courses and convinced me to go and take a couple of his classes. And lo and behold, I end up, he and I would create a curriculum that's focused on doing security stuff. So he had like an ethics course and it related to all the security. And it was the school's first hacking class because we didn't really have any, like nowadays there's like a computer security major everywhere. We didn't have any of that. So I was in a tiny school. So we just kind of had to make with what we did, right? So after doing all of that, I started looking for jobs and I can't get one because it was super hard for someone with no computer science experience and no like real programming experience. And what ended up happening was I looked on Reddit and I just started chatting with people on NetSec. And eventually I landed my first job at Rapid7 working on Metasploit. And that's kind of where everything hit for me, where I basically got a 3-month crash course in how to program like a hacker and how to do reverse engineering. And then I got really, I was really interested in bug bounties and application security. So from there I went and I was way more marketable at the time. So went to Cigital where a lot of people start their careers. Now it's Synopsys. From there, did a year of that and then moved to California to work at NBCUniversal and I just kept going from there. So, yeah, after that, it's been a couple of different media companies and a couple of different AppSec programs up until I went to Bosch, where I worked as a security engineer for their AI research team, which was cool because it was before AI was this big buzzword, right? So, I got to see a lot of really cool little tidbits of machine learning, really good machine learning. as well as the like first foundations of AI. And then Roblox, and I've been there since. It's been about 5 and a half years almost.

6:00Chris RomeoYeah.

6:00Robert HurlbutWow.

6:01Chris RomeoVery cool.

6:01MO SadekYeah.

6:01Chris RomeoSo way back to the beginning of the story. So, so, so how, like, are you like a legit clarinet player then? Like, you play, like, if I just, you know, pick, I don't know any music other than like Happy Birthday, so I couldn't even challenge you from that perspective. But like, yeah, I mean, so do you play other instruments or really just, is it just clarinet?

6:23MO SadekYeah. So I played, I played everything. So I played clarinet and trumpet were my big main ones. And then I also did vocal performance. And after that, like if you gave me a clarinet today, I could probably still play a couple songs. But otherwise I mainly play guitar now just as like an outlet, you know, I taught myself another instrument and I'll Got a small army of guitars now.

6:48Chris RomeoSeems to be a popular thing that people, once they get into guitars, then they have to buy like 17 of them.

6:55MO SadekWell, each one of them is different, right? Just like bike people, you know, it's like N+1 bikes. Yeah.

7:01Chris RomeoYou know, each one is a, is a different sound. And, well, this was Eric Clapton's, or this was Stevie Ray Vaughan's guitar. I gotta have one of those. Come on. I mean, maybe it'll sound like him if I play it.

7:14MO SadekYeah, so I still do that, do open mics here and there.

7:17Robert HurlbutVery cool. So jumping into the AppSec program at Roblox, you mentioned to us a little bit earlier that you were instrumental in building that program out from scratch. Can you talk to us about that? How did you build that or work with that program and getting it off the ground?

7:35MO SadekYeah, so that's a pretty good question, and it's a very long one, right? I think even if we, Fast forward to today, the AppSec program is still something that is being built and something that's being perfected and fine-tuned, I'd say. But back, if we went to the beginning, when I had joined, there were about maybe like 7 or 8 engineers total on the information security team, right? The company was a lot smaller, about maybe like 5 times as small as it is today. So, you know, we have this small set of engineers and I think there were only 2 or 3 of us that were focused on application security when I had first joined. So my first week it was me, one, and yeah, one other application security guy. Sorry, 2. And we each kind of claimed something different, right? So one guy was really focused on working in like, he wanted to work with the engine security team because there was a security team that worked, or maybe I should put it differently. The way security worked at Roblox before me, right? There's a bunch of different teams and we always talk about like security champions across an organization. Roblox hired a lot of people that were very security conscious. Even our like, so they had built out a game engine security team before I had joined. but it's different than like a centralized security team, like an information security team that we're used to. So, one of our first engineers on information security wanted to do a collaboration with that game security team so that they could kind of get a sense of what type of engine vulnerabilities we were seeing and how we could kind of tie that into a bug bounty program, right? And then me and my other colleague, we mainly focused on trying to figure out like, what does, what are the biggest risks? What are the types of applications and things that we want to see or look at? And what parts of the company do we think have, you know, that a big target that we can actually impact? Because again, we had such few people, it's like, what do we, what do you do? What do you spend your time on? So that's kind of where we set our focus. So That first 6 months of me joining, it was, I remember I had written a slide deck on a couple of different techniques that I had seen that I thought might be really interesting. You know, like, for example, if you had source code that got leaked, how were we able to like track exactly the time of the leak or the date and identify like exactly where that leak was coming from or stuff. So we had like conceptualized like, okay, maybe we can do some sort of like code watermarking across code bases and we can do like a unique identifier and figure that out. It was way too much work and it would consume a lot of time and it was not something that we wanted to focus on. There were times where I was literally just writing risk assessments with, you know, our legal team even and figuring out, okay, well, like, what are the biggest things that you wish that we could look at, right? Other times it was actually focusing with other engineers and figuring out what they saw the biggest pain points at Roblox were. So it was a lot of trying to figure out our landscape. In doing that, you kind of miss out on some of the everyday things that need to get done. So in parallel, we were also running a couple of vendor RFPs specifically around the automation stuff, right? So figuring out how we could get SaaS DAST. We had already been using a couple of different DAST tools. We already had some automation in place. Now it was about putting structure behind that so we didn't have to be, we didn't physically have to be everywhere and we could kind of start putting some soft gates in place. And again, that was like as much as we could do and we really were heavily reliant on other engineers and partners. helping us out there too. So I remember the beginning of that, and it was still a very difficult and challenging thing as like a single engineer. And when you're in a company that early, you can't just focus on the application security side. So as much as I'd like to say we were an AppSec team, we were really just a bunch of generalists, right? So it was incident response, it was forensics, It was infrastructure security. There were times where I was really just looking at logs and trying to figure out like what was happening where. I've missed holidays even, right? Because there were incidents. So, so yeah.

12:41Chris RomeoHow then, how does that, your experience working on those different disciplines within security, how does that overall impact your approach to AppSec? Did it change it at all with that? Did that experience help to scope and maybe change what you do so you do some things differently in AppSec than you might've done before? Or did it have no impact? Or like, what, how did that all come together?

13:05MO SadekYeah, it actually had a really big impact. One of the biggest things that I actually did, 18 months into my career at Roblox, I went to an infrastructure team. And I completely changed my job role, right? So, I went from being a security engineer to a software engineer on the telemetry team, because I wanted to figure out how other teams and infrastructure were building, how they were deploying, and I wanted to do it myself, because if we were going to ask people to do things for us, we should understand what the real roadblocks and challenges that they had were. So, I spent 2 quarters on that team, After that, I went into the security engineering org and actually built and worked with a couple of other people to build our first, like, threat and vulnerability management dashboards and scorecards is what we like to call them, right? So we built the first concepts of that, and we started to do, like, based on that experience with infra, it was like, okay, well, how do we implement this? How do we report findings in a way that infra can actually solve them and fix them. So—

14:18Chris RomeoWhat, what, quick refine the question. Were you dotted line to the infra team or did you like just change your job completely? Did you leave security and go to infra for a time with the— okay, so you, so you were, you were, there, there was no dotted line where you were going back to security. You kind of went all in and said, I'm going to go figure out how we do this. So what, like, is that something you would recommend other people do as a practice?

14:47MO SadekUm, no. So I actually prefer the dotted line model. I think at the time, um, I was very, very much like, uh, and I still am this type of person. I'm very much, I need to just like do it myself and I need to see what it's like. And I need to get in the weeds. If I'm going to do something, I need to like understand the whole thing. So I dove right in and I just sat in infrastructure and as an engineer and I saw things and I was held to deliverables at the same standard as those engineers. And yeah, I saw what it was like. But I had, yeah, no doubt in my mind. I still met with our security leader at the time. She was incredibly supportive about it. And she was like, well, if you still want to come back to security, you can. There'll always be a spot. And I'm like, I plan on coming back to security. I can't do anything else better. So, yeah.

15:46Chris RomeoHow did the other, how did the infrastructure team respond? Did they, like, were you one of the, were you considered, like, because it seems like it could be like, if I was on the infrastructure team, I might be sitting there going, oh, look, this guy from security's here. And he's pretending that he's with us now, but he's asking all these questions. Like human nature says that people might, might respond to it in that, in that kind of a way. Right. Like, so was that, and did you experience any of that or was it, was everybody just like, hey, Mo's with us, we're all in, we're going for this, we're getting it done.

16:18MO SadekYeah, no, everybody was really supportive. And I think the big thing about that is kind of the preparation you do and the reputation you build within the company. So I've never, in my entire security career, have never tried to be a blocker or like really push someone or force one of my solutions on someone.

16:37Chris RomeoOkay.

16:38MO SadekI tried that one time as a consultant, and I very quickly learned that, A, I, as an offensive security professional, I shouldn't be pushing this hard for something, right? If I do need to push that hard, it's because I didn't explain it well enough, and I don't understand the problem well enough. So I, after that experience, I actually left offensive security, right? And I was like, okay, if I'm going to do this, I need to do this from a point of partnership. So I always made sure that I was on good terms with those teams, made sure that I could communicate things with them on their level, or at least simplify some of the security things that they didn't understand to a point where they could actually do something about it. Security is everybody's job, so you should be speaking everybody's language when you're doing it.

17:24Chris RomeoYeah, that's, uh, I, I agree with you. Security is everyone's job. That has become kind of one of those things where some people don't agree with us. I do. I've been saying that for probably 10+ years at this point. And, you know, our mutual friend Tanya has been saying the same thing for a long time as well. Uh, but that's, so I wouldn't go as far as to say it's controversial, but it's, it's definitely something that Not everybody agrees that security is, at least security needs to be worked across the whole organization.

17:55MO SadekYeah. And I think the thing about that, if I can give like a hot take there, I don't think anybody really has a choice. And I don't mean this by security's going to mandate you to do something. Security's going to happen to you is what's going to occur, right? So, I mean, if we go at a real high level, we look at today, AI is like this big thing that everybody's like, whoa, like we got to be ready for it, right? It's every, everyone has a piece of the AI puzzle that they need to worry about securing in a way that doesn't maybe feel like securing it, right?

18:27Chris RomeoYeah.

18:28MO SadekSo whether it's building proper controls in infrastructure or across machine learning teams, they need to be able to build these things to be ready for compliance and regulatory standards, right? So whether they think it's a security thing or not, they know that it's a feature that they need to have ready, right? And it is a security thing.

18:47Chris RomeoYeah.

18:47MO SadekEven like your coding practices, the code cleanliness is a big security thing. Documentation is a security thing. Even though people don't think of it as a security thing, if you send a security engineer or anyone to go read your documentation and they don't understand how your solution works, they can't help you identify problems, and then you have vulnerabilities. So it's like these little things across everything that we do, or the development lifecycle, that all contribute to security in one way or another.

19:17Chris RomeoYeah. So, okay. So, I derailed you on your path.

19:21MO SadekOh, sorry about that.

19:22Chris RomeoNo, no, no. I derailed you because I wanted to learn more about something, but let me bring you back to where we were. So, you joined the infrastructure team, you then went to a security engineering role, and that's why I interrupted, but I do want to hear the rest of that part. So, you're in the security engineering role now, you've left infra, Take us from there.

19:40MO SadekYes. Yeah. So I go into the security engineering role, and again, this is a programming-heavy role. And while a lot of people will say, man, programming is like the one skill that every AppSec person needs, this is a good thing. I don't really enjoy programming that much. So I didn't feel like that's where I would be most successful. And I felt that there were definitely people that could do it better than I could. So it wasn't a role that I wanted to stay in for very long. It's a role that I wanted to stay in until we could start hiring people that could really take it over. And I just wanted to get like a base template out. And I think that was kind of agreed upon across the org. And the one thing I'll say about the security leadership and Roblox is that they made it really easy for me to kind of transition into other types of roles as the organization needed, and for me, for my own personal growth. So it was, at the time, it was relatively easy. Yeah.

20:38Robert HurlbutOkay.

20:38Chris RomeoSo security. And now you mentioned incident response. Did you security engineering? Like, when did you get to incident response?

20:45MO SadekIncident response was an all hands on deck kind of thing. So everybody was on incident response, right? Everybody was on the rotation. When you've got 7 to 10 engineers, you don't have the luxury of putting 2 people on incident response every weekend, right? So you do have to kind of shuffle that around. You have to give everybody their, their kind of slice of cake. And, uh, we did have like a little joke, um, in InfoSec, uh, even up until today, um, where every weekend when I was on call, there would be an incident. So there was like everyone's weekend and then there was Mo's weekends and everybody was like, okay, well, something's going to happen. And it always did. Even if I switched the days because I wanted, I needed something else. There was always an incident when I was on call.

21:30Chris RomeoSo, hey, makes life interesting though.

21:31MO SadekYeah, it got to the point where our manager was like, please take him off the schedule.

21:36Chris RomeoAnd you're like, I guess I can take one for the team by not being on the on-call schedule.

21:42MO SadekYeah, I was definitely crying after that one, I'll tell you that.

21:45Robert HurlbutSo, Mo, when you talk about, you know, the different approaches in terms of working with the different teams and being a part of those different teams, As you approach, let's say today, you've learned a lot of things. If you were to help some of our listeners in thinking about the foundations of an AppSec program, how would you describe those as somebody who's trying to get started with this?

22:10MO SadekYeah, so that first part, and I think it's something that I picked up in consulting, you need to understand where everybody's coming from. You need to understand if you're going to talk security with everybody, you need to understand the engineering part where you're coming from. So if, for example, you can't walk through like a cross-site scripting, for example, right? This is just a basic example. If you can't walk through it, then you're, it's going to be hard for you to be able to tell someone else how to fix it. So I would recommend like, you know, nowadays when I talk to people about jumping into security, I say maybe try engineering for a little bit. Try building a couple of things first, right? You don't have to be a professional engineer. You don't need to do it as a job, but maybe like just go build a couple of side projects, see what it's like, and then go into a security role and see how you can build from there. Or start off as a consultant, get all the training that they give you as a consultant, and then move from there. But you do need to have the ability to communicate with other engineers on their level. And there's actually like a couple of different speakers and stuff. Like, I'm so bad with names, so I wish I could call these people out, but maybe I'll share it a little bit later. There are a couple of really good, like, this is like the new AppSec engineer. Like, there are a couple like really good talks about that. And I think they all hit on this similar concept, even though they're all a little bit different. And what makes a really good AppSec engineer is someone who can speak from a place of partnership that isn't coming off as like, I'm the security person, this is my domain, you must obey by these things, right? It's the person that's like, hey, let me help you out. What can I take off of your plate to make security easier? How do we get secure to be like the default state for your organization, for this project? Like, what, what can we do to help? How can we be a helping hand? How can we actually expedite or make faster or force multiply the efforts of your engineering teams so that security doesn't feel like a job, it feels like something that we just do?

24:19Robert HurlbutYeah.

24:20MO SadekSo for me, at least, it's always been from a place of how do I make things easier for people? Foundation of AppSec is get the actual foundation, learn your basic standards, learn how to speak the language, Talk to as many people as you can. Like, I can't stress this enough. I think when I moved to LA, I went to all the meetups I could. I went to all of the little vendor things that like they were hosting. And I would just talk to people about security. And if I noticed that I was explaining something in a way that someone couldn't understand, I had found out, okay, maybe I need to find an easier way to explain that. If I found people just like getting bored when I was talking, I was like, okay, maybe I need to find a way to make this a little bit more digestible so that people will actually stay awake and not fall asleep like my mom.

25:06Chris RomeoOh, you know, so, so if you're building in, let's just, let's just say greenfield, you're building a new program somewhere.

25:15MO SadekYeah.

25:15Chris RomeoYou talked about the importance of partnership and, you know, listening. So, you know, I'm kind of summarizing what I've, what I've, what I heard, you know, the importance of listening and making people's— trying to make people's lives easier. But when we think about, if we start thinking tactically about an AppSec program, where are you going to start? You know, is there a particular class of tool that you want to embed early in the process? Is there, you know, what's your kind of playbook that you would run? Because, you know, automation is so crucial to companies these days, especially newer companies that are, you know, no longer startups, but are fast-moving, fast-growing companies, right? Like they don't want to do anything manually. They want everything to be automated from the beginning. Yeah. You know, that's just how they want to do it. So where would you start us off?

26:09MO SadekYeah. The first place is SSDLC, right? Figure out, and because this is the one thing where you can get a ton of bang for buck, a ton of signal, not great signal all the time, but you can get signal, which I think is the most important part, right? Even if you're noticing like like a ton of signal in one place versus another, you can at least see like, okay, well maybe there's something going up here, and you can get someone to investigate. The main point is you need to find some way to get visibility into what all the engineers are working on, or at least some of the risks that the organization face. So I look at SaaS tooling as like one of the big, like, first things to find. It's very available, it's very, everybody's looked at it, everybody will want to give you some SaaS, right? You can get tons of POCs, you can do everything you need to do to get information early, and figure out, like, good options that you can implement pretty quickly. So SaaS tooling is a good place to start. However, if you're looking to do something really cheap, and you don't want to pay for stuff, there's a lot of great open-source tooling that you can use for dependency checking, third-party dependencies, Open source, like I would just go for that as well. Again, we see so many of these vulnerabilities that come up that are based in outdated dependencies and vulnerable dependencies. So this is like bang for buck. Like I can just, you know, Dependabot is like a, like I'm thinking of like GitHub, right? Like they've built in Dependabot into GitHub, just the public offering. You could leverage tools and in all of your default, everything that you have to start with, just to go and see what's there. So I would start off with some of the scanning things that you can look into. I'm probably gonna get a slap on the wrist for naming a vendor, my bad. So I'll have to ask for forgiveness. The second part of that, while that's all happening, you need to understand what the organization cares about. So I know that that's not what people want to hear. They want to hear like, oh, go fix the highest priority things for us right now. You also need to understand that, I think at the same time, those folks need to understand, you need to understand the entire, or you need to have a good view of the organization. So what is the culture like around security? So you know how to interact with the engineers. What are the business priorities for the organization? So what are the teams that they care about the most? Which teams are pushing out the most code? Which teams are pushing out the most features? Which teams have features that are highly available, that are users, that are usually cost, not cost saving, but they're generating revenue, right? Because these are the ones that everybody's going to care about the most, the ones that will affect business, right? So it's how do we ensure that the business remains operational, that we are staying alive, and that we are avoiding existential risk. So that's kind of where I would focus if I had to pick like a second or a parallel activity. It would be doing a kind of this like holistic risk review of the organization to get an understanding of like the landscape.

29:35Chris RomeoSo how, how do you communicate this with senior leadership? Like, what are some strategies that you've seen that would help our listeners who might be struggling with this particular area? Because when people have a greed-filled opportunity, they come in, they start building a program, and they start putting pieces together. The senior leadership communication part is the thing that you can't get a tool for. You can't, like, you can't, there's no, there's no solution you can buy where it's something you have to know how to do. And so what, what are some things that you've been successful with in, in communicating up the status of AppSec?

30:19MO SadekSo there's a couple of different techniques you can use. I'll focus on 2 that I think about. Um, one of them is going to probably be the more controversial one. The other one is going to be controversial for a whole other reason, mainly because it's boring and tedious. I can't wait. The first one's the exciting one, right? And I say that is lean towards execution. And the startup space gets a lot of hit for being called, ask for forgiveness, not permission, right? So in this way, it's more like, figure out what the problem is and then solve it and bring leadership a solution before you've told them the problem and show them, hey, this is actually a problem and this is how we solved it. And we need to scale this up across the organization. I can't be spending all of my time doing this one by one, white glove service for every single one of our teams, for every single one of our engineers, for every single piece of code that we have. We need to be able to scale this up. We need resources. We need to prioritize these type of things. This is why this is the benefit that we get from it. And this is the shown use case and actual case study of how we were able to do this already. So it's kind of bringing leadership something you've already completed to show that this is viable. This is important. There's always the.

31:46Chris RomeoSo are there metrics? Are there metrics that you would bring? Like, because you're describing a scenario where you have like a success with one team, and I know as a senior leader, I would come back to you and say, well, that's great, but it's one— that's one proof point. Like, I need more. I need more data to be able to roll this out. So are there particular metrics that you're looking for? when you're doing this to be able to set yourself up to make that argument if you have a difficult senior leader like me who's going to say, I need some more data, Mo. I mean, that's great, but that's one example. Give me some data.

32:25MO SadekYeah, it depends on what you're trying to set up, right? So for me, it was setting up the initial partnership program. So for example, when I was speaking with my manager and I came back to AppSec after basically a whole year of a hiatus on infrastructure and security, I said, I want to build security partnerships really well. The thing was, okay, you can do whatever you want, but you have 6 months to prove that it has value, right?

32:53Chris RomeoMm-hmm.

32:54MO SadekSo what I had to do was go, I decided to tackle one of the biggest problems that our organization faced. I found a, that there was this recurring issue happening, and I was like, okay, well, let me figure out this bad pattern. and solve it for one team and then show how other teams are responding to this bad pattern just by default versus how this team has now responded to this bad pattern now that they, we've corrected course, right? And you can start gaining some metrics depending on what that problem is, right? So if it's something like key rotation, I think that's very easy to get metrics on, right?

33:32Robert HurlbutYeah.

33:33MO SadekLike this team, it, like, it takes them 5 minutes to rotate a key versus it takes this team 2 hours because they've never built a muscle for it or, any rotation type of thing, it's caused this type of incident, et cetera. In other cases, it could be, how do they respond to general incidents, or how many incidents does this team have, because they're closing vulnerabilities? What about tickets, right? Do your teams open tickets or not? Like, which teams are most involved with security, after we've had interactions with them, and how are they getting involved with security? So you can take different types of either engagement metrics, vulnerability metrics, incident metrics, figure out what's most important to your organization, right? So when I worked with the safety team, I knew that some of the most prevalent metrics in the safety space are incidents. So I measured safety incidents against other parts of our organization, and I said, well, if you look at how, like, even though this safety team may have a disproportionate amount of incidents compared to other teams because—

34:35Chris RomeoRight.

34:37MO Sadekyou know, they get some of the most public parts. Look at how fast they're able to respond to these incidents. Look at how many more they're closing. Look at all of the engagement we're getting from them because we are really showing commitment here and we're giving them solutions. So that's like really important. It's being able to bring metrics that show that you care about the things that leadership cares about, right? If this is a different team, right? And maybe if this was like, a team that is committing a lot of code, and they're more focused on, like, time to ship, then we need to figure out how do we reduce their time to ship, and how do we make it easier for them to ship features? Is it because the code quality is bad, and they are writing code that is vulnerable? Is it because they are, like, what is blocking them from doing that, and is it a security-related issue? How can security make it easier? Another good one is Trust by Design, right? So, that was another program that took a really long time for the company to kind of build a muscle around. When we started doing this, we were able to get ahead of safety issues, as well as architecture issues, and more importantly, able to start communicating what products look like across the organization. So, when you're getting that level of visibility, no one in leadership is going to say, oh, wait, we don't want that much visibility. It's more like, oh, we have a sense of what we're releasing and the quality of the features that we're releasing. So by just executing on these type of things and bringing the roadmap that you've already executed on and showing where it can go in the future, easy way to get kind of, A, acceptance. And then when you bring the metrics that they care about to support your vision in a way that, you know, shows that it's helping already, no one's gonna say stop doing that activity.

36:24Robert HurlbutYeah.

36:25MO SadekRight? Unless, you know, you're really short-staffed, and you have to focus on something else. But either way, they'll have this document that shows, or at least these processes and examples that show that this works, we should do this eventually.

36:38Chris RomeoYou mentioned, you use the term security partnership. And we certainly hear most people use the term security champion. What in your mind is the distinction or difference, I guess, between a security partnership program and a security champion program?

37:03MO SadekYeah, so I think the partnership and champions are 2 different programs in my opinion, right? So I've always seen partnership as kind of like a role that someone in security owns, right? So they'd say, hey, I will take this partnership between these teams, it's kind of like the overarching program, right? So we look at the partnership as everything that shows how our security team interacts with this partner organization. The champions are folks that we've identified in these different organizations as people who we kind of use as allies, right? So in one way, it's people that execute on the security roadmap without being a security member or member of the security team officially, or, you know, by whatever system you use to dictate employee roles. This is pretty important because when you can identify champions, you can also figure out, like, find bad patterns and things that people in these, in specific organizations are doing. And the champions are usually people who can say, Yeah, I know where this bad pattern stems from. I can go and fix this, right? So the champions are just a part of a partnership program. And the security partner on your team is kind of just like, or the security partner is the person from the security team that is doing the partnership with another person, if that makes a little bit of sense.

38:37Chris RomeoThere's multiple partners then in the way that you think about this? It's not, It's not like, Mo, you are the security partner we've designated from the security team and you're going to work with everybody. It's, it's broken down into, there's more, there's multiple security partners, it sounds like.

38:53MO SadekYeah. So like your security partner will be, it's kind of anybody who's taking, taking part of that security, taking part in these security activities, right? So your security partner is the security engineer who's partnering with someone else. your security partner is also that champion on another team, right? So partner, I know, can get a little bit confusing if we wanted to look at it from a very like academic or technical place. There are job roles in other organizations where they call their security engineers security partners, and that's the JD, right? Security partner.

39:32Chris RomeoOh, interesting.

39:33Robert HurlbutI've never heard that.

39:33MO SadekRight. Yeah. So that's kind of a little bit where I kind of modeled that from as well. So it was taking those like JDs, and job roles and understanding, okay, well, what would this look like if we implemented it at scale at this organization?

39:47Chris RomeoSo that's a dedicated, so that's a full-time dedicated role. It's not like it's like we think of champions as something that often, often 99.9% of the time gets added on top of a person's day job. It's like you have a day job and now you have this thing too. But it sounds like you're saying security partners in the way you approach this, this is a dedicated function. Like, this is my job. Like, I am basically not like an incident responder by day and then a security partner by night.

40:17Robert HurlbutRight. Right.

40:19MO SadekSo ideally, yes. Right. So when I was on the AppSec team, like, I always referred to myself as a security partner so that people could recognize me as like their security partner for this particular organization, for this project or anything like that. I would refer to myself as that because it was a lot easier than saying, oh, I'm like a senior engineer that is only dedicated to doing what you need me to do, right?

40:44Chris RomeoYeah.

40:45MO SadekInstead, I wanted to really create a relationship with the teams that I was working with. So I used the partner role whenever I communicated the things that I was executing on or any of those opportunities and the way I gave deliverables. was in a way that, again, would be communicated from a place of partnership, not a place of like, you're doing this wrong. It's like, hey, we could do this like this, right? So when you start to be a partner, you start to take a little bit of ownership and you get a little bit more respect than you would from just coming off as like this, again, this big brother type of role. I will say that internally, we still do, you know, externally, internally, we don't use the security partner role officially. it is security engineer, application security engineer, infrastructure security engineer, right? However, again, in my eyes, everybody's a partner on this, especially if you're owning the relationship, you're owning the, how, like, how we are interacting with that team. I would view that as a partner, someone who's owning that relationship.

41:52Chris RomeoYeah. Yes. And it's like, it's a next generation champions Yeah. You know, most people start with the Champion Program, but it gives you something else to grow into as well with this idea of, you know, making it a dedicated partner program.

42:07MO SadekYeah, I think when you build the Champions Program, it's really for the engineers on other teams, right? You're providing them training, how to think like security people, how to do threat modeling, how to do secure code design, right? How to even do architecture design from a security standpoint. When we look at like a security engineer, how do we prepare security engineers to like be part of the organizations that they're working with, right? Not just from like, oh, Mo works at Roblox, but how do we get like Mo to work with this team at Roblox, right? Like, how do we prepare you for that? So in my eyes, I've always thought of a security partner as like a way to kind of move towards that. Like, this is how we interact with this organization. These are their risks, right? We need to understand how they operate. We need to figure out how we can communicate with their executives, right? Because they have different priorities and they have different goals that we need to align with. So I think there's a lot more that can be built into a security partner than just like a traditional engineer.

43:10Chris RomeoYeah. Yeah, thanks for sharing that perspective. That's, uh, this is something that's never come up here before. on the AppSec Podcast tab, 275 episodes. Dirty partnership, and it's, we've talked about different, a lot about champions and a lot about different strategic ways to do, to work with champions, but this is the first time we've kind of unlocked this particular category or whatever.

43:39MO SadekI definitely can't claim it as like, oh, I figured this out, but there are definitely a lot of good organizations that have built these types of frameworks. that are really worth learning from. I know that personally, when I was working to figure out how I wanted my role to be in security, and when I wanted to figure it out, I spoke to someone who had built that program somewhere else, and I wanted to know what they thought of security partners as. And for a lot of people, these are just security engineers, so just with extra steps. Yeah.

44:16Chris RomeoAll right. Well, let's, uh, and with lightning speed, go through our lightning round. And with that, I'm going to turn it over to Robert.

44:24Robert HurlbutAll right. So lightning round, we have 3 questions that we typically ask. Uh, the first is, uh, what's your most controversial opinion on application security and why do you hold this view? Oh man.

44:37MO SadekUm, this is a tough one. So I, I was trying to, I'm trying to like think of like, controversial opinions that I have. Well, one, not every ad tech engineer needs to know how to code.

44:49Chris RomeoThat is a huge controversy. Come on.

44:52Robert HurlbutYeah, I know.

44:54MO SadekNot every ad tech engineer needs to be like a pro at coding, right? I think being able to be code literate, being able to read code in context, right? Not so much like understanding all the little functions, but being able to like read it and say, hey, I think I understand how your code works, give me, like, a little bit of time to read through it and come back. Being able to understand vulnerabilities that exist in code, and then being able to communicate that is really important. Again, I know that doesn't really fall under a typical application security engineer, but there are roles, or there are positions for these people that exist, right? They could be your application architect, they could be another type of application person. Or a security person.

45:38Robert HurlbutYeah.

45:40MO SadekThe other one I have is not everything needs to be like you're not reinventing the wheel every time a new piece of technology comes out. So like just knowing your fundamentals, I could go into that for days, but I won't. All right.

45:56Robert HurlbutSecond one is if you could display a single message on a billboard at the RSA or Black Hat conference.

46:02Chris RomeoWhat would it say?

46:04MO SadekOh man, that's really tough. It would probably be. Oh man, I actually I really can't even think of one right now. We'll give you a pass.

46:19Robert HurlbutYeah, we'll give you a pass.

46:20MO SadekThank you. Sorry.

46:22Chris RomeoSecurity is a partnership. There we go.

46:25MO SadekYeah, there you go. That's a good one. Yeah, there you go.

46:28Robert HurlbutAll right, and the last one is, what's your top book recommendation, and why do you find it valuable? And it doesn't have to be a security book.

46:35MO SadekOkay, cool. This is not an educational book at all, although there are educational books that I do like. This is just a sci-fi book that I enjoyed. It is called I Have No Mouth, But I Must Scream. It is a very short story. It is a, it's in public domain even. Just a fun read. I know that it makes, it seems like it doesn't matter, but it's technically about AI. It is actually about AI, and it's about what happens if we don't put the right controls around AI. So you should definitely read it. It is very creepy. Don't read it before you go to sleep, and definitely do not look up images for the book because they will be scary.

47:14Chris RomeoAll right, I'm going to check that one out. That's good. So Mo, what, how about a quick key takeaway then? What do you want to leave our audience with?

47:23MO SadekYeah, so funny enough, it'll probably be what you said would be the RSA billboard for me. It would be security is a partnership, security is everyone's job. I think those are the easy things to take away. But more importantly, make sure you're taking everything into context, make sure you are communicating. Communication is probably one of the most important skills that we are I don't want to say we're losing, but we are seeing more emphasis around being able to actually communicate security is almost as important as being able to implement security. If you can't communicate, you won't be able to implement, regardless of how good you are, because you won't be able to get the right people on your side to help move you forward. So learn how to communicate with the people you're working with, learn how to communicate with your management, And please write really good documentation. It will save you lots of headaches in the future.

48:19Chris RomeoMost definitely. Well, thanks for being a part of the show and for bringing some new ideas to something that we thought we talked about every— we thought we turned over every rock around. Turns out we hadn't. So we— I really enjoyed your perspective and hearing about your experience. And we look forward to— we'll have another conversation in the future and talk about something else.

48:43MO SadekThank you so much. Yeah, I had a pleasure. It was a pleasure. Thank you, Chris. Thank you, Robert.

48:47Chris RomeoThank you.

48:49MO SadekYeah. All right.