Adam Shostack — Threat modeling layer 8 and conflict modeling

0:00Chris RomeoAdam Shostack is a leading expert on threat modeling and a consultant, entrepreneur, technologist, author, and game designer. He's a member of the Black Hat Review Board and helped create the CVE and many other things. He currently helps many organizations improve their security via Shostack and Associates and advises startups. Adam is known for his work with threat modeling. In this episode, we take threat modeling to a whole new level as we explore the idea of threat modeling layer 8, or human beings, and explore the concept of conflict modeling, which is a passion project for Adam. I wanna take a moment to introduce you to Security Journey. At Security Journey, we believe security is every developer's job. We work with our customers to help them build long-term sustainable security culture amongst all their developers. Our approach is to provide security education that is conversational, quick, hands-on, and fun. We don't do lectures. Instead, we let the experts talk about what's important. The modules are quick, 10 to 20 minutes in length. We believe in hands-on experiments, builder and breaker style, that allow developers to put what they learned into action. And lastly, fun. Training doesn't have to be boring. We make it engaging and fun for the developers. Visit www.securityjourney.com to sign up for a free trial of the Security Dojo.

1:22Adam ShostackThe Application Security Podcast.

1:30Chris RomeoApplication Security Podcast. Here we go. Hey folks, welcome to this episode of the Application Security Podcast. This is Chris Romeo. CEO of Security Journey, and I'm joined by Robert Hurlbut today. Hey, Robert.

2:03Hey, Chris. Good to be here.

2:05Chris RomeoAnd Robert, what are— what would people refer to you as? I believe you said Threat Modeling Architect to the Stars?

2:14Yes, Threat Modeling Architect. I don't know about the stars, but—

2:17Chris RomeoYeah, we gotta go with it. Well, we're joined today by someone who has dedicated a significant amount of time to the world of threat modeling, and that person is Adam Shostack, who has literally written the book on threat modeling. Hey, Adam.

2:33Adam ShostackHey, pleasure to be here.

2:35Chris RomeoIt's great to have you again for your 3rd visit to the Application Security Podcast. And so I wanted to ask you, Adam, since you've been here before, we don't need to hear your origin story, but I'm curious what you've been up to since RSA, since you and I literally bumped into each other on the street in front of the Moscone Center You were looking at your phone. I think I was looking at mine. So we didn't literally bump into each other, but we did meet out there filled with, you know, 50,000 of our other friends all hanging around, but yet we still managed to bump into each other. So I'm curious, what have you been up to since RSA?

3:10Adam ShostackYou know, threat modeling is taking off. More and more organizations are saying, we need to be doing this. We need to organize the work that we're putting in. You know, for the 5-minute version of this, I borrowed a phrase, systematic, structured, and comprehensive, from Dr. Suzanne Schwartz, who's responsible for cybersecurity of medical devices at the Food and Drug Administration. They're thinking about how do they bring threat modeling into the way medical devices are constructed. There's just a tremendous amount of how do we do this? How do we refine our processes? And so I've been busy with training. I've been busy with collaboration. I've been busy taking some of the lessons that I've learned from so many people I've had the chance to work with and driving real threat modeling success at all sorts of places. And it's been simultaneously fun and exhausting.

4:32Chris RomeoWhat would you consider threat modeling success? Somebody who's done as much, as many different threat models as you have, when you walk into a room and you start taking folks through an example in threat modeling, What is success for you?

4:47Adam ShostackFor me, that's a really good question. I really like that question. And I've come to a very simple answer, which is that the people that I'm talking to about it want to continue doing it because it provides enough value that it is clearly a win for their delivery, whether that's delivering products, whether it's delivering services, For me, success is that they want to do more of it.

5:19Chris RomeoYeah, that's a great way to think of success when a lot of people would, I think, would qualify or quantify success of threat modeling a different way. I think that is a really great way to put it though, is if people are actually wanting to do it after you leave, then you might actually be onto something here.

5:38Adam ShostackWell, thank you. And, you know, a lot of people want to quantify that. But I find that engineers know what it takes to deliver a product. We've both worked at large companies. We've all worked at large companies. I didn't mean to exclude you there, Robert. The companies know how to deliver things, and the engineers are trained both formally and informally on This step is important. If we don't do this step, we deliver bad products and we're unhappy. If we can give them tools that are valuable enough that they want to do it and they will fight management for the chance to do it, then we can put some numbers around it, we can quantify it. But if the engineers don't want to do it, all the quantification, all the metrics, all the incentive programs in the world, don't result in more threat modeling. And the goal is not to threat model, right? The goal is to deliver great product that includes security that is appropriate. And threat modeling is the collection of techniques we apply to get us there.

6:58Chris RomeoYeah, definitely. And that's— yeah, I mean, I've had similar experiences in that, that big company world, and It's really a culture play when you get to the point where people are like, hey, you know, I see the value in this. It's where you're really starting to change the engineering culture because engineers are some of the toughest people on earth to work with because they have very little tolerance, at least in my experience, for things that they consider to be a waste of time.

7:24Exactly.

7:26Chris RomeoSo that quantification is huge and just, it's, that is a true, true metric of success. Well, we wanted to talk today about this idea. It's a little different, I think, than most people have ever really thought about in the world of threat modeling. We tend to associate threat modeling with the world of technology and features and releases and how are we going to fit this in our sprint. And today we've got a topic a little bit different, and that is threat modeling layer 8. And so, Adam, when we say threat modeling layer 8, What the heck are we even talking about here?

8:00Adam ShostackSo historically, when I talk about threat modeling, I talk about 4 questions. What are we working on? What can go wrong? What are we going to do about it? Did we do a good job? And within what can go wrong, we like to talk about things like STRIDE, so spoofing, tampering, repudiation, that sort of problem. As technology underpins more and more of the world, what we're working on starts to have social aspects. It starts to have user-generated content aspects. And we've seen this, you know, flame wars going back to the days of Usenet. And there's a There's a woman by the name of Wendy Grossman who's an author and historian. And in her Net Wars book, she documented how design choices that America Online— remember them— made when they created a Usenet gateway. Remember Usenet? And specific ways in which America Online represented Usenet to its customers generated flame wars inevitably, that the design of the technology either creates problems or it alleviates problems. Sometimes those are problems that are created by the platform, Twitter's use, and I'm mentioning these things as examples. I don't mean to throw shade on anyone here. But Twitter's decision to put all of your @mentions into a tab that you can look at necessitated the creation of block and mute functions. Yelp, when a restaurant is in the news, puts up an interstitial and says, hey, we know that you're upset about this restaurant for something other than the service and the food that it delivers to you. Maybe you shouldn't write reviews. These are technical choices which involve threats that are related to but different from the threats that we have traditionally threat modeled for. And so I've been exploring how do we take the same 4 questions and start to produce answers around what can go wrong and what are we gonna do about it.

10:47Chris RomeoSo when we say Layer 8, I've heard Layer 8 by different folks. I don't know where it originated, but I've heard the idea of layer 8 referred to as the human layer, kind of a bit of a joke on top of the OSI 7-layer model. You know, layer 7 is application, layer 8 is the human beings. And people will make those terrible statements like you can't secure layer 8 and things like that, which I don't agree with, but that's a whole other soapbox for another day. Um, is that when, when you use the term kind of layer 8, is that what you're thinking as well?

11:19Adam ShostackYeah, yeah. I'm thinking about threats that provoke people. I'm thinking about things that happen at that human layer and how we engineer to maximize the value and minimize the problems.

11:38Chris RomeoAnd so you mentioned your, your 4 questions that you are quite famous for. as far as in the world of threat modeling, at least. And so, I guess, do you see these same 4 questions applying to threat modeling Layer 8 and ultimately applying to threat modeling anything? Is this something universal that can be applied to any situation?

12:02Adam ShostackOne of the goals that I had in creating the 4 questions was to help people think through the problems that they face. And if you look at them, these 4 questions have nothing to do with technology at all, right? You can use this to threat model moving from one house to another. You could threat model moving into a new line of business. You know, what are we working on? What can go wrong? What are we gonna do about it? Those are intentionally very, very broad. I call them North Star questions of if you don't know how the work you're doing ties to one of them, maybe you shouldn't be doing it. So they act as a guidepost. And when we get into the human layer, We all sort of know what goes wrong, but we know it reactively. We're like, oh gosh, I'm watching this happen again. Wouldn't it be nice if when we engineer things, instead of waiting until the system has shipped to figure out what's going to go wrong, We anticipate it in some way and we start to plan for it.

13:32Chris RomeoSo, that's, yeah, it's definitely an interesting concept. And I think the best way to kind of help us work through this and then also help our listeners understand how this can be applied would be to consider this from an example perspective. And so, you mentioned a couple of different social media things here, and social media is such a big part of our lives here. I don't— you remember the days when it didn't used to be, but—

14:00Mm-hmm.

14:00Chris RomeoI'm talking about the Wayback Machine here to remember those good old days. But I think I'd love to kind of walk through this example of the abuse of social media systems. And I'd really love you to just go through and kind of walk us through, I guess, a mental exercise through the 4 questions in the context of the abuse of social media systems.

14:24Adam ShostackSure, sure, happy to. So we're building a social media platform. What are the sorts of things we ought to worry about? They include people making threats against one another. They include people impersonating one another. They include people trolling one another. They include people posting a variety of images that other people might not want them to see. And so this is a list of things that can go wrong. And we can ask, what are we going to do about it? So for example, we could create a filter that says images that don't meet our community standards, will only be displayed if you've checked a preference. Okay. Turns out that people have done this. There are lots of examples of such filters, and there are lots of debates which have occurred around them. Again, just as an example, should we allow Should we allow partially unclothed people? What is the minimum level of clothing that a network, a social network should support? I don't know what the right answer is. I suspect there's not a right answer, but it turns out that this has played out And as an example, Facebook doesn't like pictures of mothers who are breastfeeding. Is that right or wrong? You know, you can argue either way. I think we need tools that help us anticipate this question and say, here are the collected pros and cons about this question that you should consider when making a decision so that you're not reinventing the wheel. Another example is real name policies, and I'm using these as examples not to delve into the details of the policy and not to be criticizing Facebook, but to say there are— we can observe what is happening. We can anticipate, for example, there's a great deal of research that shows that trolls will actually troll harder under their real name. than under a pseudonym. And so the listed justification for real names on social media sites doesn't play out the way program managers or developers necessarily expect. It also has side effects for certain classes of people. where they would like to be able to use the social network in a way that doesn't expose their whole life to everyone around them. And so they use a nickname or a stage name or an assumed name. And when they are outed, or when that name— when their names are tied together, it has predictable negative effects. Why don't we have a place you can go that says, here's the pros and cons of your real name policy, so that when we're developing systems that work with humans, we can threat model them not on a brainstorming level, but on a structured level? Does that make sense? Is that specifically crisp? to help people understand.

18:51Chris RomeoYeah, I mean, I've got a follow-up question here. So this is almost like a document or a knowledge base is what you're describing here of— let's use this real name policy because we've been kind of using that as an example here. And so you're describing a cat— not a catalog, but a, I guess, a knowledge base of of thinking through all of the design time issues that have to do with real name policies, perhaps based on research, perhaps based on anecdotal experience, based on experience from different social networks that have been in the real world and have actually had to do this and seen some results. And so, is that kind of where you're going? Is this catalog that's been pre-filled out or—

19:43Adam ShostackYeah.

19:44Chris RomeoLet me stop there. Is that where you're positioning?

19:47Adam ShostackI think that is an important step. Let me, if I may, plug a project I'm working on, which is a GitHub site to collect this information. GitHub, Adam Shostack, conflict modeling. There, for example, I've got a taxonomy of images. around clothing, symbols, behavior, representations like racial and religious stereotypes, legal issues like copyright violations and insulting the king. And so my goal is first to catalog, and then when we have catalogs of both the threats and the controls, we can start to build out threat modeling processes, methodologies that help you actually take this from here's a list of problems to here's the way to think about this as you're building what you're working on. And so the catalog is an intermediate step.

21:01Chris RomeoThere's been a need for a certain amount of what I'll describe as freestyling, brainstorming, All of those pieces kind of coming together. And so what you're describing here almost seems like this is more of a policy-related focus as the end result versus an individual modeling exercise that would occur with the individuals responsible for the system. So is that kind of how you see conflict modeling coming together? Am I reading this correctly or am I missing something?

21:36Adam ShostackSo I haven't presented it well. I actually see this as the sort of thing that you would do as you're building features for your system. It's not a policy exercise. I mean, you could use it as part of a policy exercise, but this is, if I have a feature which is upload photograph and display photograph to people, What threats exist to that feature? There's a set of technical threats where I might upload a JPEG that exploits a buggy version of libjpeg and get remote code execution on your site. We worry about that as we're building the photo upload feature. I am arguing that the people building a photo upload and display filter feature also need to think about how can that feature be abused in ways that will threaten the human users of the system or act as a threat to other users of the system.

22:49Chris RomeoOkay, so, okay, this is— it's, it's coming together for me now.

22:54Adam ShostackYeah, and this, by the way, yeah, this is, this is super exploratory. Let's figure this out together. The only claim I'll make, 2 claims. One, I think this is important, and 2, I don't have it fully worked out.

23:09Yeah, but it does sound like, I mean, there are some similarities to how we would do threat modeling of technical systems in that you do have a goal of improving design, the features, understanding it better, and ultimately arriving at some solutions to those to those problems, right? So, there are some similarities. I mean, that's— so, yeah, when you said it's not really a policy, that kind of resonated, and especially that last example.

23:36Chris RomeoYeah, and I can see the, you know, now based on this example, it does make a lot more sense to see kind of from the, I guess, the taxonomy of how these things fit together. So, your upload feature has, like you said, some tech threats we got to think about, you know, remote code execution in a particular library. But it also has some of those abuse/conflict-related things. And there's a whole other set of issues there like what is the community standard, like you mentioned, for what can be uploaded? And what about personal privacy of posted images? What's the expectation? And I guess my follow-up question then for both of you to consider is, Are the tech people really the best people to be answering these questions about— on the abuse/conflict side?

24:29Adam ShostackIn the sense of the best is the enemy of the good, yes, they are the best people.

24:35Chris RomeoTouché, touché.

24:38Adam ShostackBut, but more seriously, the tech people must answer these questions.

24:48Right.

24:49Adam ShostackOne of the things we saw with the tragedy in Christchurch was a million uploads of these awful videos of violence being committed. There isn't a policy, and the policymakers, the Prime Minister of New Zealand, the President of France, have said that we need better technical solutions. And so whether or not that's easy, feasible, the policymakers are pushing to us, the technologists, to get better at solving these problems. And so as much as many of these questions are the sort of difficult, fuzzy human issues that people writing code would prefer to not deal with. We've made systems that make these problems important to society, and we need to think about how we— well, we don't need to. Let me pull back a little bit from that statement. Either we come up with solutions that are acceptable to the societies in which we live, the ways in which they're changing and the ways in which the world are evolving, or we will see laws passed that impose solutions upon us that are much— that are impossible to comply with because they assume that technological things are easy. It's just a small matter of code. Why can't you write code that recognizes the video the same way a human does? Stop arguing with me and go do it, please. That is what we will hear from the policymakers. And so the better we are at answering these questions with technology, the better we get at answering these questions with technology, the happier we as technologists are going to be and the happier that the societies in which we live are going to be. And some of these are super difficult. Some of these are incredibly difficult questions that have only wrong answers. I'm not trying to say, oh, we're gonna go solve this. I'm trying to say that we better start getting explicit about what the solution— what the problems are, what the solutions are, what the trade-offs on those solutions are, So that we can have a more intelligent conversation than just saying, oh gosh, I'm sorry, this is difficult. I don't want to work on this. I'm going to do something better, something else, something more fun.

27:44Chris RomeoSo if we, if we kind of pull back now, now that, now that we've kind of explored and walked through this, I feel like I have a, a much better understanding of, of what you're describing as conflict modeling. And I see how— I see that there is a void here. in that we don't have a lot of folks who are focused on these particular issues, but yet we're still building systems and cranking them out. And the last thing we want is legislation to come out that says, hey, here's what you have to do. If we kind of pulled back for a second and said, for somebody who might be listening to this and might be thinking about, hey, I want to try to apply what Adam and Chris and Robert are talking about here to the features and things that I work on. How would you imagine them getting started with this whole idea?

28:36Let me see.

28:38Adam ShostackStep 1 might be to visit the GitHub site that I'm building, and I'm happy to take pull requests. I'm working with a couple of people who are playing with this inside their own organizations, and this is a passion project of mine, so it's just, you know, we get on the phone when, when we can. But give it a shot. Take on the project that you're taking on. Describe what you're doing. This is not competitive advantage sort of things. These are problems that we all face, we all need to work on. Jump in and do, share what you're doing, pull requests, make a phone call, write a blog, post about it on LinkedIn, join the OWASP Threat Modeling channel. Let's talk about what we're doing and learn from one another.

29:41Chris RomeoDefinitely seems like a fledgling project across a lot of different— that could impact a lot of different people. And definitely does seem like the right phase is learning right now. And so, what do you envision kind of coming out of this in your GitHub repository where you're collecting various things and starting to put together some thoughts? What do you think is going to be the final output of this? Is it going to be a guide that that somebody who's not well-versed in conflict modeling or even threat modeling could take and apply in their job or their role? Or what do you see as the outcome?

30:22Adam ShostackSo, so I see a couple of outcomes. And before I get to the outcomes, I want to mention something about collaborating on this. This is simultaneously a technical problem and a human layer problem. People who wanna participate in this have an opportunity to jump in and make some pretty fundamental contributions without writing code. And that's an unusual combination for our space. And so let me jump to the outcome and the thing that I think is most useful here. And here I'm drawing on a law professor, Amanda Lewandowski, who's thought about this problem a lot. And she says the problems are relatively easy to discover. What's most important is understanding what we've already learned about the trade-offs and how to make the trade-offs well. And so I think one outcome might be tools like Stride or attack trees that help us find the problems. Another outcome is tools that help us understand or implement the solutions.

31:41Right.

31:42Adam ShostackSo maybe that is a— maybe that's a trade-off guide written for program managers that says, here's the pros and cons of real names. You must be this tall to engage in the debate. Or here's an image parsing library that allows you to send an image off and get flagged for these 6 types of inappropriate content. Maybe that would include it detects religious text or it includes political symbols that people might choose not to display. such as a Confederate flag or a swastika, and it includes documentation. Hey, if you're going to ban swastikas, which, you know, I would hope we all think at the 50,000-foot level, yes, let's ban the swastika, it's an offensive symbol, are we also going to ban historical photos of swastikas? Right? Are we going to prevent people from posting historical photographs from Nazi Germany? Are we going to ban Hindu use of swastikas on our site? And you could document that in a library that says, hey, this library will find swastikas in images. Here's the implementer's notes for the decisions you have to make when you're using it. So that's a contribution someone could make. I think we can build tools to help us find problems and tools at both the human and programmatic level for dealing with them.

33:38Chris RomeoSo as far as conclusions go, Adam, what would you leave our audience with as kind of a final thought regarding conflict modeling and threat modeling layer 8?

33:52Adam ShostackWe live in interesting times.

33:57Chris RomeoWhich is both a curse. That's mainly a curse, right?

34:01Adam ShostackI don't know if it is actually an ancient Chinese curse to say, may you live in interesting times, but I understand the sentiment in thinking that it is.

34:16Chris RomeoSo, Adam, how can people best connect with you to continue the conversation?

34:24Adam ShostackThe GitHub page, I am easy with pull requests. If people wanna reach out via LinkedIn, if they wanna drop me email, [email protected], S-H-O-S-T-A-C-K. And all of this consideration of these problems has made me pull a little bit away from the mainstream social media. So LinkedIn, GitHub are both great ways to go, as is more traditional email. And I would— I really think that this requires collaboration. It requires people from diverse backgrounds, diverse perspectives, different skill sets coming together to figure this out. And so I am eager to work with folks on what I believe is a really important and useful thing.

35:22Chris RomeoThanks for listening to the Application Security Podcast. Our intro music is 8-Bit Kung Fu by Born and TJ, and our outro music is Southern Delight by Stefan Kartenberg. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. security-podcast. You can also find Chris on Twitter @edgeroute and Robert @roberthurlbun. Remember, security is a journey, not a destination.