The Cringe Test: Why AI Can't Do Privacy Threat Modeling for You

0:00The things that I've seen is that it still lacks a lot of nuance. It still misses that broader context that we were talking about because it's not just understanding your architecture. You need to understand how it integrates with everything, what data exactly it is, what the type of individuals are that are related to it, what the expectations are of those individuals. Are you processing medical data because it's a hospital or are you processing medical data because It might be fun to build some social media app that is, I don't know, connecting you based on some kind of medical issue. It might do the same underneath from a technical perspective, but the privacy implications are completely different. So all of that needs to be captured. That, and also I think the understanding of what privacy threats are compared to the understanding of what security threats are is still not as mature. So. If we don't know it and if the body of knowledge is not in there, then the body of knowledge will also not be in, in the AI models.

1:02Chris RomeoDr. Kim Voets and Avi Douglen bring deep expertise in privacy and application security. Kim is a privacy engineer at PwC Belgium who helped develop the Lindun Privacy Threat Modeling Framework. Avi is the founder and CEO of Bounce Security and leads the OWASP Threat Modeling Project. Both co-authored the Threat Modeling Manifesto and help teams build more secure, privacy-respecting software. Welcome to another episode of the Application Security Podcast. I'm Chris Romeo, joined by Robert Hurlbut, and we've got some friends slash guests who both have been on the podcast before. Kim Wutz, I'm sorry, Dr. Kim Wutz and Avi Douglen, both well-known names in the OWASP community and a lot in the AppSec community, threat modeling community, very, very, very well known in a lot of different communities. And they've both been partnering together on privacy. And so what we're going to explore in this episode is 2 different things that they've been working on and ultimately how do those 2 things mingle together. But Kim, I'm going to start with you for the first question because last time you were with us, it's been a number of years, but we were talking about privacy threat modeling. We were talking about whiteboards and how to put this into practice. Now we're talking about a deck of cards and a verification standard. So what does the game context and cringe get us that the whiteboard couldn't get us to? And Avi's doing a product placement right there.

2:34Oh yeah, I had it laying ready for me as well to show. So I still stand by Linden and the Linden Card Game or card deck as a means to actually find your privacy threats. And I still stand by the whiteboard. What I think was missing, and that is also, so basically Context and Cringe comes from numerous discussions me and Abbie had about how to actually explain to people what privacy is, why it's important. We have been doing training together and we had at some point a participant who The first day we gave them some case. It was a very privacy, icky case, but it sounded like it's a good idea to, I don't remember, it was something with AI-assisted something because back then that was still futuristic and now we have it. But in any case, he was the first day, he was like, oh my God, this is amazing. I'm so gonna implement this when I get back. And we explained privacy and then we used that case to do a threat model, and you could see him after he reflected on it go like, oh my God, this is, it's horrible. Oh my God, it's, I feel it.

3:52It's awful.

3:53So, that made us go like, okay, that is the experience we want people to have, but ideally not having to go through 2 days of training just to get the Aha moment. So that started a whole discussion and thinking process, and that brought us Context and Cringe, which is the complete opposite of let's sit together and really think about privacy. It's let's play a game, let's make it awkward, and at the same time, we're actually thinking and talking and feeling privacy without actually making it explicit. So the whole, the whole goal of the game is, I'm gonna do some more product placements. It comes in a pretty box. It has a lot of pretty colors, but basically you have a bunch of cards. You have application cards, we have feature cards, and we have data cards, and you will combine those to create a scenario. So you each need to end up with a scenario of those 3 cards. And ideally what you build is something decent, something you can sell, something that you go like, okay, people wanna use it. And what you want the opponents to have is something cringey, something that you go like, ooh. And once you have all built your scenario, you will vote, would I use it? Or is this icky and does it make me go the feeling? And while you're playing it, so you can either give yourself cards or give the opponent cards. there's automatically some discussion happening because then you go like, haha, I'm giving you this weird application. And you see people go like, haha, giving me something nasty, but you can counter it with something that actually fits in that scenario or the way around where you go like, okay, this is a decent application, but if you combine, I don't know, a banking application with your sexual preferences, probably something icky going on there. So by combining, by trying to block each other, you make people think of, okay, suddenly the context changes, but ooh, that has an impact because now I never wanna use it anymore. So people are thinking about it. People are going, haha, we're feeling it. And you can see it land really well. It also is a great icebreaker to take in those workshops where where you teach threat modeling, or where you have an actual threat modeling session to, to get people the awareness of, okay, this is the kind of privacy issues that we're trying to prevent.

6:33Chris RomeoThis episode sponsored by Corgia. Design it, build it, ship it. Corgia secures it. Corgia is an AI-native app security platform covering your whole software lifecycle. From the first architecture diagram to code in production. It brings design reviews, AI-powered code scanning, and autonomous pen testing into one platform so your team spends less time chasing noise and more time fixing what matters. Visit corgea.com. C-O-R-G-E-A.com. Corgea, one security platform for the entire SDLC. So Avi, it's about the aha moment then. That's really, if we funneled it all down, it's not really about teaching people or having them memorize all the available cards and all the privacy things. It's about that light bulb moment where they're like, uh-oh.

7:27I wouldn't even call it aha. It would be more like a duh moment. So, well, you know, like Kim said, we were teaching this and we did a great job of teaching people and they understood it. And we realized from that moment, that visceral reaction, there's a world of difference between understanding it and getting it. Because privacy isn't just bits and bytes, it's very subjective. It can be very personal. So, Context and Cringe, it basically is about 2 things, the context and the cringe. The cringe part of understanding to learning how to recognize when you get that cringe. And actually making that cringe be a primary part of the analysis that you're doing, because you might be okay with it, but then you realize that somebody else is not okay with it because it's personal, it's subjective, it is very contextual. And that's where the second part comes in. Just because, okay, the sexual preferences that Kim said, sometimes that's perfectly okay if it's a dating app. And that's fine. If it's a banking app or a school management platform for parents, that's really gonna give you the ick. So it's both recognizing the cringe, learning when that is useful as a useful tool as part of your analysis on the one hand. On the other hand, realizing, well, it depends and there's layers of context there.

8:54Chris RomeoAnd so you really want folks to take away that cringe detector almost. So when they're building something within their own company, they look at it and that the synopsis starts to fire in their brain of like, wait a second, this doesn't seem right. And now they have a foundational platform underneath to say, now I understand. It felt weird when I was looking at this feature, its design, but now I have some concrete things that I can point at that go, We don't need to store this data for a school management app.

9:29Yeah.

9:29Chris RomeoAnd they can make an argument from there.

9:30Just, just getting that gut feeling.

9:33Yeah. Cringe detector, as you say, but also put it in context. And sometimes it's the context that makes you understand the cringe, and sometimes it gets you out of the cringe. And it is very interesting. You know, we, we've played this at, uh, at AppSec Vienna. We did a couple of pods and a couple of rounds there, and people were very creative. And you see that, aha, and they're like really cringey starter combination, but then they're able to twist it with another card and change the context completely. And we call this cringe flipping. It starts off really bad and becomes okay, or the other way around.

10:08You call it cringe flipping, but yeah. Um, yeah, Chris, Chris, what you were saying, having that foundation, it's, it's not necessarily that this game is gonna teach you like, okay, now I can do a full privacy analysis. It's more about that initial, wait a minute, we should dig deeper, like pause and let's have a, let's have another look. And then all the rest comes after that. But this is really just to raise that red flag and go like, okay, something feels wrong.

10:37The other thing I found interesting is when there's not consensus, sometimes people will say, this is okay. And other people will say, oh, hell no. Or the other way around. You think that it, because it is subjective, you're always putting it into your own personal context. We did this the first time, the kind of a beta testing at DEF CON a year ago. And there was one person there and clearly a parent of small kids, and she was always putting this in that context, even when it wasn't about that. And she was always reacting from her gut about her kids. Where's everybody else talking about something else? And it makes sense, but it's personal.

11:18Yeah. And that's a good reflection, right? Because it's not because for you it's fine. If you see everybody around the table go like, wait a minute, this is not okay. So again, to readjust your gut feeling, if all the rest is saying, okay, this is weird, and maybe your feeling of cringe when it comes to personal data gets a bit readjusted while you're playing.

11:41What do developers love saying about bugs? It works fine for me. That doesn't work with privacy. It works fine for me doesn't work because everybody else will have a different context.

11:51Yeah.

11:52Chris RomeoAnd DEF CON, tough audience to do a beta test at. Like, that was a, that was a daring move on your part.

11:59I remember that.

12:01Yeah. And we got such great feedback that we're like, okay. we're onto something here.

12:06Chris RomeoIf you can, if you can make that crowd happy, then, and I'm not saying it's a bad crowd, they're just a super technical, it's a, it's a conference that's not marketing driven for those, those that haven't been there. And you really don't understand DEF CON until you walk around DEF CON. Like you can read about it and say, I think I understand it. And then you get there and you're like, okay, this is, this is the, the, the, the collection of the most dangerous people on planet Earth. And A lot of them couldn't beat anybody up. It's not a physical danger, it's a capability danger and knowledge danger that they have just because they just know how to break stuff. So tough potential audience. If it landed well there, it's going to be well received a lot of other places. So Avi, let's switch gears for a second because I said we want to try to funnel this thing. So we've got a game, but we also have a new set of requirements for privacy. And so, I haven't really heard anything about APVS at this point. So, give us kind of the basics and build up what APVS even is.

13:12I want to back off half, 2 steps backwards to talk about why we were doing this. Very often, and Kim heard this even more often than I did, it's like, oh, okay, privacy's important. And they get, you know, we're not just talking about Compliance regulations, but we need to build this into the system. We convince people. We, Kim and I, have been talking about this for years. I've been talking about it for a couple years. Kim has been talking about this for decades. We need to build privacy into applications. You convinced me. Okay, now what? How do I do that? What's next? So we took a cue from ASVS, you know, the Application Security Verification Standard, which I'm assuming most of your listeners are familiar with. And went along the same path to create very specific, implementable, testable requirements for privacy under various chapters and headings and so on. For application privacy, I'm building a system. I want my application to be privacy-respecting. How do I do that without becoming an expert or without hiring him? We wanted to give a bunch of very specific requirements, like ASVS, a set of default technical requirements that an architect or product manager can take and borrow into their system design, feature design, to the level of, it's not just generic, you know, high-level 30,000-foot principles, but actually something that gives me the information that I need to implement this as a technical feature. So we have a few chapters, things like the architecture and data mapping. We have a whole chapter on data minimization. There's high-level, that's a high-level principle, right?

14:55Chris RomeoThat's a tough thing. And that's what I found when, when I first met Kim and I started looking at privacy, that's a hard concept. Like, I understand, I can explain the 20,000-foot view of data minimization. Like, we're trying to limit what we're keeping, what we're storing, and throwing away things that we don't have a good business reason to keep. But after that, I don't have a great, I have to call, pick up the phone and go, hey, Kim, what do I do next? Like, what do I do? So let's pull on that data minimization thread for a second because I'm truly, don't feel like I have a great grasp of it after a long time.

15:28It's easy to say, minimize data collection and use. Don't take data you don't need, right? That's easy to say. It's easy to understand. It's very hard to implement because that's what applications do. That's what companies want. They want the data. So, how do you correlate that? What level is good and what is not? And we broke this down to high-level, you know, not just general principles, but minimization by necessity. For example, each data element, is it necessary? And there's a very specific test of how to implement it when it's okay, and how to test whether that's compliant or not, right? ASCBS also has that. This is the goal that you need to implement at a very specific level. And then there's guidance as how to implement this in your system, how to design the feature for that. And you also want to be able to test that binary yes or no, it is compliant or not. And that's the logic that we went. Is this sufficient? Is it minimal? Is it the right level of precision? And then we get into a whole nother section around the granularity of specific data classes. For example, our requirement, was it 2.2.1? I don't remember this by heart. I have it open. Requirement 2.2.1. Location data granularity. Location data, simple. Don't collect more than you need, right? But what does that actually mean in practice? How do I implement that? When is it okay? So we give some criteria around that. What is the level that you should be taking? And there's increasing, right? Maybe you just need region. Maybe you want country. Maybe you need specific street address or GPS coordinate. So we're giving some, some abstract criteria that people can apply that reasoning to their system. Where am I supposed to fall and where not? And by the way, when you come as a QA or a pen tester, you can now review that and say, well, this is, this makes sense, or this doesn't.

17:29Chris RomeoAnd in the age of AI right now, this is the right moment for APVS to hit the scene. Because I don't know about how everybody else here is approaching requirements in AI, but I've fed ASVS into my agents and they have full visibility for anything that I'm building, full visibility into ASVS. And I go through steps of, let's check this spec or this feature against ASVS, pick out the requirements that are applicable here, attach them to the spec. And then those are exit criteria before we mark, before the part of the definition of done of a feature or a spec that's being built. And so APVS, this is the right moment to have this information because agents are writing all the code. Like we can say we don't like it or we do like it or whatever. That's just the reality of the world. Why not feed the agent a set of solid privacy requirements and start building what we've thought of as privacy by design, but really wasn't a reality. Now we've got a chance to make it a reality. So, how has AI, Kim, kind of impacted the work that you've been doing on APBS?

18:45Yeah, so we are trying to really focus on the foundation, and I think the same applies to security as well, by the way. I think there are specific AI-related privacy and security threats, but in a way, they kind of are an instance, an instantiation of the typical security and privacy things that we know that we're trying to do, that we're trying to build, or that we know that can go wrong. So, APVS, I need to look at Avi again because he has the thing open. I think we have some overlap or have some extension related to AI, but we really try to just focus on privacy for systems, for products in general. And indeed, one of the applications that we see for APVS is to include that into agents and whatever to build or to verify or those things. Maybe also you mentioned before that APVS is a new concept. That is because We're still working on it and it's not officially released just yet, US or San Francisco. And then we will be needing a lot of feedback from the community because, well, we can only take it this far. We have our ideas, but of course, there are so many security and privacy engineers, architects, developers out there that will be able to give us feedback and improve this. It has already been a very interesting balancing act between how to make it specific enough without being prescriptive, but without being too fluffy of minimize data. So that's also why I think I'm going to repeat Avi, but about the granularity, that is, I think it's going to be one of the most interesting sections for people to read because We try to have specific data types and not really tell you, you can only use this because as context anchorage also shows you, it's very context dependent, but give guidance on, okay, for data, for location granularity, you can have specific coordinates. You can have a street address and a number, a street, a town, a state, a country, and so on. So within that range, pinpoint what you specifically need for your scenario. So it's still not gonna be a very yes or no answer without context. You will need to bring the context into those decisions because, well, privacy cannot be fully generalized anyway.

21:31I'm also gonna jump in about, you know, specifically about feeding this into an AI. We took that into consideration. You know, we're fully aware of that and we're trying to write this in a way that would be useful for AI to feed an AI with it. On our roadmap is also to set up a bunch of prompts specifically based on this, based on the requirements to feed the requirements into the designated prompt. And that's going to be further down our roadmap eventually. But that's exactly the point, right? We wanna be able to feed this. into our AI, whether you're generating code or generating spec. Either way, you're coming from the AI, but you want to teach the AI how to do this in that way. We also have a specific section on data lifecycle.

22:16And Chris, I think I didn't actually answer your question. I think you asked, what is the impact of AI on privacy in general? I think it's big and it's amplifying the consequences, but I still feel, well, I kind of answered that. I still feel we need to focus first on the basics because if you don't minimize your data, then AI can abuse it. That's not an AI problem. That's the foundational problem. So, yeah.

22:45Chris RomeoAnd I like the way you described that where you said, we're going to, we're going to get the foundational pieces together and then AI can use those pieces. Like, I think that's the best overall strategy. Like, I don't think APVS needs to be tailored for AI. APVS needs to be the best possible set of privacy requirements that exist in the world, and then let AI do what it's good at, which is absorb it and then apply it to how it creates tests, how it writes code, how it does, does help assist in the design process. Like, that's the key is that the, that you have to have that. that core that's battle-tested through practitioners such as yourselves.

23:26So, I'm actually going to point something out. Very interesting. You said it should be the best possible set of requirements for privacy. And, you know, we kind of, we agree with that, obviously, but we're also realizing that the best possible set won't be so usable right away because this is a new concept. So, we are grudgingly making intentional compromises to make this more accessible, at least as a first version.

23:56Chris RomeoSo, multiple levels like ASVS has. So, you can scale, you can start and say, okay, I'm working towards level 1. We absorb that into our organizational culture, and we deal with the pain that it creates.

24:09Exactly.

24:10Chris RomeoWe heal from the pain, and then we look at level 2.

24:13And that level 1 is probably already going to be impactful for a bunch of organizations. So—

24:22And we want to make level 1 accessible. So, we're making those compromises, pushing things that absolutely should be level 1, but so many organizations don't do it. And unfortunately, we're making those compromises saying, right now, level 1 needs to be accessible 'Cause we want people to start doing this.

24:40Chris RomeoI mean, I think that's the best overall strategy, right? Like, you're not creating this standard to, or set of requirements to try to say or force everybody to do every possible right thing for privacy. I know both of you and how you, but I know how you work. Like, you wanna see the industry move. Like, you didn't do this because you wanna have a hammer and anybody that doesn't meet Level 3, their product gets smashed with the hammer into tiny little pieces. You know, like, it's gonna be a cultural move inside of organizations. This can be the driving set of requirements that opens up the roadmap for people to say, okay, I actually understand what a basic set of privacy requirements are now. We know that we can, once we tick the boxes on the pieces of level 1, then we can start to look to level 2. So, I think that's the only way to create something that people are actually going to use and it's going to provide value.

25:37Exactly. Because it, it, it's not a hammer. It's absolutely opt-in. And if people feel like it's a hammer, nobody's gonna wanna use that. So we're making that compromise, like you say, based on value. It's a valuable thing. We want people to use it and we'll help you get better. And not so secretly, in a few versions, we'll start making level 1 harder.

25:57Chris RomeoOoh, moving the goal line.

25:59Yeah.

26:03Chris RomeoThis episode is brought to you by Origin. Quick question, would you hire 1,000 people without knowing what they can access or what they're doing all day? Of course not. But that's how most companies roll out AI agents. Today, you get the prompt and the final answer. Everything in between is gone when the terminal closes. Origin finds the agents on your endpoints and rebuilds their work. Every command, every file, every handoff between agents. on one timeline. Origin, endpoint AI observability. See what they found in their own traces at originhq.com/appsec. That's originhq.com/appsec. Well, let's take a look at sort of the combination or how they relate to each other. A team plays the game, back to the game, on a Tuesday. And somebody says, I wouldn't be comfortable with that. But then on Wednesday morning, what do they do with that? And is there a real handoff to APBS, or are these 2 things that don't touch yet? And did building one change how you design the other?

27:16Yeah.

27:18So, I think the, the, the, the, the game is really the first step. I hope that the game triggers on— I forgot which day you said— on Wednesday. Okay, let's think about this on Thursday. And then hopefully on Thursday, they will go to the whiteboard and start doing some threat model. And it can be very extensive, but I'm already happy that they just do the basic checks, like, okay, what data do we have? Do we really need it? How is it going to be used? APVS is kind of the underlying thing when you actually start implementing it. Okay. What are the bits and pieces that need to be there when you, when you implement and that you can check later on as well? Avi, I saw you had a—

28:03Chris RomeoYeah.

28:04So APVS is basically how to implement the things that we want to do. It's how to implement a privacy-respecting system. Context Encringe helps you understand the why. Right? Simon Sinek, Start with Why. So there's no handoff, and they meet completely different needs. There's organizations where talking about privacy, anything beyond the absolute legal required minimum, it's practically taboo. There is no conversation of cringe. There's no conversation of, our users won't like this. Ha, screw 'em. Even better. You know, so content experience kind of pulls you into that world of trying to imagine how your users will feel, how those individuals will feel when their privacy gets violated, starting to recognize that cringe. Some places need to have that conversation. Some places you'll be able to get beyond that and start using it as a tool to recognize that cringe and how that context affects that cringe. We actually built into the game a facilitator's guide if you want to use this game as part of a workshop, not just make each other feel uncomfortable. And we designed that, you can use it in training, use it as a workshop, but there's no, there's no direct, okay, you know, level 1 is Context Synchrony and level 2 is APBS. No, it's not like that. It meets completely different needs.

29:27Chris RomeoYeah, that's helpful. Okay.

29:31Maybe highlighting that the APVS is part of the OWASP Privacy Project. So the goal is not just to create APVS, but to build more resources to actually support the technical needs from a privacy perspective. So there are some plans, but we first want to, you know, get this thing up and running. But because we feel that there is that need of, okay, let's do privacy and now what? Let's read 2 academic papers and Yeah. So we wanna, we wanna build some more resources there and, and yeah, create some awareness, but mostly give something tangible, something useful to the community.

30:09Chris RomeoAnd that's, we've seen that pattern play out in OWASP in like, we look at, look at the success of the GenAI project at this point. I mean, they've, they've, there's, I can't even tell you how many subprojects. There's a lot of subprojects that are happening underneath that banner now. As every, a lot of interested people went in that direction. And so hopefully that's what'll happen with APVS. It'll spur more people to enter the privacy space and work on these things with you. 'Cause there's a lot of privacy-wise people out there. They're just, it's not as clear that OWASP is the destination for them. Like, if you're in AppSec and you don't know OWASP, that's probably an interview question that causes the interview to end. If you're like, OWASP, what's that?

30:52Okay.

30:52Chris RomeoYeah. Thanks for the time. I appreciate it. We'll get back to you and we're not giving you the job. Bye. Click, hang up the phone. Kim, I want to just explore AI in the privacy threat modeling perspective, just because I have you here and I'm really curious to know your take. AI has been adjusting the security threat modeling, or, you know, Robert's done a lot of work in that front over the last number of months, but how does AI change privacy threat modeling and what are you seeing? results-wise from the models? Are they good/decent privacy threat modelers?

31:28I need to give it another spin because the thing, it evolves so quickly. So maybe what I'm saying now is not reflecting today, but only a month ago or so. But if you need to get an initial starting point and it's not your expertise or whatever, Having something is better than having nothing. So if you get something useful out of it, by all means. The things that I've seen is that it still lacks a lot of nuance. It still misses that broader context that we were talking about because it's not just understanding your architecture. You need to understand how it integrates with everything, what data exactly it is, what the type of individuals are that are related to it. What the expectations are of those individuals. Are you processing medical data because it's a hospital or are you processing medical data because it might be fun to build some social media app that is, I don't know, connecting you based on some kind of medical issue? It might do the same underneath from a technical perspective, but the privacy implications are completely different. So all of that needs to be captured, that and Also, I think the understanding of what privacy threats are compared to the understanding of what security threats are is still not as mature. So if we don't know it and if the body of knowledge is not in there, then the body of knowledge will also not be in the AI models. So I've seen some results. And so in Linden, we have non-repudiation as a a privacy threat while non-repudiation is a security property, right? So you have that difference. I've seen pretty much all results of AI-generated privacy threat models have non-repudiation as a property rather as a threat because there's way more information out there saying, ooh, we shouldn't have this while we're doing privacy. So, you know, you should have done the opposite. So it's those bits and pieces You're getting something, you're getting more than not doing anything, but the core still needs some work.

33:44Chris RomeoSo, it's likely a training data challenge because, to your point, the models have been trained extensively with security history, vulnerabilities, source code. Whereas, now that I think about it, if security has 99% of the split, you know, it's, if there's a split, it's probably 99 and 1.

34:07Oh yeah.

34:07Chris RomeoIt's probably less than that. It's probably 99.9 and 0.1, training data feeds a privacy story versus training data feeds a security story. So, it sounds like the model providers have more work to do before we could say privacy threat modeling is something that we're comfortable working through a model, having a developer work through a model and getting results that they can take action upon.

34:31I want to add something onto that beyond just the training data. I'm gonna wax philosophical for a moment, if I may. Privacy issues at the most of the time are not tech, just technical issues. They're intensely human personal issues. Now, that's not to say that you can't train the AI to recognize that, but the AI by definition is not going to have that very subjective personal view. How does it feel against that cringe, right? I mean, the AI can talk about cringe, but doesn't have that feeling of my privacy's being violated. Now, on the other hand, what it can be very useful at is giving you the perspectives that you yourself aren't thinking about. You know, I'm a straight white guy. I can't imagine what it'd be like to be a, I don't know, gay Black woman on a dating app in Malaysia, randomly throwing out there because we have cards around that. You can prompt your AI to come up with those, you know, edge case nuances and those personas that you yourself would not thinking of and give you that perspective that you can't otherwise. But you need to know to prompt it that way because it won't do that 99% of the time if it's not told to.

35:45Chris RomeoYeah, you're onto something there. This is a concept that came up in a previous conversation that we were having here on the podcast about the gut reaction is the thing that's missing from a model. The model can't measure risk like we can as humans from a gut reaction.

36:03Yeah.

36:03Chris RomeoI mean, it's the cringe. It is. It's cringe is a great word to use to describe it. Right. But the models can't do it. And I don't even think they can, they can't even fake it because that gut reaction that we get comes from a lifetime of learned experiences and everything we've ever studied. Like there is so much happening to result in us having a visceral reaction to something. That it's a human function and maybe AI will model it in the future and be able to give us a take on it. Hopefully it'll be probability-based. So it's not a cut and dry gut reaction. It's not always cut and dry for us as human beings. Sometimes it's like A and B are the possible, you know, I got a bad feeling about this, but there's more to it. There's a layered bad feeling about something.

36:56I think you can prompt your AI not to give you the answers, but to prompt you, to give you the thinking points that you wouldn't have thought of before, and that you can have that gut reaction to it.

37:09Chris RomeoYou just described 50% of my prompts, Avi. Ask me questions to refine this design. Ask me questions for clarity. Like, I do that literally all day long. I'm like, keep asking me questions. So yeah, that's a good point. Like, we can get closer to the answer through properly interacting with the AI.

37:28Yeah, and it feeds your brainstorm, but it still comes down to that human reaction.

37:33Chris RomeoIs there a particular privacy assumption that most often teams are thinking about, or they may be actually more confident about, but are coming up that you've seen?

37:44The major one to me is we don't have privacy concerns. We don't deal with personal data. You know, we don't collect Social Security numbers. We're all good, but almost everything that has some kind of association with a person is considered personal data. So even if I am using a weather forecast app, it's not the weather that is personal data, but the fact that I am connecting to it, if it's collecting, you know, usage data, it's not because it's just giving weather forecasts, it still has users. So there is personal data involved there already. Or opinions on things. Okay. That opinion belonged to an individual. If you're certain that that link can never be made, then okay, then maybe it's not personal data. But so many times, especially in this era of big data where it can be recombined and becomes reidentifiable, you have, you have not just potential personal data, but identifiable personal data with a huge potential to be linked and inferred and so on and so on. So. I think the first misconception is already like, yes, you do need to think about personal data and about privacy.

39:01Chris RomeoI'm going to get controversial here because I just had a thought. Our data is all out there at this stage. So are we building privacy for the future so that we can have a generation of kids that are born that truly are, do have a private experience because I can't make an argument that privacy is something that I can recapture at this stage because we were so bad at it for so long. So is there truth to this statement or am I just going in the wrong direction?

39:38Can I go? Could I go? I've been listening to the teacher often. I want to show off that I actually learned the lesson. Privacy is an evolving concept. They first started talking about privacy in the 1800s, and Kim, please do correct me when I mistake. We were talking about privacy in the 1800s, and if you are being captured by a journalist on camera, that's completely different from the way it was 50 years ago, which is different from the way it was 10 years ago, which is different from the way it's going to be 5 years ago. Privacy has to continue to evolve. Based on society's understanding and expectations of that. Technology does change that, but you still want to have that privacy.

40:20Chris RomeoBut can I regain my privacy though, at this stage as a, as a 52-year-old person?

40:25You could keep losing more of it.

40:27Yeah. It's, it's more like damage control. Yeah.

40:32Chris RomeoSo we're in a damage control phase from Kim, from your perspective. I just started thinking about like, Like, could my grandchildren who are very internet isolated at this point on purpose, could they live a life where they have true privacy because privacy's going to catch up here over the next 10 years? Or are they going to be in the same spot that we are now? And listen, I don't like where we are now. I'm just saying, I'm just, I'm a realist. I'm like, I know my data's out there in various places because it's been breached so many different times, but could we have a world in the future 10 years from now where my grandchildren have true privacy?

41:16I want to say yes, but I'm going to be a realist and say 10 years might be optimistic, but also it depends on what you mean with true privacy because as Avi said, it's an evolving concept. So I'm hoping that we in 10 years have reconfigured how our data can be used, what data can be used in models, what data can be used with AI systems, what can be, you know, stored and reused and all of that. Because I'm guilty too. I'm guilty too. I, at first I was like, I'm never going to use an LLM. Okay. I'm only going to use an LLM, but I'm going to not share any personal data. And then it's like, Maybe I'll just share a tiny bit because my background is important for this or this or that. And you just easily roll into it. And, and, and I think we all, we are all sharing that because of usability, because of efficiency, because of all of that. So I'm, I'm very much hoping that we will now kind of have reached the, the maximum of, of not use, but of abuse. And then we can, can come to a scenario where it's not just the 3 privacy activists, but the entire society is gonna be protected. I'm skeptical, but I'm gonna keep, you know, doing my best and, you know, baby steps, APVS, if we can have it rolled out and updated and integrated in all. future implementation development, you know, it's one step further.

42:57Chris RomeoWe're making progress towards the future.

42:59So about this, I actually disagree. I think it's going to get worse, a lot worse over the next few years before it starts getting better. I was going to say, can your grandkids be completely privacy isolated? Sure. Stay offline, don't live in a modern country, go find someplace in, I don't know, Africa backwoods or South America jungle or something like that. And in a few years, that won't be the option either.

43:25Yeah.

43:26Chris RomeoAt least there won't be any flock cameras there. Those are, those are, those are much, uh, they're, they're right. They've, they've raised the, the level of outrage across the United States.

43:38There's going to be automated drones with built-in AI and multiple cameras floating all over the jungle to Google Maps the entire Amazon jungle. Just as an example, right? But at some point, the need for systems to be privacy respecting, I think there will be pushback. And I think at some point, there will be a drive for society to regain the rights to privacy respect.

44:05Chris RomeoOkay, so let's get practical again. We got philosophical and Avi, you started it, but I finished it. I took us down a rat hole of philosophy and futuristic looking. Let's get practical to wrap up our conversation here. So, Avi, for you on APVS, I'll let Kim handle context and cringe. I've got the opportunity to do one thing this quarter with my team. Starting from the APVS perspective, what's that one thing that I can do to, to cause some positive change?

44:39So, much like with ASVS, you know, the ASVS leaders don't say, take 360 controls and implement them tomorrow. But rather filter down based on what you need. Use your threat model to highlight what is the most important to solve specific threats. And this is what we're trying to, to build that in as part of the plan. ASVS, yes, ideally most organizations should do most of level 3 of everything, but that's not realistic. Start with level 1, see which categories are the most important. Whether it's data minimization, which is the easiest to understand and the biggest gap between that principle of minimize data versus, and here are 15 different requirements to implement it. Each one is straightforward and understandable and implementable and testable. In the same way, we're going to do that across it. And if I can quickly mention, so we talked about the data mapping, we talked about data minimization. Data lifecycle management is the 3rd chapter. A bunch of requirements around storage, around collection, around a lot of different things. We have a chapter on transparency and notice, and we have another chapter which name pending, but might be the biggest drive because of GDPR. We're currently calling it data subject rights. I don't like that because it's very compliance-based naming, but that's the chapter that's probably going to have a lot of demand for how to develop these specific features that you need in order to not be violating GDPR and the California notice and all the other regulations.

46:09No, no, no.

46:10It's not just about compliance. I'm sorry.

46:12No, it's not.

46:13No, I agree. That's the drive. It's, it's really about giving transparency and intervenability. I hate the word, but it's what they have been calling it in the privacy community.

46:25And control.

46:26Giving, giving agency by lack of better words to the individuals. So making sure that you can decide what happens to your data, that you're informed about what happens to your data, that you can request updates, deletion, all of that. So it's, it's really not about, okay, let's be compliant with GDPR. It's really about let's give the individuals, which is not necessarily the users, but all individuals whose data the system is handling, give them rights for, to handle their data.

46:58Agree. I agree with that. And I think there will be a drive for that because of the compliance efforts, but this isn't saying be compliant ABC. It's just, you have to build out a whole bunch of features. The difference between secure features and security features, right? This is where you have to actually go and build out new workflows and new things. Yeah.

47:21Chris RomeoAnd Kim, you need to get Izhar to make you a privacy nerd t-shirt. That'd be so cool.

47:26He actually made us a context and cringe t-shirt for Vienna.

47:29Chris RomeoSo, he's, he's become the t-shirt maker, uh, on our other podcast, Security Table. He probably takes away one t-shirt idea per episode. So. And then often shows up the next time in that t-shirt.

47:41We should all make him a t-shirt that says t-shirt maker.

47:46Chris RomeoYou would, that's a good idea. He would love that. So, Kim, context and cringe, how do we put this into action? You know, I've got my team, we've got a little bit of time to do something here this next quarter.

48:00What do we do? Yeah, I think if you have your team building day or some, you know, lunch and learn session, or you have a workshop on privacy or on security threat modeling even. If you need an icebreaker, this thing, I mean, ideally more than 10 minutes, but one round would be 10, 15 minutes depending on how talkative people are. Take half an hour, play it with your team just to get a feeling. Ideally, you integrate it, but already it's to get that gut feeling to build in the, ooh, this is what privacy is about. We just had feedback from from somebody who bought a box, played it with the team, and went like, oh my God, this is great. So they bought a bunch more boxes. They're going to bring it to their citizen developer event later this year to inspire other people to also think about this privacy thing without making it a heavy threat model, but just to have that light bulb moment.

48:58Chris RomeoAnd such a small—

49:01you can play around in 10, 15 minutes. The problem is, Almost always they will ask to play more.

49:06Chris RomeoYeah, that's a good problem to have. We want to learn more about privacy. Okay, I guess this one time.

49:12And that the cool thing, the magic for me about this game is that it turns it— privacy, that's for lawyers, that's for compliance. It's a stupid, boring topic nobody cares about.

49:23Just go away.

49:24Just go do your thing, you stupid nerd. And this turns into, oh, actually, not only does it affect us, it's actually could be interesting. And there's stuff to talk about here.

49:33Yeah, we saw in the workshop in Vienna that there were some colleagues, and at some point, I saw them point, like, at one of their card combinations and go, like, this is what we do. So—

49:46Chris RomeoThat's what you want to have happen. You want to have that moment where they take an action for when they get back to work. Like, okay, we remember that thing we were talking about? Yeah, we do it. We got to fix it. So, just to wrap up, where can folks get this if they want to buy a copy of Context and Cringe? If they want an autographed Is that an option or?

50:05Of course. Come find us. For instance, at OWASP San Francisco, we actually handed out, well, not handed out, we gave some autographs in Vienna. You can find more information on the Context and Cringe website, just contextandcringe.com. It's available through CyberSec Games or, you know, follow us on LinkedIn or send us a message or so on.

50:28Chris RomeoSend us a message and Avi will put a bunch of games in a box and ship them to you. Out of his, out of the warehouse.

50:35We'll get CyberSec to do it, but yeah, basically.

50:38Chris RomeoWell, Kim and Avi, thank you so much for, for joining us to enlighten us about Context and Cringe. I'd certainly heard the buzz about it, but didn't know as much as I do now about it. Look forward to playing it in San Francisco. And then also thanks for the work you're doing on APBS. I think it's something that we need and, uh, I will be a consumer of it. Once it's a— once you— once that draft has hit style, I'm going to start working with the draft and feed it into my agent army to, to think about privacy and what we're building. So, once again, always a great opportunity for us to have you on the podcast, and thank you for the time.

51:15Thank you for the invite.

51:17Thanks a lot.

51:18Chris RomeoThanks for listening to the Application Security Podcast. If you enjoyed this episode, subscribe, leave us a review, and share it with a friend or colleague who would enjoy the conversation. We'll see you next time.