Brook S.E. Schoenfield -- Security in the Design and Architecture

0:07Chris RomeoThe Application Security Podcast. Here we go. Hey folks, this is Chris Romeo and welcome to the second season of the Application Security Podcast. Robert and I are back at it interviewing experts from across the world of application security and really breaking down the things that they're really good at and helping them to explain those things to you, our listeners. We do need to ask our audience for a favor. If you would, go into iTunes and provide a positive rating and comment for the Application Security Podcast. We're really trying to reach out and connect with many other people from across the industry, lots of people that aren't into security now, and that's one way we can do it by having a higher rating so that we'll appear in searches and things there. This first episode of season 2 is an interview Robert and I did at RSA with Brook Schonefeld. And this interview took place in the hallway, so it's a little noisy at times, but we talked about secure design, secure architecture, threat modeling, just a really deep conversation. I know I got a lot out of it being there live, and so I hope you get a lot out of it as well as you listen. Thanks. Folks, welcome to season 2 of the Application Security Podcast. We find ourselves at the RSA conference in San Francisco. We're in the hallway, of all places, so you're going to hear some noise and things coming back and forth. We're joined today by Brook Schoenfield. And Brook, would you just introduce yourself?

1:58Brook S.E. SchoenfieldSure. I'm a principal engineer at Intel Security, and my job is product security. I'm the technical lead for product security. So all technical things, if there's a problem, it winds up on my desk, and I'm the last point of escalation, technically, for something before it goes into management. I'm a principal engineer at Intel. Intel is akin to everybody else's distinguished engineer. So just to place that in context, it's a jury, you know, you go before your peers and meet the requirements. And I've been doing this for— I'm in my 5th year for them, although I've been in security for 19 years, I think.

2:41Chris RomeoSo how'd you get started then? So one of our big things is we like to ask people, you know, superheroes always have an origin story about where their powers came from.

2:48Brook S.E. SchoenfieldRight. Where my powers came from. Whatever powers I may have and whatever I've learned actually is dependent on a lot of great mentorship. But I will say that I was a lead designer at a little software house called Innosis. I had been their director of software development and then I did— went to an IC role or individual contributor and whatnot. And because I wrote TCP/IP stacks, when we got hooked up to the internet, we were paying a consultant quite a bit of money to come in and change our ACLs, our access control lists on the router.

3:19Chris RomeoRight.

3:19Brook S.E. Schoenfieldright, to the internet. And whenever we had a new protocol, because we did real-time communications, we had to have this person come in and do it. And they said, we don't want to pay this. You know about networks. Why don't you do the ACLs? And I said— it was some little Cisco router— and I said, yeah, I can learn that. And then I started seeing some hits in the logs and I went, I need to know more about what's going on here. That's weird.

3:45Chris RomeoRight.

3:46Brook S.E. SchoenfieldSo I got Shadow, which really takes us back into the late '90s. It came out of the Naval Intelligence folks and they gave it away as a bunch of Perl scripts, basically.

3:59Chris RomeoYeah, I remember that.

3:59Brook S.E. SchoenfieldYeah, Shadow. And I brought that up and went, oh my gosh, we are being attacked. Now, why would they be attacking us? We were thinking we had all these modems and point-to-point stuff that we did real-time communications. Why are they attacking our router? That's weird. And then So then I learned about Snort, which was like version 1.3 at that time, maybe 1.2. And so I brought that up and went, oh, that's a better tool. And I knew what a PCAP was because I did network debugging and wrote TCP/IP stacks. So I said, oh yeah, yeah. And slowly I descended into the basement of high tech, that is being a security person without actually making a decision or realizing it. I became the security person. Can you change the VPN on our router? please? Sure, no problem. And before long, I was doing the security management along with my day job as a chief designer and developer and technical lead. I also had to do this other piece of work. And really, I thought, I need to get some training because I'm out of my head. I can understand the— I mean, I could take a network packet apart every day long. That was not a problem. But I wasn't understanding the attacks well enough. I mean, like one time I called Akamai when they were just started and I went, you know, it looks like you're attacking me. And they said, no, no, no, we're not attacking you. They had made a mistake in their code and I finally got an engineer who went, oh my gosh, you're right. This DNS thing ain't right. We'll fix it. So, you know, that sort of thing was in those days.

5:33Chris RomeoSo that's how you got into the—

5:35Brook S.E. SchoenfieldAnd I just sort of descended into it and then dropped my card in a Cisco fishbowl at a SANS thing where I was— I did get to— I was an intrusion analyst, GIAC-certified intrusion analyst number 144, I believe. So, you know, I let that slide a long time ago, but nevertheless, I was a GIAC-certified intrusion analyst. And so then I was really into it and realized, oh my gosh, I really care about this stuff and, you know, dealing with real attacks.

6:03Chris RomeoYeah.

6:04Brook S.E. SchoenfieldYes. And Cisco said, hey, can you come over and help design some future IDS stuff? And I said, sure.

6:12Chris RomeoYou were off and running.

6:13Brook S.E. SchoenfieldAnd I was off and running. That was about 2000. So, all right.

6:15Chris RomeoSo I've heard that you talk about this design problem of application security. I know this is something that you're really super passionate about.

6:23Brook S.E. SchoenfieldI am.

6:23Chris RomeoCan you just introduce us to when you say the design problem of application security, what does that actually mean from kind of a layman's perspective?

6:31Brook S.E. SchoenfieldFrom a layman's perspective, it means that if you don't get the structures Correct. You actually haven't built the right protections. Let me use that word, defenses. And where those go is kind of tricky because when you're thinking like an attacker, you don't really care whether it's a design problem or, you know, you don't really care. If it gives you more access or is a good stepping stone, you use it, right? It's very utilitarian. And so even if you do your coding, that is the implementation, really correct, if you haven't designed the right security features, and when I say features, I mean like you have some data and the data needs to be protected from attackers and you want to build some defenses around that. If you don't build those defenses, and those are things you have to design in, like you don't just say encrypt this field. You have to build an encryption capability or you have to build— and then you have the problem of what you do with the crypto keying materials, the things that make the encryption. I mean, I don't want to get too deep in here, but you have to protect those too. And so that's a design problem. How do you protect them?

7:54Chris RomeoSo the problem is how you protect something that's buildable.

7:59Brook S.E. SchoenfieldLet me put it a different way. It's the security functions or the security security controls, we call them, or protections that you will build into your system, just to be really blunt about it. And it's good to think those things through before you've actually built something. It's a lot harder to do it later.

8:17Chris RomeoAnd so you consider yourself a security architect as well as a—

8:22Robert HurlbutI do.

8:22Brook S.E. SchoenfieldI have been for many years.

8:24Chris RomeoOkay. What does it mean to be a security architect? I think some of our listeners may not understand what that role actually entails. Sure.

8:35Brook S.E. SchoenfieldI'm actually writing a book about this, but nevertheless, I hope I can articulate this in a simple enough way. It's the person who understands architecture, that is the structures of things. What is architecture first? Let's step back and really define that, what we mean by architecture. Architecture is about structure. It's not about the details of the implementation. It's about the structures of things.

8:59Robert HurlbutOkay.

8:59Brook S.E. Schoenfieldand how you put the right structures together to build something. Whether that's a building or a system, set of systems, or a piece of software, it has to be structured well or you can't maintain it. It's just a mess.

9:14Robert HurlbutRight.

9:14Brook S.E. SchoenfieldIt's about structure and it's also about abstraction because what you're trying to do is take a view of some part of the structure and obscure the details and the other parts of the structure. So what I mean by that is if you're thinking about building an enterprise network, you don't really care the runtime of the stack of applications. What you're really thinking is which applications need, you know, what kinds of applications will need what kind of isolation and how will I build enough throughput and enough backplane to carry my traffic and problems like that. So what you— what the structure you want to look at needs to obscure the functions of things. On the other hand, if you're building an enterprise architecture, you don't care about the physical layout of the— it's just assume there's plumbing. And instead, you want to look at business processes and then the systems will come out of that, right? Whereas if you're in the middle, which is usually called the component or functional architecture, then you're looking at functions, but you still don't want to see the physical, like, layout of networks or whatnot. Security plays in all these spaces. However, your view is very different. What you're obscuring and what you're highlighting is different. So there's actually no one architectural view that works for security, by the way. So a good security architect ought to understand architecture. That's number one, both from, you know, from the software level right up to the big system level. If you're working in a big organization where you have enterprise architecture, you really— and I've learned a huge amount from enterprise architects who've taught me a great deal about how to think about these problems. But it's about that structure, and it's about hiding detail in favor of other things, an abstraction of what you need for this to solve certain problems. It's also our play area. you can mentally play with, well, what if we hook it up this way? What if we hook it up that way? What if we put it here? What if we put it there? And think, it allows you not to have to build anything. You think through these problems.

11:24Chris RomeoSo you're drawing, you're creating this architecture on paper and almost— so part of the security architect's role then has to be to consider what are the threats that exist in this thing.

11:35Brook S.E. SchoenfieldYes, exactly.

11:36Chris RomeoSo tell us a little bit about that process.

11:37Brook S.E. SchoenfieldAll right, so when you get to be a security architect, I would argue strongly that the intersection between what is considered standard information security, which is vulnerability and threat, you know, technical threat as opposed to threat agents, although that's important too, technical threat and how those things work, that's the computer science part of this. You have to understand how memory works in a runtime operating system, whether you're an interpreted language and how interpreters work, or you're natively compiled. All that computer science stuff is critically important. You can't even start to be a security architect till you actually have a good grounding, I believe, in solid computer science. It's not going to happen. Okay. Having done that and gotten that part of your knowledge base, what you're really concerned with are classes of attacks. Bits and bytes of each vulnerability and what opcodes you might need for that processor to exercise that memory issue are much less important than understanding the mechanism of a heap spray or a— and these are very technical terms and I'm not going to explain them— but, or a buffer overflow, a stack overflow, or an SQL injection, which is a radically different beast than what I just said, which are memory errors, you know, and you have to understand where those apply Like if you're writing in Java, you don't really have to worry about heap spraying or buffer stack overflows. You're protected. The programmer is protected from that. But if you're writing in C, then those things are radically important. And so you have to have that sort of understanding classes of attacks. Injection errors as opposed to environmental variable and things that have to do with shells is another class of attacks, which is sort of a system and many kind of thing, or scripting attacks, or browser attacks, as opposed to server-side attacks. These are all classes of attacks that you have to be familiar with. And what you do is you apply that against what is being built, whether it's at the enterprise level where you're thinking about, well, we'll need these kinds of network protections and this kind of monitoring, and you're thinking about big—

13:58Chris RomeoYeah.

13:59Brook S.E. Schoenfieldsecurity systems, and I wrote that in my last book, quite a bit about that, by the way, not to plug my book, but that's not the point. I have written a bunch about this. Or you're thinking about— you have to understand these different attacks of where they apply and where they don't apply, and that's the art of the security architect, to understand structure and be an architect, to understand computer science and put the attacks together with risk tolerance and system under analysis. Does that really capture it?

14:32Chris RomeoYeah, I think that really helps me to understand.

14:34Brook S.E. SchoenfieldAnd that is the guts of the art of security architecture. And a security architect who's only interested in attacks is, you know, whatever their title, then they're missing that structural part. And the structuralist who doesn't really— isn't really familiar with the computer science of attacks is missing an important part. I consider myself the juncture of those 2 things. My specialty when I come in, I mean, of course, I am an architect and I could just design systems and have, but my specialty is I come in and a system is being built or thought of, conceptualized, structured, that is architecture, and I can look and go very early on and go, You know, customers are going to need an authentication system here, and that's going to be a whole set of problems that we're going to need to deal with, and we don't have one, or we do have one, we should use the standard, or whatever the right answer is depending upon the context. And one of the things I always do when I get a job is I run around and find out what the risk tolerance is. So I have some finger to the wind to guess whether something's going to pass or not.

15:42Chris RomeoWho are you asking? How do you determine that?

15:44Brook S.E. Schoenfieldexecs.

15:45Robert HurlbutOkay.

15:46Brook S.E. SchoenfieldSo really, I go up, I go up the chain, and I also go— well, I've been hired as a director-level position now for many years, so I go across my own peer network and find out what the— what— how, how risk has been handled and what people are doing. I might— if there's a risk and compliance organization that does some kind of enterprise risk analysis every year, I want to see that, what execs think is the, the big kahuna. for us to deal with. I want to talk to architects. I also want to talk to people on the ground, you know, who are actually building things to get a sense of what they think, where their frustrations— you often find with the security people, they've been very frustrated because they've been raising a flag all the time and they've been getting shot down by execs who are going, yeah, my bonus isn't on that, it's on this, delivering the system, right?

16:33Chris RomeoYeah.

16:33Brook S.E. SchoenfieldSo you have to kind of ferret out all of that stuff. And my first 3 months on the job is just running around figuring it out. I mean, I may be giving threat models and whatever to deal with and help people with, but there's a lot of time I spend around wandering around.

16:47Chris RomeoWhat do you do with the— so once you determine what the risk tolerance is, I mean, I can't imagine that you're going to make different conclusions. You're going to still come in with your same— you're going to find the same problems and things.

16:59Brook S.E. SchoenfieldYes.

16:59Chris RomeoBut what do you do with that? How does your approach change based on when you learn those skills.

17:04Brook S.E. SchoenfieldI might be more lenient or I might be more tight depending upon what I find. Because the big problem here is playing like an attacker once you get to play that game is really fun. It's a lot of fun. And those of us in security, maybe we're weird, we like doing that. The hard problem is actually in a world of limited resources, limited time, what do you actually deal with? And so there's always some set of cutoff lines where, you know, against business drivers and deliveries and other things where people will say, I want, you know, I, you know, how bad is this?

17:48Robert HurlbutYeah.

17:48Brook S.E. SchoenfieldAnd what is it going to do to our business? And my job is to try and assess that honestly. One of the big mistakes I see security people do, and I imagine both of you have seen this, is security people walk around and they say, the sky's falling, the sky's falling, the sky's falling. And the third time you say it, you have no audience anymore.

18:05Chris RomeoYeah.

18:06Brook S.E. SchoenfieldSo, one of my superpowers, if you will, is to build the trust that I'm actually assessing for real by highlighting low-level risks and medium-level risks wherever I find them and bringing them to decision-makers. Over time, given 6 months, a year, people come to trust my analysis. So that day when I really say, I think this has a huge impact and here's why, I have already that trust built that I'm not playing sky is falling, that I actually mean it. And that's why one of the reasons I get listened to a lot more than my colleagues at some jobs I've been at, because I've built this trust that I'm just saying, this is a low-level risk. I just want you to know about what you're carrying. You know, we can deal with this in the next year. This is a medium risk. How soon can you get to it? You know, it's not terrible. And I think we have some grace time before people discover it. So, you know, and I play that whole game to build trust. It's just one of my techniques I've built over the years is that I find that if people come to trust your analysis, and then I've had a lot of training in risk.

19:16Robert HurlbutRight.

19:17Brook S.E. SchoenfieldI was very lucky. This is, you know, you might want to cut this out, but I was very lucky to be around when Jack Jones was developing Factor Analysis of Information Risk, FAIR, which is now an Open Group standard. I'm sure you both know about it. And it's great. It's fabulous stuff. And I learned so much from Jack. But he was developing it when we were both in this sharing forum that we both attended, and he would present it. And the first, like, 3 times he presented it, I was like, What is he on about? But then it kind of clicked in my brain, as these things do, when you've seen something enough and you've thought about it enough. And I realized, oh yeah, I've just improved my understanding of risk a whole heck of a lot here. The problem is something that has some really interesting math— it's casino math, by the way, and it's in the public domain. You can just go look at it. Kind of over my head, but nevertheless, it's got some interesting math and You know, it's calculatable, but it's a little slow to get all the factors correct. And I realized right away that my security architects needed something faster. So I kind of dumbed the whole thing down, and it came out as Just Good Enough Risk Rating. I'm not the sole author. Vinay Bansal, who would be somebody to talk to, by the way, brilliant guy from Cisco.

20:38Chris RomeoYep.

20:39Brook S.E. SchoenfieldAnd a very dear friend. and I did it together. We said, let's take FAIR and let's see how we can get this to a place where a security architect who might have to do this 20 times looking at an architecture, 20 different attack vectors, and winnow out the stuff that's not important and really focus on the stuff that's going to be killer, he needs something faster. So I kind of wired it to the kill chain, which is an inside term, but what it means is is how does the attacker, what are the things the attacker has to put together in order to actually make an impact, a real impact, right?

21:14Chris RomeoYeah.

21:15Brook S.E. SchoenfieldAnd we developed this sort of pseudo risk rating thing we call Just Good Enough Risk Rating, which SANS was gonna publish as part of their SMART guides that they were doing around 2010, 2011. Then they killed that program. So it's actually up on my website, brookshonefield.com, and people can go and download it. But we wrote it for SANS, me and Vinay. And so it's out there for people to grab, but it's kind of like risk dumb.

21:42Robert HurlbutNot—

21:42Brook S.E. SchoenfieldI mean, a lot of people have risk spreadsheets and stuff, but they make a lot of mistakes in that area. That's the hard problem. So once you get the design stuff, deciding which of the attacks you want and prioritizing those, that's the tricky hard problem because there are all these business drivers that are saying, I don't want to build any of this stuff because I have all these other priorities.

22:02Chris RomeoYeah.

22:03Brook S.E. SchoenfieldAnd then that gets to the problem of how do we elevate security to be one of the things that's considered right along with performance and scalability and maintainability and all these other illibidities that are part of developing great software.

22:24Robert HurlbutYeah.

22:25Brook S.E. SchoenfieldSecurity should be right up there. It's not prime. That's the problem. I think security people forget We're not the prime thing. We're one of business's objectives, assuming you're in business, unless you're the NSA and then it's prime, you know, but they have a different—

22:39Robert HurlbutDifferent charter.

22:40Brook S.E. SchoenfieldYeah, they have a different charter. For your average, you know, company and for your average startup, they have a lot more grace of time to get their security right and we should recognize that than say, well, I work for Intel Security, which will be spun off to become McAfee and in a few months again. And it's, you know, we can't deliver something out the door if it's full of holes.

23:05Chris RomeoWe have a different charter, I guess. Yeah, I've seen the same experience looking at startups. And one of the things I recommend to them is there are some small things that you can focus on. You're not going to do a whole lifecycle worth of things because they just need to get a product out the door so they can build some revenue and get things moving. So it's about simplifying. I'm a big fan of the keep it simple method. Yeah, it sounds like you are too.

23:27Brook S.E. SchoenfieldOh, absolutely. And focusing on what you can do and remembering that attackers will take a while to figure out some things. Figuring out, so that's another like place where it's a little hard to teach people how to think about the problems. And you just have to have enough experience with different kinds of attacks and how long it took them to be discovered. And then, you know, I haven't figured out a way to really show people or a system or anything like that, but there's a sense I have It's kind of like a feeling in my belly, if you will, which is really unfair to everyone. I can't transmit that, but it's a sense I have, probably based on experience in art, that some things can be easily discovered and those are pretty obvious, but some things are a lot harder to discover. It's going to take attackers, you know, you don't know if somebody's working on the research right now or they're going to wait 3 years before they come to it. You know they will discover it.

24:26Chris Romeoeventually.

24:26Brook S.E. SchoenfieldSo, take— here's a classic problem. Somebody wants to have some kind of beginning encryption key. This happens all the time. And so, they say, well, it's not really important because we're only using it, you know, to start off our conversation across the cloud. And so, we're going to put this in the code and we'll compile it in and it's now static, one key across the entire product, 5 billion people or whatever though. population is. And it looks really great. Well, the thing about that is, with reverse engineering tools today, that's ridiculously easy to find out. So you don't have much time. You get only a month or two before you're going to get discovered that. I just know that from experience, right? Whereas if you have this thing where you have a token and it's not really tied to a use case but rather tied to a class of users—

25:16Chris RomeoYeah.

25:17Brook S.E. Schoenfieldbut it's got some high entropy and it's, you know, it's going to take attackers a while to pull that apart and tease it apart.

25:24Chris RomeoYeah.

25:24Brook S.E. SchoenfieldAnd so, you have, let's say, 3, 6, 7 months. Why demand that it be fixed on version 1 when you know that by version 3, you can have this— you can have everything you need in place and you can improve it? And then, if they discover it in version 1, you can just say, well, use the new product, right? And that way you get some grace because you're not going to get everything done right away, right? And so you got to figure that piece out. Even though both those situations are radically important from a security perspective, there's really some very different timing related to them.

25:58Chris RomeoYeah. One of the other short-sighted things I see security people sometimes fall for is expecting everything to be done now. Like, it should be done now.

26:07Robert HurlbutYeah.

26:08Chris RomeoAnd they don't realize that there are— Resources required across even a startup. They can't necessarily take everybody and put them on security right now. And even in the enterprise as well, sometimes you have to look further out than just immediately and realize that there is a business running here. And that's one of the big lessons I've learned in the last 10 years or so is always step back and say, what is the business? What's the business things that they're doing right now?

26:33Brook S.E. SchoenfieldYes, absolutely.

26:35Chris RomeoBecause if we fight, if we get in their path, Then they have a problem with you. I love your description there because it's from the previous discussion about building trust because I see that's a soft skill from my perspective.

26:45Brook S.E. SchoenfieldOh, it is, but it's critically—

26:46Chris RomeoBut it's critical to this job.

26:48Brook S.E. SchoenfieldIn fact, when we hire security architects— interesting story. This is a true story. I'm not going to name names, but it's a true story. We hired this guy as a contractor. We needed help really, really well. I'm not going to say what organization because it gets too close. His paper and his presentation in the, you know, his resume and his presentation in the interviews was just fantastic. This guy clearly knew his security upside, downside, inside out. But he was a rule-based guy. And all the projects came in, but they couldn't deliver everything. And so they never left the person's queue.

27:28Robert HurlbutMm-hmm.

27:29Brook S.E. SchoenfieldAnd luckily, for whatever reasons, I'm not going to go into those, He, you know, before we had to do something about it with HR, he needed to do something else and left. And then we had to all sort of help these people climb in, you know, into getting their projects done. But, you know, if you really rule basically, this is the perfect thing and I'm going to drive to that, you will not be successful in this business.

27:53Chris RomeoYeah.

27:53Brook S.E. SchoenfieldYou really have to have business acumen. And that's something we used to— in fact, when we set up the security architecture roles and I was involved in that, the security architecture HR. You're probably familiar with those because you worked at Cisco too.

28:08Chris RomeoYeah.

28:09Brook S.E. SchoenfieldThat stuff, me and Vinay and Carolyn Thrasher and Richard Puckett all had a big hand in building that stuff out, all names from back in the day at Cisco. And we really tried to recognize that when you're starting out, you can pretty much take the business acumen of the business and just run with it. But as you grow, you have to have that business. We wrote that into our plan so that— Into the role growth. So that, you know, into the role growth so that people would realize that if they wanted to get to senior security architect or to whatever the next thing was at the next level, I forget what it was. the rules are, but, you know, there were 4 grades there or 3 grades there. They would have to be able to think in business terms a lot more and demonstrate that, really demonstrate that before they would get the promotion and could move up. I wanted to actually touch on Twitter because I think it's a really interesting case.

29:12Chris RomeoOkay.

29:14Brook S.E. SchoenfieldOf how this can play. listeners may not know that Twitter was a side thing they built while they were building something else, and I don't know what the other thing was that they were building.

29:28Chris RomeoIt's been long forgotten.

29:28Brook S.E. SchoenfieldYeah, but, you know, then they discovered, oh my gosh, this thing has legs. Let's do this other thing, Twitter, and they built this big thing. They did have authentication built in, simple authentication in the very beginning, right? So it wasn't that they weren't thinking about that from the start. I mean, I heard one of the founders talk about this. Now I know. So this is third-hand knowledge. Hopefully it's correct. I'd hate to lead anyone astray. But it was interesting because you may both remember that about 8 months after Twitter— I think I've got the timing scale right— 8, 9 months after Twitter went live and was achieving huge penetration and success, they had a series of terrible security incidents.

30:12Chris RomeoYep.

30:13Brook S.E. SchoenfieldAnd that must have been very hard from the inside when they sort of got their slap on the wrist from the attackers to go, oh, this isn't a side thing. This isn't just an authentication problem. This is actually something we're going to have to build in and think through very, very carefully considering our population. And, you know, that happens to startups sometimes. You know, you get this grace period.

30:40Chris RomeoYep.

30:40Brook S.E. SchoenfieldBut eventually, you're going to get it. You know, Facebook had their, you know, apps that people built that then were actually malicious and would run through, you know, through and generate ad clicks or whatever the scam was. And they had to put a stop to that. You know, think about it. You only need an email account for Twitter or for Facebook. So they have every attacker in the world on their systems and they know that. They have to know that. They wouldn't be doing their jobs if they didn't. And so there's an interesting design problem for you when you know your population includes every malefactor on the planet.

31:18Chris RomeoAnd you have to let them in.

31:19Brook S.E. SchoenfieldAnd you have to let them in.

31:20Chris RomeoBecause you can't tell, you can't find them.

31:22Brook S.E. SchoenfieldRight, and that's very different. So sometimes you find, you know, in design discussions, just coming back to the design problem, you'll find teams will say, Well, we're authenticated, as though that suddenly strips off, that reduces the— you have to understand as a security architect, that reduces, might reduce the attack surface, but it might not depending upon your population and how easy it is to get authenticated. Every online store in the world must realize that they have some attackers on their, you know, that have profiles because all you need is a credit card and an email address, and you can buy credit cards for 25 cents apiece on the black market. So when one shows up bad, you just put in another one, right? And you just keep turning it. And so you have to do this behavioral stuff, and I'm sure they do that on Facebook. They watch behavioral stuff in order to kill accounts, or, you know, and then they get more sophisticated about who's bad and who's not and how much you can get at and how much you can't get at just because of your account. So, you know, Authentication, for instance, as a security control, it might buy you a lot at the NSA because you pretty much have vetted every authentic— authenticatee, but it might buy you zip except be able to identify the account and kill it in Facebook or Twitter terms, right?

32:44Chris RomeoYeah.

32:44Brook S.E. SchoenfieldAnd so there's some very interesting things that each control needs to be looked at. That's a design problem. That's a design problem right there if we want to get to the heart of it is to understand what does the authentication bring you and why are you using it and what other controls you might need. But to count on any single control in today's world is probably generally a mistake.

33:07Chris RomeoYeah.

33:07Robert HurlbutYeah.

33:08Chris RomeoSo, Robert, you spend a lot of your time working with developers. What do you see as the ways that developers and architects such as Brook can work together successfully? Have you ever seen any Any examples or things that have not worked out well when developers and architects work together?

33:26Robert HurlbutWell, I think there's the— and I know Brook can talk about this some as well— that is that idea of 2 camps and having opposing viewpoints and directions. But really, how can we talk together? How can we work together? So try to not make it so separate. And I think that's the one thing I see more than anything else. How can we get them to work together as opposed to go off on their separate ways and then somehow or another finally meet in the middle? But can we get them working together earlier and all throughout the project?

34:01Chris RomeoYeah. So have you seen, like, do architects usually work in like a different hierarchy than the people that are developing, writing the code from your perspective? And then Brook, I'm curious about your answer on this too.

34:12Robert HurlbutWell, it's an interesting question because I remember recently talking with some architects this very thing, and we all said among ourselves is that we had experience where we were developers first and now architects. And so we didn't want to have that kind of, we're away from all of what's going on.

34:31Chris RomeoOkay.

34:32Robert HurlbutAnd so I think if that's how it is, is that you have somebody who's just completely separate from whatever's going on and then comes in and says, okay, now do this, like a rigid rule-based then it's not going to be effective. And so I think that's pretty key, is that they're still understanding where the issues are, talking to people, understanding, and getting in there with developers and managers and so forth and understanding.

35:01Brook S.E. SchoenfieldNobody really reads those 250-page architecture documents that are sitting up on the shelf in the binder. Nobody, right? So, I'm adamantly opposed to ivory tower architecture. Absolutely adamantly opposed. I believe deeply— I mean, I think that's one of the places where great Agile, when it really works well, or great Scrum, let's say, to pick one of the Agile processes, really can tear down those pieces because, A, people are making coding experiences all the time.

35:35Robert HurlbutRight.

35:36Brook S.E. Schoenfieldwhich then should influence the architecture, which then should influence the security. So we talk about iterative security. We don't talk about— it's not a point in time. You know, the old way or one way, and I think it doesn't work, especially in an agile environment, security races in. They're way too over-resourced. There's only 10 of them against 10,000 or whatever the numbers are. And they race in, they drop their threat model and their requirements in on everybody and they disappear and then they're really surprised at go-live or the governance check, whatever that is before go-live, that half of their stuff didn't get built. They don't understand the response from the agile teams that say, well, things changed and we couldn't change the requirements, so we just didn't build it because it didn't make sense anymore.

36:26Chris RomeoYeah.

36:26Brook S.E. SchoenfieldInstead, what we do is this iterative work where we say, There has to be, if you need a security expert, they are available to you. Even if they're not on your team, which is the best, where there's just absolutely someone dedicated, not dedicated, but holds the security for that Scrum team and is right there in the daily standup pulling stuff off, grooming the backlog, pulling stories off, can think through the security that can be right available. My friend Owen Carroll, who works in Cork, Ireland, brilliant guy and I love his work. He started putting up the threat models. He has beautiful threat models in the stand-up room. So, as people worked and as they talked about it, they could just look at the threat model and go, oh, this is going to have security implementation, you know, implications.

37:17Chris RomeoYeah.

37:17Brook S.E. SchoenfieldOr this one doesn't. We'll just go off and do whatever we want. This one, okay, so let's enter into this You know, I'm looking— the audience can't see, but I'm looking at the wall as though there's a threat model there. But, you know, this was a very powerful shift in his Scrum teams, and it was entirely his idea. I didn't think of it. We were working together on some threat modeling training. When he told me about that, it shifted my brain, and I went, oh my gosh, right, more people involved, not less people involved. Before, I used to think of myself as the parachuter, and I parachute in, and yeah, I'm available, and you guys I'd already gotten into, been agile. I trained and been through the agile transformation. So I was already agilized, to coin a word, you know, sort of my mind had been bent badly, or it is badly bent anyway. But, you know, I'd already taken, drunk the Kool-Aid of agile long since. But nevertheless, I went, that's the key we need, is to make it part of the iterative process.

38:16Chris RomeoRight.

38:17Brook S.E. SchoenfieldAnd let— the opportunity there is your security gets better along with everything else as people learn what they're building. I mean, imagine, as developers, I'll just ask you both. As developers, when you're trying to do something new and trying to come up with a decent, elegant way to express something as an algorithm, have you ever made a mistake? I know neither of you did.

38:41Chris RomeoI don't think Robert did.

38:42Brook S.E. SchoenfieldYeah, Robert, because his code is always flawless.

38:45Chris Romeoalways flawed.

38:46Brook S.E. SchoenfieldRight, exactly. And then you kind of learn from that and also I got better at being more elegant and less like spaghetti and whatnot over the years. I got a lot more sophisticated about thinking through things as I got more algorithms under my belt rather than just coding my way into stuff. But nevertheless, you try, you work with stuff. Well, why not let security have that same iteration? You try a design and you get down a rat hole and you say, nah, this is terrible and it's way too hard and it's going to be awful. Let me back up and reconsider the problem. If we let security iterate, you actually— what we find is you get much better solutions at the end than you thought you could figure from the threat model when you started, when you wrote the requirements. So we let it iterate and then we try to have a security person, we have a 100-person team, that, you know, across our 3,000 developers or so, we have a 100-person team who are assigned to each product plus a lead for each, you know, product group or BU as we call them, where, you know, who's, you know, my colleague essentially, who is a lead security architect and a proven security architect who can then be there for the product people if they're new or if they have questions. And then we build our governance that way too. So it's very fast and very nimble. We don't waste time with boards where everybody has to go through the board.

40:15Chris RomeoYeah.

40:16Brook S.E. SchoenfieldAnd that's in my presentation tomorrow morning. I have my little cartoon about boards. Not that I'm against boards entirely, but they have their place, but not every day in an Agile environment. Instead, what we do is we say, you got to get peer review, and one of those people has to be senior to you, and one of those people has to be outside your product group. who can look at the problem dispassionately, just like reviewing code, right? And get peer review. And when the 3 of you come to consensus, that's the agreement. That's the governance.

40:44Chris RomeoThat's more than a board. Yeah.

40:46Brook S.E. SchoenfieldAnd if 3 of you can't agree, then pull in someone else, right? And eventually, if no one can agree, it winds up on my desk and I try to get them to agree and we figure out what to do, right? Or we go up to management or escalate or whatever needs to happen.

40:59Chris RomeoAnd that's— I think you're might be the most resourced team, which I think is a great model. Most of the places that I interact with are never at like a 30-to-1 ratio.

41:11Brook S.E. SchoenfieldNo, but of course, that's because we built it. None of these people work for me.

41:15Chris RomeoOh, these are security champion-style people that are in the business.

41:18Brook S.E. SchoenfieldBut they're not security champions in the classic sense. This is something that I learned at Cisco in 2005 and 2006. that if we wanted people to actually do the security role, we had to empower them as a virtual team of InfoSec. And they were the policy drivers, and they were the technical people, and they were the real architects, and we would support them and teach them how to do that. But they were not just red flag throwers. I think we have a security problem. I'll get somebody with some real power. But we would really give them power. That was— John Stewart was totally behind that.

41:50Robert HurlbutYeah.

41:50Brook S.E. SchoenfieldHe gave me every bit of head to make every mistake possible. possible in the world to run that program. I did it with this Ferris Jabri, is the guy's name. He was my PM. I'll never forget when Enterprise Architecture said to me, no, we're not ready for you to do this. We walked out of that meeting and I was so crestfallen. Ferris just looked at me and said, Brook, we're doing it. We need to do it and we're going to do it. We're just going to do it our way and then we'll learn. Yeah, a year later they were coming to us and said, how did you do that? That was pretty funny. But nevertheless, now I've done it 4 times in my career. I'm on the 4th time and I know this works. But you really empower people. You don't make them just eyes and ears. We do have those. We have about 25 of those.

42:33Robert HurlbutYeah.

42:33Brook S.E. SchoenfieldBut we have 75 people or 80 people who are actually our security architects. They are empowered to drive policy and to say what is right and what is wrong. Of course, they vary in skill and whatnot and we support them and we have an escalation path that gets them out of the way with their dev manager and all that stuff. That's more complex stuff I don't want to talk about now, but nevertheless, organizational stuff. Nevertheless, they really are— we have them from senior management 25% resourced.

43:03Chris RomeoOkay.

43:03Brook S.E. SchoenfieldAnd our leads are 50% to 100% resourced to security. And so, you know, if their manager starts saying, why are you doing that and not this? James, my boss, goes and has a little chat about, you agreed to give 25% of this person's time to us. That doesn't mean on their spare time on the weekends. That means this is part of their job. You know, we'd say it nicer than that.

43:28Chris RomeoYeah, of course. And it sounds like, I mean, that's a great approach. I've seen this done a number of times, and to have that level of empowerment and dedication is going to make everybody a lot happier. So I know we're almost out of time here, but I got—

43:40Brook S.E. SchoenfieldWell, let me just add. There's a con to this. There's a really big con. First off, you have to be willing to let go of control.

43:46Chris RomeoWhich is scary.

43:48Brook S.E. SchoenfieldWhich is really scary for security people and really trust and mistakes will be made. It's part of the whole deal and so you get to try and map that and how bad can a mistake be. But also, you lose the center. When we first started this, the first time we did this, we had the most wonderful archive of every single project that had had security review and we had data on exactly what they did. Within a year, that was so out of date and people hadn't filed their artifacts and we didn't know about everything and it was a real hard lesson that I had to think of others or we, me and Ferris and the members of the web architecture team, WebArch affectionately called.

44:33Robert HurlbutYeah.

44:34Brook S.E. Schoenfieldhad to think through how we were going to get around that problem because we certainly weren't going to beat up our volunteers, not exactly volunteers, but volunteers. I always treat them like volunteers, you know, treat them very politely and thank them for their work for us because they obviously weren't going to be filing all the documents anytime soon. So we had to go out where they had their documents instead.

44:55Chris RomeoGo to their—

44:57Brook S.E. SchoenfieldYeah, go to there and make it part of— well, that's another thing I've learned. You got to make this all a part of the natural flow of the development. If you don't, it will only get to be another thing that gets forgotten when push comes to shove. Yep.

45:13Chris RomeoI'm with you there. I mean, it's about lightweight. How do we make it lightweight?

45:16Brook S.E. SchoenfieldSuper lightweight.

45:17Chris RomeoPart of the existing process.

45:18Brook S.E. SchoenfieldAnd natural.

45:19Chris RomeoNatural, because then they can't— it's going to be hard for them to say, that's just too hard, because you're really not asking them to do anything other than go through their standard process and do the things they do. Right.

45:27Brook S.E. SchoenfieldYou get stories, security stories built onto the backlog. And they're just part of that and they have a priority and they get pulled and they go through the process and that process generates the artifacts you need for whatever auditing you need in your situation. All of that should be the natural flow, like if they're using one of the, you know, many agile tools, for instance, since we're agile, you know, one of those tools, naturally the security stuff needs to just fall right through that. Or if they're just using a whiteboard, give them templates and say, here's the stuff that's got to be done. And here's the reasons to do them and make it simple and easy. And of course there's someone to talk about the hard problems, but make the simple stuff simple. Really dead simple.

46:08Chris RomeoYeah, that's good.

46:10Brook S.E. SchoenfieldAnd easy and natural and stuff you would think to do anyway if you knew how to do it.

46:13Chris RomeoYeah, definitely. So I got one more. We're going to call this a lightning round type of question because—

46:19Brook S.E. SchoenfieldWell, I'm sorry, I'm wordy.

46:21Chris RomeoNo, no, you can use as much time as you want. I just like to call it lightning round.

46:25Robert HurlbutSounds cool.

46:26Chris RomeoBut I'm thinking about, so somebody who says, I'm thinking I want to be a security architect. I want to get where Brook is now. And they're just, say, just brand new in their career. They're a few years in or whatever. What would you recommend to them as a path? What should they do to try and get to be a successful security architect in 5 years from now?

46:49Brook S.E. SchoenfieldYou know, every once in a while, probably a couple times a year, this is a little joke, lightning round, but I'll make it quick. Someone does say, Brook, I want to become you, and really, I'm too crazy. You don't want to be me. I am so crazy and so impossible. You really want to be you. But there are things for mere mortals, and I call myself a mere mortal, and that I want to point out. I have seen one or two people who took to architecture like ducks in water. and who were very, very young, very, very relatively inexperienced, and who just seemed to have a preternatural disposition to think structurally about things and then be able to work down through the details. There are those. There are the Olympians, let's call them. The Olympic racing type security architecture. If you're one of those, bless you, go forth and architect. But for those of us who are mere mortals, there are some very particular things you can do. One, get your basic computer science down together. So if you're working in an interpreted language and you think that third-generation languages are dead, disabuse yourself of that and find out that every interpreter you're working with is written in a third-generation language. And that ultimately that comes down to opcodes. which are read by a CPU which can only add and subtract, and subtraction is a form of addition, and multiplication and division are both forms of addition and logic. And that's what CPUs do, they move stuff around, right? Very sophisticated today, but that's the guts of it. Get your computer science together because you will need that in order to understand attacks, number one. 2, assuming you've got that in your head. I find that learning to work with structures, and I was very lucky in my career because I just happened to go into this startup in my sort of second job, if you will, in high tech after writing a bunch of dBASE stuff with these brilliant designers. There you go. And I swear to you, I sat in the design meetings when I was learning to be a programmer and whatnot, and I had no idea what they were talking about for 6 weeks.

49:11Robert HurlbutOh no.

49:12Brook S.E. SchoenfieldAnd finally I asked a question and John Carone looked at me and said, Brook, good question. You're learning, right? And so, you know, become part of design meetings even if you don't understand what the heck they're talking about. It'll kind of, you know, work into your brain and you'll begin to develop the right pathways. Read other people's code. There's nothing better. Become a good coder. Really, the best security architects, especially on the product security side, come out of software. I hate to say it. If you're coming from some other place, do some coding. But for coders, get that, but then start structuring software. Work with your software architects because the thing is the ideas of abstraction and structure are the same, even though you're applying it sometimes in a much broader perspective. Learning to do, you know, to think abstractly about the structures— how do I want to structure this code— as opposed to just write my, you know, that will teach you about structure and data hiding and some of those things. And then what we say to people is, once you get there, The best thing that can happen to people is to work in some other technical field where you also get some technique or go deep. Because there's a funny thing. When people are deep in just one area, it's hard to see the patterns. But when people, most of us mere mortals again, suddenly become deep in a second area, Like constructing software and writing it, and then you go and you're working on networks and trying to understand how networks work, or real-time communications, or big system integrations, anything. It doesn't matter. Just become deep in a second thing because what happens to the brain is it begins to see the similar patterns on both sides, and then you develop your ability of thinking patterns, and then you're on your way to being an architect.

51:19Chris RomeoYeah.

51:20Brook S.E. SchoenfieldNow, in order to do security, you also have to explore attacks. So somewhere in there, you have to explore defenses and attacks because those are our alphabet, if you will, the building blocks of what we do. We have to understand attacks that are relevant for a particular type of system, and we have to understand defenses. And you don't put defenses at the same place usually where you get the attack. You often put them at a higher level or a different level. I like to say you decompose an architecture into its attackable— attack surfaces, standard. Everyone talks about that in all the books and everything. But then I like to say, and then you have to also decompose it into defensible boundaries, because where you put stuff is maybe a different structural problem than where you attack it, right? And those are not the same thing. Sometimes they are, sometimes they're not. Learning to think about those building blocks and think about that in terms of an architecture will then turn you into a security architect.

52:20Chris RomeoCool.

52:20Brook S.E. SchoenfieldThat's kind of like the last step. The other thing that I like to do is do vendor assessments. They are the most god-awful horrible thing in the world. You do the first 3, they're really interesting. The next 10 are like work, and after that it's pure torture. pure factory work and frustration. However, there's something about doing vendor assessments for security that gives you that breadth in security that allows you to understand information security in a really broad way. There's nothing like doing vendor assessments. There really is nothing like doing vendor assessments. And I hate to wish that on any nice person, but really there's nothing like doing vendor assessments to build your strength.

53:03Chris RomeoYeah.

53:04Brook S.E. Schoenfieldacross the entire field of— because you have to examine their SDL, you have to examine their policy set, their, you know, the way they set up InfoSec, you know, all of those things. And you sort of get the— by doing that, you get experience in thinking about the entire set of information security rather than your little focused area. Vendor assessments, awful job.

53:28Chris RomeoGreat experience.

53:29Brook S.E. SchoenfieldGreat training.

53:30Chris RomeoGreat training.

53:31Brook S.E. SchoenfieldYeah, and as soon as you've done 5, And you got it. Find another job.

53:39Chris RomeoWell, Brook, thank you so much for taking the time today and for taking us through on this journey.

53:43Brook S.E. SchoenfieldOh my God, I'm supposed to be at my—

53:45Chris Romeoand you're late, so talk to you later.

53:47Robert HurlbutThanks.

53:48Brook S.E. SchoenfieldThanks for listening to the Application Security Podcast. Our intro music is 8-Bit Kung Fu by Boring and TJ, and the outro is Southern Delight by Stefan Kartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.