Chase Schultz -- AppSec and Hardware

0:00Chris RomeoHey folks, welcome to Season 3, Episode 16 of the Application Security Podcast. On this week's episode, Chris speaks with Chase Schultz about application security, product security, and hardware. They discuss how these move together, and Chase explains how Meltdown and Spectre actually worked and all the security ramifications of hardware.

0:17Hope you enjoy.

0:18Chase SchultzThe Application Security Podcast. Here we go. Hey folks, welcome to this episode of the Application Security Podcast. On this episode, I'm flying solo again, and— This is Chris, by the way. And I am joined by Chase Schultz, who is going to help us explore the world of application security and how it relates to hardware, which is kind of a different type of topic. But with all the things we've seen in Meltdown and Spectre, and there's new potential vulnerabilities on the AMD processor side, I thought it was time to kind of explore this issue of application security and hardware. how these 2 things fit together. So Chase, we always start out with what is your security origin story? All good superheroes have an origin story. It's in comic book episode number 1 of whatever your comic book is called. How'd you get into security? What's your security origin story?

1:43Chris RomeoYeah, thanks, Chris. I guess my security origin story starts way back in high school. We used to play Half-Life with the network administrator/computer teacher, and he hosted the binaries and everything to play Half-Life on the teacher's drive. And he would log us in in order to allow us to, you know, play with him over the network and access the binaries. However, I thought it kind of unfair that only his account, you know, was able to access So I ended up installing a keylogger on one of the file print share machines and waited for another administrator to come by and drop their password in there. And so that was kind of where I got started. And it's been a fruitful endeavor because I've ended up now working as a penetration tester for several years. And right now I'm currently working at Autodesk as an application security engineer doing penetration testing and live red team exercises.

3:11Chase SchultzOkay, now I gotta ask, did you, uh, did you get in trouble for planting that keylogger? Did they track you down and come after you?

3:20Chris RomeoYeah, so, uh, you know, this is, this is back in the day before There was stringent computer crime laws and they were really looking to prosecute. Basically what ended up happening, they found out later that I had added my own account to the teacher's share. When they found this out, they kind of freaked out and they went back and they checked my grades and then double-checked my grades and they saw clearly that I hadn't changed my grades because I wasn't doing so great. Then they came to me and they said, well, we have a choice. Either you can show us how you did this and we ban you from the computers for a month, or you can not tell us how you did this and we're going to expel you. So I opted for the former and ended up working with the security— or, well, working with the administrator to, you know, kind of explain what I had done. And oddly enough, he didn't change his password after the fact. So that's important.

4:32Chase SchultzSo you may in fact be the person who the XKCD cartoon is actually speaking about, Little Bobby Tables. I don't know if you've ever seen that cartoon, but that might have been you, actually. Maybe that was— you were the source of that. So I'm going to throw a new question at you that I've that I just thought of now, and I'm going to ask to every guest into the future. So, you get the honor of being the first person to answer this. But since we talk about security origin stories and that has such a superhero kind of a breakdown, if your career was represented by a comic book, what would be the name of that comic book?

5:10Chris RomeoI don't know. What would be my comic book? I don't know. I mean, I love Deadpool so much. I could only hope that, but nothing, nothing that, nothing even remotely close to his origin story. I don't know.

5:31Chase SchultzMaybe you can be the, uh, the Deadpool of the penetration testing world.

5:35Chris RomeoYeah, the anti-hero.

5:38Chase SchultzAlways, always dropping jokes and all kinds of fun stuff there too. All right, so let's, let's jump into the world of, um, kind of hardware hacking. And so, I'm curious as to your perspective here on, when I think about all the different hardware-related problems that have occurred even historically, and I think about my experiences in the world of application security and software security and ultimately product security, you know, there's some things that I've seen done in the improvement of say product security from a hardware perspective. But what do you see as the intersection here? Is there, Is there a connection between application security and the things that we do for software and what we should be doing for hardware?

6:22Chris RomeoYeah, I mean, I definitely think there is. There has been significant effort and investment in recent years to look at AppSec harder and to understand these challenges and problems that we're facing. With all of these, you know, new IoT devices and various different, you know, hardware that is all running around out there, we're having to kind of address some of the same AppSec problems that we saw, you know, in the past, in the '90s, 2000s. You know, it's all a new game for some of these systems here.

7:09Yeah.

7:11Chris RomeoWe're also seeing that there are novel ways to do very, very interesting things in software that affect very specific hardware architectures and microarchitectures.

7:32Chase SchultzAre you talking from the attack perspective when you say novel ways or interesting things? Are you thinking like there are certain types of ways you can scope attacks that will ultimately impact the hardware? Is that the direction you're going?

7:46Chris RomeoYeah, yeah, definitely. I mean, the Spectre and Meltdown were incredibly interesting from that perspective because I had never really seen anything that affected the microarchitecture state and the the ability to affect the caches inside of the processor itself. And so this research was kind of a new experience to me. I mean, they— the way that they approached this and looked at it to, you know, kind of abusing what we would think of as, you know, like processor optimizations and that sort of thing. And, you know, things that we wouldn't think would be normally interesting and would have a security impact indeed do, and, you know, have the ability to be fairly devastating in that effect.

8:48Chase SchultzYeah, so, I mean, just to I mean, is there secure coding from a hardware perspective? I mean, I'm, you know, I've spent the majority of my career and my time focused in on the world of software. And so that's why I'm almost thinking out loud here, but is there a secure— is there a need for secure coding in the world of hardware? And is it even possible to do secure coding in the world of hardware?

9:16Chris RomeoYeah, I mean, I definitely think that there is a need specifically, you know, where it meets in the intersection of these places. You know, we're on a lot of IoT systems or, you know, places where we're trying to run trusted code, you know, bootloaders, that type of thing. You know, vulnerabilities that are— that we end up finding in bootloaders and areas closer to the hardware is definitely security critical because it basically allows an attacker to either bypass, you know, boot restrictions or boot into single user mode and drop to a root shell, those types of things. So I think it is definitely important. As to your question about, you know, is there a way to kind of code securely in hardware, you know, FPGAs and people that have been working in that space, you know, there wasn't trainings a couple years ago that I remember, but I definitely saw that there were trainings at TwerkCon this year focusing on FPGA programming and some of that, you know, getting closer and closer to the ASIC level. And then, you know, now we have folks doing research out there The Meltdown and Spectre folks, and then the CTS Labs more recently with the AMD flaws. Yeah, I think it's definitely interesting stuff. In reading about the AMD flaws stuff, it sounds like they outsourced that creation of some of those modules and the coprocessor and the processor and that sort of thing to third-party companies.

11:15Chase SchultzMm-hmm.

11:19Chris RomeoIt's one of those things like now we're going to have to be looking at the supply chain security for these coprocessors and that sort of thing and addressing security like we do with AppSec every time we either bring a company in or acquisitions, that sort of thing, third-party code review, those same types of things. of assessment probably definitely need to be applied to this space as well. Yeah.

11:49Chase SchultzAnd so, you know, there's a couple different threads I took away from the comments you just made here. So when we're talking about kind of the heart, the pure hardware side of FPGAs and ASICs and all of the things that are happening there kind of at the very low levels of the hardware, so it sounds like there are some movements forward into the secure coding space and that. But I wanna stop and talk for a second about, you know, you mentioned the bootloader, you know, on an IoT device and the fact that that has to be trusted code and the criticality of security there is so important. So like a bootloader is normally written in C though, right? Or a bootloader's written in something else. They don't write bootloaders in assembly, do they?

12:35Chris RomeoNo, no. I mean, most of the bootloaders out there are written in C. I mean, the most common one for embedded systems is probably U-Boot, that type of thing.

12:45Chase SchultzOkay, so, so you're— so you're with your bootloader, you're suffering with, or working through the same secure coding challenges that anybody else that's writing software in C, it's the same type of problems. It's not like there's anything— the bootloader really isn't introducing any new types of programming constructs or something that would be different than straight C that's being written into a firewall program or a firewall product or something.

13:15Chris RomeoRight. Yeah, yeah, that's correct. That's correct.

13:18Chase SchultzOkay.

13:19Chris RomeoBut the, you know, these— the bootloader in a way is kind of responsible for the whole loading of a firmware image and, you know, whether or not that's signed, doing those checks. you know, the various functionality for flashing the device or, you know, some of those types of problems and access to how the kernel parameters are loaded as the device boots. I gave a talk at DerbyCon a couple years ago where I showed the Insteon Home Hub which was one of the first Apple kind of smart home things that came out. And by messing with the bootloader stage, you know, I could go in and kind of mess with the kernel boot parameters and actually get it to boot into single user mode, which would then drop me to a root shell once the device had actually loaded. And from there, you know, was able to do further research.

14:23Chase SchultzYeah, and that makes me think about, you know, when I think about hardware and AppSec or product security in general, trusted boot is really one of the first things that I'm thinking about there. And so I'm wondering, what are your thoughts? I mean, is trusted boot really a large part of the answer here? Now, I don't think that would've solved the Spectre and Meltdown problem, and I want to get into that in a minute, but But is Trusted Boot really the answer? Is that something that we should be doing for all hardware-based devices as a, just a good security property in general?

15:00Chris RomeoYeah, definitely. And I can kind of point to a really good example. As of recent, I've been looking at some Samsung smart cameras and the, smart camera that I'm looking at just downloads firmware blind from, you know, one of Samsung's websites, and there's not any firmware signature or anything like that that's checked to see whether or not this, you know, firmware was signed from Samsung, etc. So anyway, a potential attacker that would be set up in a man-in-the-middle position during a firmware update would be able to unpack that firmware, make modifications, add their own password hash, or turn on SSH, etc., and then, you know, repackage that firmware and then make sure that it got down to you. And then when your device, you know, is looking at that and then like loads it and flashes it, you know, there's not that, you know, check then during the boot stage to see whether or not this firmware is signed. And so you can actually end up, you know, getting compromised in that way as well.

16:15Chase SchultzOkay.

16:16Chris RomeoSo yeah, I definitely think like Trusted Boot is a big deal because it's one of the most like fundamental stages where we can verify that the software that we intend to run is indeed, you know, the software is valid and and from the folks that we intend it to be from.

16:37Chase SchultzYeah, and then, um, and when I'm thinking about the other things that were kind of, uh, I guess lumped into the AppSec or product security and hardware, some other things are starting to come back to me now. Things like ASLR, Address Space Layout Randomization, which is kind of a hardware-software, I guess, combination. And then the no-execute bit within the Intel processors, I guess, is another— I don't know, maybe that's extended to other processor lines now as well. But I think of those things as kind of the traditional hardware capabilities to help protect software in general.

17:17Chris RomeoYeah, yeah, definitely.

17:19Chase SchultzSo let's, um, let's switch gears a little bit and let's talk about the Spectre and Meltdown in a little bit more detail because I think it would be beneficial for our listeners to kind of— for us to kind of break this down. And then, but because we're going to keep our AppSec colored glasses on though while we do this and be thinking about how AppSec and maybe even how some of the things we do in AppSec would've helped them in that process. But kind of from your perspective and based on the study and research and stuff that you've done, what's kind of the story behind this? What happened with Meltdown and Spectre?

17:55Chris RomeoYeah, so Meltdown and Spectre are really interesting because they both kind of focused on this area of speculative execution. Basically, what that means is that as a processor is going along and executing instructions and retrieving data from memory and loading that into a cache and the way that data is accessed as it's, you know, needed for execution, there may be times where there's either, you know, a significant performance impact of loading, you know, something from specifically like a lower latency such as cache versus loading something all the way from DRAM That sort of thing. And what we saw in this case was that they were able to actually train the speculative execution. So when the program— or like when the processor is waiting for a specific bit of data to be loaded all the way from RAM and and whatnot, the processor can go ahead and continue on and looking at various different conditional branches and executing ahead, as it were. And with Spectre specifically, it looks like they were training the kind of conditional branch evaluation. So when it would look at this, it would say, okay, I don't yet know what the value of this variable is. but I'm going to proceed as if this conditional were true. And then, you know, it goes ahead and tries to execute the rest of the function inside of that while, you know, while it's waiting for that data to catch up to see what the actual value of the conditional was. It's going ahead and and doing this speculative execution, which, you know, if it turns out that that conditional is evaluated, you know, differently and it wasn't supposed to be true, then, you know, it rolls back to that checkpoint to say, you know, okay, so this wasn't true, so now we go down this other path. But in that meantime, what was actually happening here was the microarchitecture state changes actually happen through this. And so during that time, there was, you know, further data either accessed or loaded into cache. And the kind of novel thing I think that was great in this paper was that the ability to kind of use some bit of timing analysis between, you know, how quickly data was accessed from the cache versus from like L1 cache or L2 cache or like the shared L3 cache, the timing is different for each one of those. So through that, you know, they were able to actually see and kind of peek at data that was, you know, supposedly secret because it changed the microarchitecture states and they were able to actually see some of that, the timing analysis, and be able to say, okay, this is this byte or this is this byte.

21:46Chase SchultzSo there was a window. There's basically a window of opportunity while this speculative execution— the speculative execution goes down kind of the path that it thinks that it's been tricked into going down. So there's a window of time before the processor figures out, oh, I wasn't supposed to do this, and it reverses course. So you're saying when that— during that window of time is when they were able to access things, when the attack is able to access things that it shouldn't be able to access from memory.

22:17Chris RomeoYeah, yes, yeah, exactly.

22:19Chase SchultzOkay, so this is— so it is an access to places in memory that you shouldn't have access to is really the sum of what this problem, which could, I mean, Memory's filled with all kinds of goodies, right? So, I mean—

22:36Chris RomeoOh yeah.

22:37Chase SchultzYou could, uh, you know, there's, there's keys and passwords and all kinds of good stuff that's stored in various places in memory. And if you figure out how to figure out where those are located, then you might be able to dump a key, a crypto key, for example, right out of, uh, right out of the, the memory of a system.

22:53Chris RomeoYeah, yeah, that's, that's the interesting part. And, you know, as the, the original papers kind of came out, I, I kind of wondered to myself, like, Is this, you know, another one of those kind of theoretical things? And, you know, what is the true practical impact here? You know, over time we started seeing, you know, more proof of concepts come out and, you know, the ability to, like, how this affects the running in the cloud and that sort of thing with Meltdowns, you know, specifically.

23:27Yeah.

23:29Chris Romeoit being able to access the kernel space and, you know, being able to read out the entire— the entirety of the kernel space where, you know, you might have very sensitive cryptographic operations like you said, that sort of thing. And, you know, people that are running in Docker or, you know, LXC, you know, kind of the platform as a service providers, out there could be majorly affected by this. And I, I think another thing that's interesting as it relates to AppSec is that the Spectre attacks were kind of specific to a bit of code. And one thing that I think will be interesting to see is if static code analysis tools start picking up these patterns to see if, you know, your code has similar patterns that need to be looked at to see if, you know, there's any other protections that can be done in software to kind of prevent the situations that make it, I don't know, amicable to, to an attacker, right?

24:44Yeah.

24:45Chase SchultzYeah. So I wonder So the— what's the solution to this, right? You can't patch a processor.

24:52Chris RomeoWell, yeah, that's the hardest thing here. We're dealing with hardware. I did see that Intel came out with some microcode updates, that sort of thing, so they're thinking about it. But a lot of this stuff is really going to be one of those types of situations where we have to wait on on them and the changes to silicon and the way that these are handled here. You know, we can do some of our defenses and mitigations in software, such as, you know, the Meltdown paper specifically calls out Kaiser and KASLA, Kernel Address Space Layer Randomization.

25:39Chase SchultzOkay, okay.

25:40Chris RomeoAs one of the areas that helps mitigate Meltdown.

25:46Chase Schultzspecifically.

25:47Chris RomeoAnd so through that, they, they were already kind of thinking about these side-channel type attacks.

25:55Chase SchultzYeah.

25:56Chris RomeoAnd hardening, you know, the Linux kernel against that. And I believe that since the papers came out, OS X and Microsoft and Linux have all, you know, kind of applied these patches.

26:10Chase SchultzOkay. Yeah. And I, um, I just realized some of our listeners might not know what ASLR is. So ASLR is Address Space Layout Randomization. In a, the most simplest form that I can think of to just describe it, when a program executes in its normal state using virtual memory, it always puts things in the same places, the same virtual memory addresses. And so what ASLR, and, and as Chase was just alluding to, when you add kernel ASLR on top of this, What that's going to do is it's going to provide some randomization so that spots or variables and things that are stored in memory are not going to ever be stored in the same exact spot. There's going to be some randomization that's going to occur, and so the virtual memory addresses are always going to be moving around, which makes things like buffer overruns and things like Meltdown more difficult to pull off in a— with a single exploit because things are always on the move in memory. So I don't know, is that— Chase, you want to add anything to that as far as that?

27:12Chris RomeoNo, no, no. I mean, that, that hits it on the head. Basically, you know, when you have ASLR enabled, uh, it makes it much harder from an attacker's perspective to, um, you know, figure out where your, your, uh, memory addresses are going to be, you know, where your shellcode lives or where your ROP gadgets are, are, you know, that's, that's kind of like one of the the other ways to kind of get around some of this, but you almost need an info leak or some way of knowing, you know, the base address of things or where things are going to live in memory. And so ASLR makes it harder for an attacker to perpetrate those types of attacks.

27:52Chase SchultzYeah, and that's, you know, I mean, that's what we're— that's what application security and product security are all about in general, is making things harder for the attacker. so that they can't— so it's not just easy when they get into a certain point in the system and they can do whatever they want. We're making them— once again, it all comes back to defense in depth and providing multiple layers that people have to jump through, and hopefully they get snagged on one of those layers along the way.

28:16Chris RomeoYep.

28:17Chase SchultzI wonder about— I don't know how much experience, I mean, you've had with the world of threat modeling, but I'm wondering, like, Could this have been caught? Could they have threat modeled this and found it in the processor design, or is this Meltdown and Spectre thing so low level that it would have been almost impossible to see this if you did a threat model?

28:40Chris RomeoThat's an interesting point. I kind of wonder that. It's a different world than my type of threat modeling where it's all in application, that sort of thing. Somebody that I think would be interesting to query that question to would be Joe Fitzpatrick, who used to work for Intel as a security guy on a hardware level.

29:08Okay.

29:10Chris RomeoThis would be interesting to ask him because I wonder if it could have been caught. You know, this was all in the name of optimization and had been, you know, kind of standard practice for a while. Because we see that, you know, this affected processors all the way back to 2010 or so. So, you know, it wasn't caught at the outset and probably just remained in there and that sort of thing. But I am curious to see whether or not, you know, Intel has security folks, you know, as part of these conversations of, you know, an optimization or this or that.

29:51Yeah.

29:51Chris RomeoDoes it have security implications?

29:53Chase SchultzYeah, and it's, you know, like you said, it's threat modeling at the application level seems daunting sometimes when you're looking at it from the outside, but it almost seems easy compared to threat modeling. When I think about how I would have to threat model hardware, and I know just a tiny little bit about how architectures and things work and instruction sets and memory addresses and all that, and I'm thinking, boy, that seems like it would be really hard to do. Not impossible, but really hard. So Chase, thanks for taking the time today to walk us through what you know about Meltdown and Spectre and how AppSec and hardware kind of come together. And we'll have to have a further conversation maybe in season 4 about some of the things you've done in the IoT space and IoT product investigation in regards to AppSec. I'd love to pick your brain on that as well.

30:43Chris RomeoSure.

30:43Chase SchultzBut for today, we're out of time. So thank you for taking the time, and it's really been good. Thanks.

30:48Chris RomeoYeah, thanks, Chris. Thanks for having me on.

30:50Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @appsecpodcast or on the web at www.appsecpodcast.com.

31:11Chris Romeowww.appsecpodcast.org.