Brian Andrzejewski -- Containers Again

0:05Brian AndrzejewskiThe Application Security Podcast. Here we go.

0:11Chris RomeoHey everyone, welcome back to another episode of the Application Security Podcast. This is the final episode at the AppSec USA conference in Orlando, and Chris and Robert are joined by Brian Najewski. He talks about containers, their usage within AppSec, and about orchestration. As always, thanks for listening and enjoy.

0:46Brian AndrzejewskiHey folks, Robert and I are back at AppSec USA, and today we're going to explore containers. containers and application security. And so, Brian, right off the start, why don't you introduce yourself and tell us your security origin story? How did you get into this crazy world of application security?

1:10Chris RomeoMy name is Brian Urjeski. I'm actually the lead information security engineer for US Citizenship and Immigration Services. We're a DHS component formerly out of INS, which then split into ICE and us. My particular background is actually— was kind of weird because I got started in IT by accident and then grunged and then worked my way up to where I am now. So I did help desk, I did desktop support, I did all the tiers of the help desk, got into some analyst work, worked in system admin, started doing config change management, a little bit of risk management, and then— I was actually hired by the government as a contractor originally to do application development in PHP, which was like 2 generations old, and they paid more, so I was like, okay, sure, and kind of spawned from there. My government boss at the time took me under her wing, did a direct hire, and I was doing digital forensics exercises with the public, so, and then all the collegiate and CNTI 7 activities, which was accelerating cyber education in the federal government.

2:26Brian AndrzejewskiOkay, and then so did that land you in the— so you kind of landed in the world of AppSec, so you transitioned into this PHP kind of developer role and then somebody kind of helped to bring you along. Was your next stop AppSec from there?

2:42Chris RomeoYes, from there, because I had the developer background, I had the sysadmin background, and what happened is DOD sequestration hit and this department actually got completely canceled. So all public outreach got killed out. So I got reassigned to the IT department, had IT background, and I started doing their risk management and security controls since I had a prior healthcare background. Okay.

3:04Brian AndrzejewskiYeah, it's funny, we ask this question of everybody and there's the sysadmin side, there's the developer side. You kind of come in from both of those perspectives too. So you're doing a talk today here at AppSec USA. And so, from what I understand, it's about your story or your experiences in applying AppSec to an enterprise in regards to container security. And so I'd say let's just start, just tell us the story here that you're going to tell today.

3:33Chris RomeoI had started at USCIS. I was only about 3 months in. I was hired to do the application security because I was trained with certain products, commercial products, and I had experience doing those. The CIO goes, that guy knows of your IT, of your security department, knows about application security. I want to tiger team him. I want to deploy containers and I want it in 6 months. I was like, okay, yes sir. So I sat with the developers for literally 3 months. We did like basic Docker containers, figured out a little bit of the policy of how to manage that. Trying to figure out how to get the security tools to work with that was fun. So we were doing stupid stuff like, oh, we enable SSH on the container and let me run my standard security tools to see what comes back, or poking and prodding the processes from root and see, oh, if I destroy this process or take its volume away, what happens? A lot of that was from my prior digital forensics training background when I was working at a defense cybercrime center. Then a lot of that comes into the policy and process, which a lot of that technology still doesn't have today, especially at the enterprise level. We played with a lot of the orchestration frameworks, both open source and commercial. We had vendors on site, we gave them requirements, we could send them through the battery of tests. A couple of those vendors stayed, a lot of them left because they couldn't meet from federal requirements a lot of the things that we have to do.

4:59Brian AndrzejewskiWhat type of requirements are those?

5:02Chris RomeoSo I'm about— since we process significant loads of Privacy Act information, so anybody that's coming in with a visa, immigration statuses, We all treat it as like US citizen PII because they have the possibility of becoming a US citizen. So all the Privacy Act, so system of records, privacy impact assessments. So we had to look like down at the kernel level because the kernel is what's being shared on these hosts to do the processing of the containers at a virtualization level.

5:33Brian AndrzejewskiSo you were talking about how you had SSH enabled on the containers in the early days, and I'm kind of I'm kind of taking away that you might not have thought that was a best practice for where you are now. So what's your take on— I mean, should containers just be isolated to the point where there's nothing, like you can just burn them, throw it away, and make another one? So I guess what's— I mean, would you use SSH in a container today, or is that something that's kind of a lesson learned you figured out?

6:00Chris RomeoIt was a lesson learned, and a lot of the benchmarks weren't around. So Center for Internet Security benchmark wasn't that mature yet. They had it as a possible non-scored control at the time. Now it's a scored control, a little pressure from us. And we also made sure some of those best practices we learned, we contributed to both drafts of the new NIST publication for application container security. Okay.

6:28Brian AndrzejewskiAnd so you— I guess we kind of— we were talking about how you had kind of made the— that made the move. You started to do some initial things and Where did you go from there?

6:38Chris RomeoWe started working on the orchestration. So we already had—

6:43Brian AndrzejewskiWhat does that mean too, by the way? Our listeners may not understand what orchestration is.

6:48Chris RomeoUsing automated means for deployment and release or blocking release. We're all about as much automation as possible. We have several hundred developers and several hundred programs, so resources are limited, and we also typically use contractors to do our development practices. There can be anywhere from 30 to 40 different contracts spread across that with timelines of 6 months to a year and then renewals of options if we elect. So that means your staff is always rotating. So having those practices down is crucial, especially documented. And of course, each team that comes in, especially if they're coming from typical commercial industry, kind of has their own little practices they like to do, which then don't match up for what we like to do because—

7:35Robert HurlbutRight.

7:35Chris Romeofrom what our testing has provided.

7:36Brian AndrzejewskiSo you talked about policies and processes, and I spend a lot of time in the startup world where there is no— they just laugh if you're like, oh, policy?

7:46Chris RomeoWhat is that?

7:47Brian AndrzejewskiWe'll do that after our Series B round. Then they'll force us to have policies. So what type of policies then would you recommend to— and processes that are well-defined for an enterprise that may be not as far along as you are right now as far as having a solid container security approach?

8:08Chris RomeoIn the maturity model I'll be discussing later today, I'm talking about from the host level, the data level, and the image level, 'cause it goes back to those 3. Your host is where everything kind of commingles, 'cause it commingles in that kernel, so you gotta do certain things there. You also have the images, so everyone's like, oh, it's a Docker image, it's all on Docker Hub because it's open source, so I'll just use that. Well, people don't realize, and a lot of those, even that image technology and the other image technologies, A lot of them are copy-on-write systems, so you have layered file systems. You can see everything, the history that's in there. Sometimes there's malware baked in there. We have alerted several of those container hosts about that.

8:52Brian AndrzejewskiSo those are Docker images on the Docker Hub that somebody has sneaked through with malware embedded in them?

8:59Chris RomeoYes, that's increasing. Some of the companies have made mitigations against that, but they're doing it at a CVE level. They're not doing it at a, oh, this is a custom piece of malware. that phone tone. And the only way you're going to do that is through traditional black box pen test assessment where you're watching processes, network, and then all of your syscalls.

9:19Robert HurlbutYeah, I remember a couple years ago I went to a talk by Tony Uvee, who we interviewed before on the show, and he had a talk about threat modeling tainted containers and talked about that particular attack vector that That's another way that we're not thinking about that. You just pull down these containers, you don't know what's in them. They're still untrusted data and pieces of hardware, or not hardware, but some software and other things that may be inside and you don't know. And so you need to treat it as such. So 2 years later, you mentioning that, it kind of brings that to mind that, yeah, it's out there. And he was kind of warning ahead of time to say, As this continues to ramp up and continues to be popular, we need to think about this stuff.

10:08Brian AndrzejewskiI mean, it's a common— I mean, that threat is starting to make its way across other places. Like, you hear about corrupt apps. I've been reading about these different game apps that it's basically like a shell that sits on top of malware to get people, and they make them look like something that's a real game. So if you're not paying attention, you might download this one, you go to run it, and then it says, oh, we need your credit card to continue. So, and that's gonna— I think that's gonna be a more and more popular threat over time when we— it's always been a threat in the open source world, but open source has been pretty good about catching that stuff in code by not just letting people check in random things to the Linux kernel.

10:45Chris RomeoYeah, it all comes down to maintainer and how many maintainers and who's watching. So if it's like one maintainer, it can have anything at that point. But from what you're talking about, what we see is By default, a lot of these container technologies, they run as root, which means you can escalate right up back to kernel.

11:04Robert HurlbutMm-hmm.

11:05Chris RomeoThe syscalls are usually wide open. Docker open source, I think I may be misquoting the number, it's about, it's 100, it's like high hundreds, maybe low 200s of the possible like 350 syscalls, and some of them are just wide open. A lot of people are not looking at that right now. There are some tools to help profile that on an application basis and you do it on a runtime.

11:30Brian AndrzejewskiSo from your perspective then, is that something that you're— in the way that you recommend people roll these types of programs out and handle these things, are you recommending that some of those things, like should containers actually run as root or what's the best practice here?

11:47Chris RomeoRight. Always run as a non-privileged user. That's in the CISS benchmarks from day one and a lot of the vendors now You said it does nobody, application, user, as long as it's not root. What one of the practices I'm not seeing, which we like a lot, we have difficulty as well because it requires some manual fiddling with the app, is read-only containers.

12:08Robert HurlbutHmm.

12:09Chris RomeoSo basically you're setting the container at runtime in a read-only state and you're giving it only certain mount points to write data to, like tmpfs or tmp, and it cannot write to any other portion of that file system. What that also helps with is your data management. So you're making sure no data is being written to that container. So when that container dies, guess what? The data dies with the container.

12:29Brian AndrzejewskiSo you can deploy a web application then in a read-only container, and then that— well, that sounds like it'd be a really good, just a general security feature to say I don't have to worry about something compromising this system. Even if it was able to compromise something in memory, it wouldn't be able to write anything back out to disk.

12:47Chris RomeoRight. They'd have to deal with memory There's possibilities there, but you're lowering your surface area of attack.

12:52Robert HurlbutSure.

12:55Brian AndrzejewskiSo we— so I mean, any other best practices that you're kind of thinking about, the things that people get wrong from the default out of the box that you think people should look at?

13:04Chris RomeoThe one I was surprised about that did not come out in the NIST work and we gave comment on in the last draft was data management, persistence of data and how to do that. A lot of apps still want to bind to the host, which means then you have no resiliency in your host. And it requires then some modifications of the host because of that. You also have some of the more software as a service technologies that operate as like data brokers. A lot of the drivers to those are very alpha/beta. So especially if you have like high I/O apps, I'm personally against having any databases actually in containers. It's been improving as the technology is improving. But anything that's like high transactional, which with high I/O, I've crashed containers multiple times, especially in prod. I have no problem with that for mocks and tests and dev so developers can get their work done until they get to their release environment, the CI/CD release environment, and they can deploy there and run through their battery.

14:06Brian AndrzejewskiNow you mentioned this maturity model, and so tell us more about the maturity model.

14:14Chris RomeoAgain, that's focusing on the host, the data, and then the operations. So I'm basically ranking, okay, if you're like kind of starting out, these are the things to focus on, usually like top 2, top 3. And then as you get more advanced and you start getting to like true optimization and knowing your environment, these are going to be your like your top 3 things to do. We're not even fully there yet. A lot of it's because I'm testing the technology, putting it through its paces. Of course, there's upsets like finally Windows Docker finally works, but of course it just literally just reaches right into the kernel with no controls on syscalls. So you have some— now you have Linux versus Windows on how they operate, and you got some of the older technologies like LXC, they operate a little differently.

15:01Brian AndrzejewskiSo how does somebody put that maturity model to use? Is it like the first level has some basic things that they want to do And then there's corresponding levels and with the goal of them reaching— what's the top level?

15:17Chris RomeoOptimization, full optimization.

15:18Brian AndrzejewskiOkay, so flow. So their goal then is to transition. Now, is there application security things built into each of the levels?

15:25Chris RomeoYeah, I'm doing hardening, CVE detection, vulnerability detection, like, and then runtime protections, which are more in the laters. And that's basically from how we learn how to use the technology and how I see other people using the technology.

15:43Brian AndrzejewskiSo from a hardening perspective, you're focused in on the kind of automated ensuring that you can just push a button and the container comes out with all the things that we've talked about already. It's not running as root. It's got some restrictions on the system calls.

15:59Chris RomeoTrusted sources.

16:00Brian AndrzejewskiTrusted sources for things. So that's kind of all— so I guess in the container world, that's all in your automation scripts, whereas in the old days when I used to do stuff like this, It involved visiting hundreds, if not millions or thousands or millions of servers around and typing in 17 pages of commands hoping that it all got done correctly.

16:18Chris RomeoI mean, the key thing to remember about containers, it's just a package management system. It's literally— most of them are just tars with an MD5 hash and some metadata around them so that when you do it run, it is going to run consistently. Now the question is, when it's in that runtime, is it going to be consistent? Because what's happening is some of those apps are now drifting. And then they somewhat change their config depending if they've been attacked or it's just a really bad configuration deployment.

16:43Brian AndrzejewskiSo then from a CVE detection perspective, is that something that from a process perspective happens at build? So when you build the container, you do some type of a check to see, are there any known CVEs? Do you have like a manifest or a manifest of all the packages that are running? So you can check to see what CVEs might exist?

17:06Chris RomeoSo all the container technologies will typically show the steps of how they did the build. It's a matter of the tool of how deep it can go. Some of the tools will only start at the OS level and maybe, um, actually a lot of them will just kind of stop there right now. A couple other tools that are on the market, both open source and commercial, will start drilling down into the native package managers like pip, npm, And then they'll do their CVE detection and pivot from there. Some more of the advanced tools are starting to actually start doing typical kind of like malware sandbox kind of activities. So they start taking, they'll start watching the processes, recording the PID, how it executed, and start hashing it. And then you can start doing your typical attack vector matching for indicators of compromise from that direction.

17:55Robert HurlbutHmm.

17:56Brian AndrzejewskiSo from the CVE detection perspective, is that something in the container that's only happening at the build time? Or, I mean, I guess, or are you— is it best practice to just recycle your containers once every 24 hours and just have them rebuild?

18:10Chris RomeoYou can. It depends on your organization policies. Like, if we have like a major— if there's a major US-CERT alert that's a critical or high-von, typically they have a reporting period of 7 days. to either report what your status is or if you fixed it. So in the container world, none of our containers last past like maybe 48 or 24 hours.

18:32Robert HurlbutOkay.

18:33Chris RomeoWe made an organizational decision to do that.

18:36Brian AndrzejewskiContinue to recycle.

18:37Robert HurlbutRight.

18:37Chris RomeoThe other one is patching. So a lot of the containers, especially on public market, do not patch even from their upsource image. So they'll have all sorts of runnable things. So it's making sure you're building those patching into your image build process so that Since it's a package, you want to make sure the patches are applied in that image build. So when they go and do their testing of that image, it's already been tested and vetted with the patches already involved instead of just doing it after the fact and then they'll blame the patches.

19:06Brian AndrzejewskiSo then on the vulnerability detection side, I'm guessing this just means, I mean, is there some type of interface that you're able to access from outside the container or how do you detect vulnerabilities that are inside a container?

19:22Chris RomeoSo, I mean, it's pretty much a binary at that point. So what the tool does is it'll take like the static image, pull it down, and then rip and strip. And then also basically do the file system first. For the runtime, it requires that container to be running in a test environment. And then what it'll do is usually it'll use some kind of privileged container to watch those actions. Privileged means it's running as root and has access to all the other containers on that host. And it'll sit there and just watch the activity and then report it back and then hit its compliance rules or enforcement rules.

19:53Brian AndrzejewskiNow, when we talk about vulnerability detection and then these kind of runtime checks and things, is this related to the applications that are running on top of or inside of these containers, or is it more the infrastructure of the container, the Linux kernel, the different packages that are providing the operating system?

20:14Chris RomeoSo some of the technology today will actually go through the whole entire stack. So it can see into the host and actually flag CVEs in the host at the daemon level and at the container and the application level, depending. Okay.

20:27Brian AndrzejewskiNow, do they do any type of dynamic scanning like we would from a general AppSec perspective? We think of a dynamic scanner as being something that is a tool we can point at a running application from outside and try to see if there's any different vulnerabilities that it can deduce. Is that possible in the container world? Is it needed in the container world?

20:49Chris RomeoEveryone's still using traditional means to do that. There's one or two vendors that are looking at basically injecting web application firewalls at the container level. So literally, as their wrapper, they're doing that WAF at that level. before it allows the network execution coming into the container.

21:09Brian AndrzejewskiSo WAF on board, right inside the container. So it's almost like we're gonna have this minimal world, we're gonna start adding all these other pieces into it, and it's basically— oh, inside your container, that's the cloud now, right? Pretty much.

21:23Chris RomeoIt's a mini cloud.

21:24Brian AndrzejewskiYeah, you're gonna have everything. That's a good way to think about it, actually, a mini cloud.

21:28Chris RomeoBecause for us, it's about It's the whole problem— well, it worked on my machine problem. So for a dev, they can run that container at that local— on their localhost as long as they have their mocks to help with the data side. And then they can deploy to test. We've seen very little variation there because they make configuration, they inject their environment variables at runtime. So they can say, okay, for my dev environment, these are my environment variables for the app, for test, and then here's for And then, of course, the secrets management. Now, I've been seeing a lot of secrets being baked into the file system. Of course, they're in the clear. Anybody can just run a simple history command, you can see those. Some of the tech is getting better, like Kubernetes and older versions on the host, the creds are stored in the clear.

22:17Brian AndrzejewskiMm-hmm.

22:18Chris RomeoSo they're assuming a risk trust boundary that, hey, if you have root to that You would get it anyway. You'd get it anyway. So in our world, no, that doesn't fly.

22:28Robert HurlbutNo. We should all learn some DevOps.

22:31Chris RomeoRight, exactly.

22:32Brian AndrzejewskiSo now, does your maturity model have something about secrets management in it? Yes. Okay. And what's the expectation then from a container perspective as far as our environment variables, passing secrets to environment variables? Is that an accepted practice?

22:47Chris RomeoThat's pretty much like the middle-of-the-road practice. The better practice is They come in encrypted, then you use some kind of third-party tool when they need to be accessed to decrypt.

22:58Robert HurlbutAnd let that manage it.

22:59Chris RomeoAnd let that manage it, 'cause then you're not keeping them in memory the whole time.

23:02Brian AndrzejewskiRight, that makes sense.

23:03Chris RomeoAnd you can't use Volatility or something to scrape it out. Yeah, right.

23:07Brian AndrzejewskiSo I guess kind of a way to wrap up our conversation here, and I'd love to get your thoughts on, so, We'll kind of put ourselves into a scenario here where, you know, you've got a lot of experience in rolling out containers and making sure everything is secure. If we had somebody who's listening here who maybe just, Brent, might be starting out in the world of Docker, maybe they know Docker, but they don't know Docker from the enterprise perspective.

23:36Chris RomeoRight.

23:36Brian AndrzejewskiWhat does that look like from your perspective? What are some of the things that you would recommend, kind of the lessons learned you took away now Now that you've got the scars and everything just to demonstrate, like, you know, you've already been down these roads. You've mentioned a couple as we go, but I'm just— I want to kind of spend the rest of our time focused in on what could somebody do who wants to take away— who wants to borrow a little bit of your knowledge and apply it to what they're doing.

24:02Chris RomeoThat's why I'm— that's the big thing why I'm putting the maturity amount on that, because I'm government, so public affairs publishing is a little hard. This is a backdoor method to kind of be like, hey, by the way, this is what we're recommending. They may not be making these other 2 main publications, CIS Security Benchmarks and then NIST, because they have committees.

24:23Robert HurlbutMm-hmm.

24:23Chris RomeoSo we're kind of saying this is what we believe how things should be. The other thing is I keep seeing these containers increase more as binaries, almost being treated as firmware. So You're going to see a lot more Internet of Things starting to pick this up.

24:39Robert HurlbutHmm. And start running with it, okay.

24:43Chris RomeoAnd start running with it.

24:43Robert HurlbutYeah.

24:44Chris RomeoBecause it's an easy deployment model for that.

24:45Robert HurlbutSure, of course.

24:46Chris RomeoFrom a starting standpoint, I highly recommend the CIS Security Benchmarks. They have a Level 1 and a Level 2. The Level 1 is like general compute and then Level 2 is more sensitive operations. And they have scored and not scored. We tend to take them all as scored. There's some that are subjective, but a lot of them are objective now. We've been working that out.

25:09Brian AndrzejewskiWhat does scored versus unscored mean?

25:12Chris RomeoAnd on some of the open source tools and the commercial tools, the scoring means you can enforce them, that it should occur. The non-scored are typically more of the subjective ones where you have to kind of figure out what you really want.

25:25Brian AndrzejewskiOh, so like how many vulnerable packages are you able to find? 22 on your device. If greater than 10, then you need to just quit and go back and start over, start the process over again.

25:35Chris RomeoYeah, DoD STIG and a couple of the hardening policies have the same stuff, like find all the accounts that may have 777 privilege and then how you deal with that.

25:46Brian AndrzejewskiYeah, I'm still thinking back to that containers as firmware thing you mentioned here from the IoT perspective and I'm kind of debating in my own mind, like IoT security's been pretty bad in general as an industry now. I'm wondering if containers might actually make it better.

26:05Chris RomeoThat's what I was wondering when you mentioned it. We just pull from registry every once in a while.

26:09Brian AndrzejewskiYeah, I mean, do you think it's— do you see that as something that could actually make IoT better?

26:14Chris RomeoYeah, I mean, there's already the signing technology there, so I mean, it's a signed package. they could easily set the trust that's only to them, so that prevents— that's an integrity check. They don't have any encrypted yet, but I don't think that's really needed. But your registry sources would matter too.

26:33Brian AndrzejewskiIf any IoT people are listening out there, get Docker, build stuff with Docker, all the things.

26:41Robert HurlbutThink about it, yeah, that's great.

26:43Brian AndrzejewskiBecause it will improve your security model. So any resources that you recommend? Obviously the maturity model is something that you're going to recommend, and I know I'm going to go take a We're gonna give it straight to OWASP to post. Oh really? Awesome, even better. That's great.

26:58Chris RomeoAnything I do speaking, because this is a public forum, is considered public content.

27:02Brian AndrzejewskiOpen source.

27:04Chris RomeoActually, government cannot hold copyright, so because of how the agreement for OWASP is, they're holding an attribution, but they're typical.

27:13Brian AndrzejewskiNow, is there a project name for that, or you're just gonna release it as that? Just release as that. Okay, so that's where folks can go to track it down. What else? What other resources helped you in your kind of container security journey? Where would you— what would you recommend people go and take a look at?

27:28Chris RomeoThere's actually a couple of like pre-made public clouds. We actually cut our teeth first on Amazon ECS 'cause it was just an agent and auto scaling just kind of handled having that same container on that same host and then scaling it out on CPU or memory load. Red Hat has one as well, which is free for use and kind of playing around with. Of course, you can always use the typical, like, cheap cloud providers that are out there as well, where they charge by the hour.

28:03Brian AndrzejewskiOkay, so yeah, so that sounds like good advice though for folks that want to get started, is to get out into these kind of public, you know, use some like you guys did, use some of something that's a little easier Has less of a learning curve as a way to get started here versus trying to dive in and build all the infrastructure first because you're never going to get anywhere with that.

28:25Chris RomeoAnd some of the enterprise orchestration tools can be kind of daunting, so you have to understand what that container is doing, what it looks like at runtime, and then start pivoting and orchestrating out.

28:35Robert HurlbutCool.

28:37Brian AndrzejewskiWell, Brian, thank you for your time to share this with us. I learned quite a bit about containers that I didn't really realize, and I'm going to go start telling anyone in the IoT space, Brian said you should look at this maturity model and you should build your software in containers so that it's signed and don't make it run as root and control your system calls. So yeah, thank you very much for being here.

28:59Robert HurlbutAppreciate it.

29:00Chris RomeoThank you.

29:01Robert HurlbutThanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.