Apollo Clark -- Malicious User Stories

0:00Chris RomeoHey folks, season 3, episode 18. Robert interviews Apollo Clark at the Source Conference in Boston. They talk about malicious user stories, DevOps, and some of the contributions Apollo's making to the world of OWASP. So we hope you enjoy. The Application Security Podcast.

0:25Apollo ClarkHere we go.

0:47Chris RomeoHello folks, this is Robert and I'm at the Source Boston conference here in the Boston area. And again, I've mentioned before, this is one of my favorite conferences. I enjoy coming to this one and just meeting the different people that go to this particular conference. It's really a great place, what they call a hallway conference, where you talk with people and meet them and so on. And of course, great speakers as well. Today I'm joined by, or with rather, Apollo Clark. Apollo, welcome.

1:22Apollo ClarkThank you for having me.

1:23Chris RomeoSo Apollo, introduce yourself. Tell us a little bit about yourself.

1:26Apollo ClarkYeah, so I currently work as a DevOps and security consultant here in Boston. Started originally as a web developer back in 2001. Okay, so I have more of a technical background, trying to learn more business development recently.

1:41Chris RomeoOkay, great. And so one of the things we like to do when we start off with our Application Security Podcast interviews is to ask, what is your security hero? You know, every superhero has an origin story, so a security hero— what is your origin story? How did you get into security?

1:58Apollo ClarkAs I tell people, security found me. I was working at a startup and it got hacked 5 ways to Sunday. So as the web developer for it, you know, it was a bit of a pride thing to me, you know, to see a system that I helped design just be completely wrecked by somebody else. Ever since that day, I ended up joining OWASP, which made sense as a web developer. I started teaching myself how to do things like SQL injection, cross-site scripting. I got more into DevOps as well, and I started realizing that the the automation toolsets actually make it easier to do security.

2:29Chris RomeoOkay, great. And so you said you started also as a developer.

2:33Yeah.

2:34Chris RomeoWhat were some things that you did in terms of— did you make a transition? Did you go from maybe only developer to just security, or did you do a little bit of both, or are you still combining both?

2:44Apollo ClarkI still combine both to this day. So one of my current clients, I'm helping them deploy their security logging pipelines.

2:50Okay.

2:52Chris RomeoSo, still keeping active in the latest development techniques and languages and all those kinds of things, as well as applying security. Okay, great.

3:00Apollo ClarkMy GitHub is very active.

3:02Chris RomeoOkay, fantastic. And so, at this conference, you're speaking on, from what I understand, malicious user stories integrating security into DevOps. So, tell me about what that talk was about. Yeah.

3:14Apollo ClarkSo, the idea of user stories, a lot of it came out of marketing and sales. So, the idea was that if you have a customer segment that you would sell to, Those become the end users of a given system. That concept then evolved into what are known as personas. These personas are the people that will be using the system, and they're used very often with Scrum and Agile systems to say, what are we building? But more importantly, who are we building it for and how are they going to use it? And really, to what end? Like, what value are you providing? I took that same concept and said, how could you apply that to a security mindset? What's interesting is doing some research on this. I've never done this before until very recently. was, you know, what have other people done? And there's quite a few articles you can find about people attempting it without too many specific examples. So I'm working with my client right now. They're trying to adopt more DevOps methodologies as well as integrate security better. So I've been talking to them recently about this, and I thought this would be a really good presentation to give here at SourceBoston, particularly with a lot of the executives and business focus for the conference.

4:14Chris RomeoOkay, so in terms of, as you call it, malicious user story. Give me an example of one of those, perhaps.

4:21Apollo ClarkYep. So some of the common attackers, we call them personas, would be things like script kiddies, you know, the Russian gangster hackers, the hacktivists, etc. So a good example with this would be, as a script kiddie, I should not be able to use a known vulnerability to exfiltrate data for the value of protecting brand identity. And what's nice about that user story is that it's very simple, very concise. The point of a user story is that it should fit on a note card. shouldn't have more than 1 or 2 commas, and you should not use the word and too often because it's really, really easy to make these long and rambling. Sure. So once you develop that, it's something that I like to say acts as a contract between the business development, project owners, project managers, developers, operations, and security. It's a single document that everyone can agree to and say, this is what we're going to do.

5:12Chris RomeoOkay, so it— okay, that makes sense. So it's focused just like a user story, a regular user story. So not many ands. It's attacker-focused in this sense because you're defining who the attacker is. So as a user, I want to do something as an attacker, and you specify what kind of attacker. Yeah, I shouldn't be able to do something versus I should not be able to do something. Correct, right. And then the outcomes and so on. So how do you— and that also can lead to how do you test it, right?

5:40Apollo ClarkYep.

5:41Chris RomeoSo because you give some of that, like for example, you said should not be able to exfiltrate data.

5:45Apollo ClarkYep.

5:45Chris RomeoSo then the test is, can you exfiltrate data, you know, and so on, for example? Okay.

5:50Apollo ClarkYep. So the user stories work really well because you can give these to business executives and say, you know, here's what we're shooting for. And with my current client, we went through a very specific regulation, New York State NYDFS, and we went line by line and said, okay, how can we convert this regulation into a user story? Total, we ended up creating over 85 of them, which isn't to say that every project should have 85 security user stories. That's a bit much. I suggested that each software team, either already deployed or greenfield or anything in between, pick 10 of them. Pick 10 of them, work through them over a few months, see how it goes, see how far you can get. And that really should be the goal. And as you get through those 10, pick off 5 more, pick off 5 more, etc., etc. One thing that helps a lot too is that the user stories, if they're high-level enough, can be universal for the entire company. You could say, for For example, data exfiltration. That's a generic enough user story. You can say every piece of software this business is deploying should not be vulnerable to this. And that's something executives can say, yep, we agree with that. That is a great idea. And you can use that later on to justify things like budgets, headcount, as well as buying third-party products. And that goes into specific implementations. Ideally, instead of saying to developers, here's the requirements, figure it out, it's even better if the security team can say, here's the requirements, But also too, here's a possible solution you can use. So this comes into things like third-party vendors, or if you're using cloud resources saying, oh, here's how you configure that firewall, or here's how you configure this service securely. Okay, great.

7:22Chris RomeoGreat. So it makes sense. And you mentioned integrating that into DevOps. So with DevOps, you know, why is this in particular a great way to do security? So tell me a little bit about maybe DevOps, but also how this is great to integrate Security into DevOps?

7:38Apollo ClarkYeah, so what's happening very recently, like literally the past year now, is you're seeing a lot more DevOps practitioners wanting to do security, which is great. On the flip side too, DevOps is taking over all of IT. It's replacing Scrum and Agile as far as methodology goes. On the security side, people are wanting to learn more DevOps techniques, so it's kind of fun seeing this blending of worlds. So it's happening regardless of what anyone wants.

8:03Right.

8:04Apollo ClarkSo everyone has to give the program. But the good thing is, again, these user stories, they allow us to bridge communication between various departments and requirements. Taking them a step further, so once you develop your personas, the attackers, the use cases— I'm sorry, the user stories— you can develop them into specific use cases that say, okay, for this given product, this given project, this given server type, be it Windows or Linux or whatever, here's how we're going to implement it. And it gives you It gives the development team something to point to and say, yeah, we're working with the security team and here's how we're going to launch it. Okay.

8:37Chris RomeoSo let's talk about DevOps in particular. How does this— how would you define DevOps? I know there are lots of definitions out there, but how do you define DevOps?

8:46Apollo ClarkYeah. So what's really funny is I went to the Boston DevOps meetup literally 3 weeks ago, and it was a 10-year retrospective on DevOps. And it was funny because we had a panel of 5 people. And they were asked the exact same question, what is DevOps? And we got 5 different answers.

9:02Chris RomeoOf course.

9:02Apollo ClarkYeah, I mean, people look at it from different things. The big joke I tell everyone during my presentations is biz dev only hears one thing when they hear DevOps, which is more features faster, which is great, but there's not a lot of details there as far as implementation goes. The tech people though, they see specific technologies, mostly things like continuous integration, continuous deployment, which are great. But the DevOps Handbook and the Phoenix Project, they weren't overly prescriptive as far as saying, here's the DevOps Manifesto. There is no DevOps Manifesto, unlike Agile and Scrum. And that was intentional. The DevOps practitioners like Gene Kim, Patrick DeBois, they intentionally said, we want to have DevOps be more of a conceptual framework that says, what are you trying to accomplish as a business and what do you value?

9:52Right.

9:52Apollo ClarkHow does that translate into specific business practices? And then how does it translate the last time into the technical implementations? So given that, I would consider DevOps to be a collection of values that are reflective of your business and your IT operations. Within the DevOps mindset and within the Phoenix Project, they mentioned the idea of the 3 ways. So, way number 1 is that you want to do as many deployments as possible, as quickly as possible, using very small increments. And you want to be able to focus on global values. So, for example, it's very easy to be a developer and saying, oh, we pushed out a bunch of features, or ops would say, we've limited the number of incidents. And that's great in confinement, but the question is, how does that relate to the larger business? You know, how does this help the sales team? How does this help the marketing team. So I would argue things like pushing out newer features, it helps the sales team because suddenly they have new features to sell to customers. They can reach new market segments. If you look at things like operations and say we're having fewer SevZeros, you can directly tie that to customer churn.

11:01Chris RomeoYeah.

11:02Apollo ClarkSo that's what the idea of taking a larger perspective outside of just your specific department and even IT itself. The second way is to go through and make sure that you're actually measuring what you're doing. So once you're measuring things, you can actually start setting goals and saying, okay, are we actually improving at all versus just sitting here spinning our wheels? The third way, which is very rare, most companies actually get to this point, is are we continuously breaking our own systems?

11:29Chris RomeoAll right.

11:30Apollo ClarkYou know, are we really pushing the envelope? Are we creating chaos and havoc? But are we also learning it and practicing mastery of the recovery? So Netflix has their famous simian monkey army.

11:43Yep.

11:43Apollo ClarkAnd those things go through and they literally turn off servers. They turn off entire availability zones. Like, okay, the entirety of US West, we're turning it off. Recover.

11:53Chris RomeoRight, right.

11:54Apollo ClarkYeah. Not many companies get to that point, but that's kind of like the goal. Okay.

11:57Chris RomeoOkay.

11:58Apollo ClarkMakes sense.

11:59Chris RomeoI've also heard, you know, culture, which I think relates to some of the things you mentioned. Yeah. Continuous star, you know, delivery and integration and all those kinds of things. things as well. Okay, very good. What about— you talked about some ways to integrate security through the stories, but are there other ways to integrate security into DevOps?

12:17Apollo ClarkOh, absolutely. So what I spend a lot of my time doing with my clients is showing them— I'd say maybe 25% of my time is spent showing management methodologies, and that includes things like using ticketing systems, ensuring metrics, etc., and how you want to manage those expectations. But 75% of my time is spent actually implementing the technology behind it. So, things like, how do you automate deployment? How do you automate building integration? And that leads directly back to security because when you can build a service from scratch, like literally raw source code, and install all the latest patches, security updates, newest packages, and run that through a testing suite, that's pretty damn good security to me.

12:58Chris RomeoYeah.

12:59Wow. Great.

13:00Chris RomeoYou also mentioned about OWASP, that you're pretty active with OWASP. What are some things that— any projects that you've looked at or worked with in the past?

13:10Apollo ClarkYeah, as of last— as of 3 days ago, actually, I did this 4 years ago, which kind of blows my mind. I designed the t-shirt for the Boston Application Security Conference. In the process of doing that, I thought it'd be really cool to create icons for the OWASP Top 10. And that became the backside of the t-shirt.

13:27Chris RomeoI remember the t-shirt. Yeah, I have it actually.

13:29Apollo ClarkNice. Yeah, I took that to DEF CON a few years back and people just loved it. They wanted to buy it left and right, but it was only if you were in Boston. So I gave that to the OWASP Foundation. And, you know, recently, I think it was back in November, they released the update to the OWASP Top 10, the OWASP Top 10 2017.

13:45Right.

13:46Apollo ClarkWhat I've been working on the past couple months is making an update to that. So 2 days ago, I published that and released that on Twitter.

13:52Chris RomeoOh, wow, fantastic. I'll have to take a look. So have new icons that match the OWASP Top 10 2017. Yep.

13:59Apollo ClarkFantastic. Hopefully we'll see that. I was talking to somebody there from the OWASP Foundation. He said, oh, you got to make posters of this now.

14:04Chris RomeoRight, right. Yeah. Okay, great. I'll definitely have to look for it. All right. And then some other projects. I'm familiar with Gauntlet. I've used it. That's something you've also worked on as well.

14:15Apollo ClarkYep. So I started with the Gauntlet project 4 years ago. I was a bit of a latecomer to it. James Wicket deserves 99% of the credit. I just came in there, closed a few bugs, and I gave him a really interesting concept change from the original project.

14:30Chris RomeoAnd just for our listeners, tell us what Gauntlet is.

14:33Apollo ClarkYeah, so Gauntlet is a continuous integration tool that works with things like TeamCity, Jenkins, CircleCI, and allows you to automate your security testing, which Sounds pretty generic, but what it does is it allows you to call command line tools, but the way that you write those scripts is using behavior-driven development, which is a very popular methodology within Scrum and Agile. So you're given— they call it given-when-then syntax. So for example, given I am logged in, when I try to change my user permissions, then I should not become admin.

15:08Mm-hmm.

15:08Apollo ClarkAnd that's literally what the exact scripts are. And then we're able to map some of those specific phrases to function calls that will run a security tool. So we can run things like a SQL injection attack, a cross-site scripting attack. You could run curl to try and attempt the privilege escalation. So it allows business owners who don't know how to write code to write these user stories and use cases and then hand that off to the security team and say, okay, here's what we want to test for. And then the security team can implement it more easily.

15:35Chris RomeoOkay, great. And I think I remember maybe a year ago or so I was talking with James and He mentioned, I mean, now at least at the time you could download it, install it and all that. But he also mentioned there might be a container version of it that you can just pull down the container and run it in Docker or something like that. Is that available today?

15:52Apollo ClarkOh yeah, I think that came out maybe 2 years ago.

15:55Chris RomeoOkay, well then maybe my time's off. Maybe it was 2 years ago. Okay, fantastic. So that's available. So you can just pull it down and deploy it and use it and do what you need as far as pushing in scripts, I'm assuming, or does it point to scripts somewhere else? How does that work when you're using a tainer?

16:11Apollo ClarkYeah, so the scripts themselves are Cucumber scripts, which again are used with behavior-driven development. So on a lot of my projects now, I'll have one folder for functional requirements, things like I, I could be able to go and create a new user account, or I can create a new book entry. So I'll have those ones which are more functional user stories, but then I'll have a whole other section for the security ones.

16:34Chris RomeoOkay.

16:34Apollo ClarkYeah, and then what Gauntlet does is a command line tool. It'll go in, it'll look for those scripts in whatever folder you point it to, and it'll say, oh, you want to do SQL injection here, you want to do that, blah blah blah, and it will grab all the scripts up. And it has a lot of pre-built steps that you can reuse. You can also write custom ones, obviously. And it will run through all the scripts, run the tests, and it generates a pass/fail. That outputs an XML file, which is really nice because you can then give that to something like Jenkins and it will generate reports and graphs.

17:03Okay.

17:03Apollo ClarkAnd it can send out email alerts to the development team. So for example, if it fails a cross-site scripting attack on, let's say, the comment page, well, you can set up Jenkins to say, oh, if this test ever fails, send the whole team out and let them know.

17:16Chris RomeoRight.

17:17Apollo ClarkSo you're getting immediate feedback versus running a tool like Qualys or Nessus, which will look for more generic security issues. Gauntlet is easily tailored to your specific application. So for example, privilege escalation, that attack is going to be completely different for every application. There's no privilege escalation attack within Qualys or Nessus, nor should there be.

17:37Chris RomeoRight, right.

17:38Apollo ClarkSo it allows you to say, you know, keep using Qualys and Nessus, but Gauntlet lets you create more application and context-specific attacks.

17:45Chris RomeoOkay, very good. And then the Gauntlet set of tools, it continues to update, I'm assuming, as well. So if you've got the latest container package, you could get the latest tools as well.

17:58Apollo ClarkYes and no. So the funny thing about that is when James first wrote this, it's all written in Ruby, is he was installing it to Ubuntu and he was pulling in common tools. So things like SQLMap, Cross-Site Scripter, Garmer, SSLWise, etc. That was his original approach. And I said, oh, you know, having gotten more into security and using Kali Linux, I said, well, what you're basically doing is you're bringing the tools to Gauntlet.

18:22Chris RomeoYes.

18:22Apollo ClarkJust bring Gauntlet to the tools.

18:24Chris RomeoOkay.

18:25Apollo ClarkAnd I said, well, just run Gauntlet within Kali and suddenly you get access to all the command line tools. And he's like, yeah, that'd be a lot easier than installing 250 tools.

18:33Chris RomeoOh, fantastic.

18:34Apollo ClarkYeah. So I kind of put that thought in his mind and I developed my own container for that too.

18:39Chris RomeoOkay.

18:40Apollo ClarkYep. So it runs Kali Linux and then it just installs Gauntlet on top. And what's nice about it is Gauntlet is effectively a wrapper around the command line. So whatever you call the command line, Gauntlet can just reuse. Now, as far as updates go, Kind of bad on me, I haven't updated in quite a few months, is we try to write adapters for the various tools. And you can still call any command line tool through Gauntlet by itself, but it's not as clean. You got to be a little more explicit with the various options. When we write adapters, we basically just make it a little easier for you to use it.

19:09Chris RomeoOkay.

19:10Yep.

19:10Chris RomeoAll right, great, great. And so any future updates or anything else going on with Gauntlet or?

19:19Apollo ClarkYeah, it's funny. James started this about 4 months ago. He's rewriting the entire thing in Golang.

19:24Chris RomeoOh, wow.

19:25Apollo ClarkOkay.

19:26Chris RomeoI'm hearing more and more things about Go that people really like it. And so that's interesting that he's looking to do that.

19:34Apollo ClarkYeah. Again, the goal for that is to give cross-OS support, is really what that is.

19:39Chris RomeoOkay. That makes sense. Makes sense. Great. Okay. Anything else going on for you right now?

19:46Apollo ClarkYep, just doing the consulting. I'm working on a project that I released about 4 months back, 5 months back in January. It's a combination of a system called Terraform, which automates the creation of cloud resources on Amazon, releasing the ELK stack with the intention being to do threat hunting.

20:04Okay. Yep.

20:05Apollo ClarkSo I've had to deploy 4 different ELK stacks at various companies, so I'm getting a little tired of you know, doing the same work over and over again. So I figured, okay, I'm gonna finally automate it for myself and open source it.

20:17Chris RomeoOkay.

20:18Apollo ClarkYep. And, uh, just, just to see if I even could, because I've never actually done this before, is saying, okay, if I can install the resources, configure them, build the OS from scratch, and automate deploying everything to Amazon, could I get it down to a single command? And I actually did.

20:32Chris RomeoWow.

20:33Apollo ClarkYeah, it basically just calls another bash script, but the point is I got down to one command. So you can do it. Fantastic.

20:38Chris RomeoOkay. Well, it's been great talking to you, Apollo. I know we've been talking about trying to do this for a while, so finally we were at the same conference and we get a chance to talk. Any last thoughts or comments for our listeners?

20:50Apollo ClarkDevOps is coming. Just know that. You're not going to hide from it. There's a lot of confusion around DevOps, but I think the confusion is misplaced. People are looking for a specific checklist of things to do. But that's the whole point. DevOps isn't about that. It's about saying, what do you value? What are you trying to accomplish? And how can you get everyone on board, all the teams working together and communicating together and managing themselves effectively? That's really the heart of DevOps. It's about that cultural change of saying, we're not just ops, we're not just dev, we're not just security, we're not just IT. We're looking at things more holistically. And I'm working with a large enterprise right now, a few in the past years, and it's hard. It really is. But when you start going down that pathway, you start realizing that there's actually a lot of common ground between departments. And a lot of the things that help you individually within your projects and your teams actually does benefit the rest of the company in some larger way. And when you find those bridges, it just makes everything a lot easier for everyone. So I think security should focus on saying, we want to be secure, which is good, but how does that overlap with what developers are trying to do? How does it overlap with what ops is trying to do? How does it overlap with what sales is trying to do?

21:59Yeah.

21:59Apollo ClarkYou know, when you have a secure system, it's easily deployed, easily updated, and that, that's developers and ops right there. They would love that. And again, it helps create new sales opportunities. When you have a hardened system, it's pretty rare to find a product that's genuinely end-to-end encrypted, and that is a huge sales opportunity.

22:16Chris RomeoAbsolutely.

22:17Apollo ClarkAbsolutely.

22:18Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.