Mohammed Imran -- Back to the Lab Again with a DevOps

0:00Chris RomeoHey folks, season 4, episode 8 of the Application Security Podcast. On this episode, I'm joined by Mohammed Imran, and the title of this episode is Back to the Lab Again with DevOps. So Mohammed does classes on securing DevOps, and he's also the project lead for this new really cool project called DevSecOps Studio that's part of OWASP. And so we dive into all these things and so much more. Hope you enjoy. The Application Security Podcast. Here we go. Hey folks, we are once again at AppSec EU this week, and I am joined by Imran, who is at AppSec EU teaching a training class on DevSecOps. And so we're going to get into many different areas of DevSecOps in the OWASP universe. But first, Imran, why don't you tell us what is your security origin story? How'd you get started in AppSec?

1:23Mohammed ImranYeah, first of all, uh, thank you for the opportunity. Uh, so I'm Imran, as Chris just mentioned, and I've been doing more of offensive security from close to a decade now. And then my journey started in this, uh, from an open source conference, uh, back in India.

1:44Chris RomeoOkay.

1:44Mohammed ImranAnd then I was just helping someone build a distribution which is specific to one of the Indian languages. I was helping them to build that.

1:52Chris RomeoThis is Linux distribution?

1:54Mohammed ImranYes.

1:54Okay.

1:54Mohammed ImranBased out of, uh, it's a fork of Debian.

1:58Okay.

1:59Mohammed ImranAnd then, uh, one of the presenters there was a guy called Manu Zakaria, and then he showed us how you could use SQL injection to get a— get hold of credit cards from a, you know, from a website, a typical, you know, normal SQL injection, airbase SQL injection. And that led me to dig deeper into security, how, you know, what you could do. And but I did have some experience in terms of tweaking cartridge, if you remember, game cartridges. So I used to tweak them to make sure, you know, without keys as well, or it works on some other console as well.

2:39Chris RomeoOkay.

2:39Mohammed ImranYeah, so I did have some experience, but this was quite interesting to me and Oh wow, this is something I should look into. And that led to me going deeper into it. And then I used to hang a lot in IRC, on Freenode, and then I was part of a bunch of those, a couple of groups. And that led to me getting a job in one of a company from right after my school, immediately after school. Okay.

3:07Chris RomeoAnd so you started on the offensive side as a pentester then?

3:10Yes.

3:10Chris RomeoOkay, and then you spent most of your time— are you still a pentester today?

3:14Mohammed ImranOccasionally.

3:15Chris RomeoOkay.

3:16Mohammed ImranYeah, so I, I started as pentester doing mostly vulnerability assessment, pentesting, code review of firewalls, internet devices, network devices. And then after a couple of years, and then I looked into more like product security aspects of it. Like, I'm definitely good at getting into networks, what can I do to prevent people from getting into it.

3:40Chris RomeoYeah, I mean, it seems like in our industry we have such a focus on offensive. Nobody likes— maybe, I don't know, maybe defensive isn't cool. I don't know, but everybody wants to break stuff. Nobody wants to put the pieces back together. So it's great to hear that you've made the transition now from your thinking to say, I can break it, now let's go turn around and say, how do we prevent it from being broken?

4:02Mohammed ImranRight, and to be honest, defense is more challenging than offense.

4:07Chris RomeoIt's always, uh, it always is. The defender has to be right every time, whereas the offensive person has to be right just one time. That's— I didn't come up with that, that was Richard Belichick. That's a quote from a, a badly copied quote from Richard Belichick, but that was, that's what he, that's what he was very clear on, that you got to be right as a defender all the time, and that could cause people to lose hope, but there is hope out there that we can create defenses and we can make this possible. So you did this class here at EU talking about DevSecOps, a 3-day class. I'm just curious, what do you even cover in 3 days? Seems like that's a class you could teach for 365 days.

4:49Mohammed ImranYeah, exactly.

4:50Chris RomeoAnd we wouldn't ever be done. We'd still have, oh well, we still got half more of the content to cover. So what did you actually teach over 3 days?

4:59Mohammed ImranOkay, so before I go into what I teach, let me give a couple of background details of it, right?

5:05Chris RomeoYeah, sure.

5:06Mohammed ImranSo as I mentioned, I switched from offensive side of security to the defensive side of security, and in doing so, I realized quickly that the ratio between DevOps and security is pretty skewed, right? For example, and I'm being very optimistic here, which is for every 100 developers, you have 1 security person. Or for every 10 ops, you have 1 security person. And this is, this is just a, you know, usually quoted ratio, but it's worse than that. Usually I see like 500 to 1 security engineer.

5:39Chris RomeoYes.

5:39Mohammed ImranYou know, 200 to 1 security engineer. But then, as we can see, that we are outnumbered, and no matter what we do, we cannot do security in a very you know, do security up to a satisfiable standard, right? Up to like where you feel, yes, I'm confident that I did a good job and it's good enough, right?

6:02Yep.

6:02Mohammed ImranSo that led to me looking into, hey, how are ops solving? Because their ratio seems to be pretty messed up as well, 100 to 10. And then I digged into how they do, you know, infrastructure as code, how they do immutable code. And that led to me digging more into DevSecOps and how security can fit into this amazing revolution we call DevOps these days.

6:30Chris RomeoYeah, so you learned some of your perspective from what the DevOps people were doing natively before security got involved?

6:37Yeah. Okay.

6:38Mohammed ImranAnd I was blown by the tooling they have. Like, and the first time I looked at how they do configuration management or infrastructure The first thing which struck me was that, holy shit, security is like 100 years behind Dev and Ops. And then that led to further looking into further tools, like for example, Docker is there, which is pretty amazing, lightweight, easy to get started, and you can do tons of amazing stuff, right? And that, because of that, instead of me doing one project before I had automation, I could automate my entire scanning for the entire organization in, in a week or so pretty easily. And I could create some generic components which everyone can go and then put into their pipelines without me explaining them what it does, how it does. I just say, hey, pull the Docker container and then just run it.

7:32Chris RomeoOkay.

7:32Mohammed ImranRight, those are the 2 things. And obviously the techniques you use to do that is different. For example, if you want to do static analysis, you might be using Bandit, which is an open source tool for Python. Or if you, if you are running Ruby on Rails code, you might use something like Brickman. And similarly, every language has such, right? Golang has GAS, and then Java has Find Security Bugs. So every language has that.

7:57Chris RomeoYeah.

7:57Mohammed ImranSo that's what I teach in my class. So it doesn't matter which language you are using, but the techniques to run a static analysis or a dynamic analysis or convert your hardening scripts into configuration management so that you can do hardening as soon as the code is committed into your repository. So instead of you doing off the, you know, off the CI/CD, on a regular interval you do it in the CI/CD.

8:23Chris RomeoYep.

8:24Mohammed ImranSo you get your hardening for free. And not only that, with concepts like immutable infrastructure, which is what Docker is all about, you, you don't harden anymore. What you do if you have a patch, what you do is that you take out your existing infrastructure down and then you bring new infrastructure up. And we were able to do this is because of this amazing technology like infrastructure as code, which includes the platform like Amazon, GCP, Azure.

8:52Yeah.

8:53Mohammed ImranOr if you're running internally, something like OpenShift will help you. So we need to have 3 things in place to do this. One is a platform, and then you need to have some way of converting, hey, how much memory do I have to put in this? How many— how much RAM I have to put in? That as a definition in a file, mostly a YAML file, right? DevOps loves YAML files.

9:15Chris RomeoYeah.

9:15Mohammed ImranSo in a YAML file, and then you need— once you spin up a machine, you might be willing to configure it according to your spec. You might say, hey, you know what, you have to change your password every 90 days because you are PCI compliant, right? You can tweak it and twist it however you want. And no matter what compliance you have to be compliant to, you can tweak it. And not only you can tweak it, you can put it in the same repository where developers are coding.

9:44Got it.

9:45Mohammed ImranAnd that gives you tremendous visibility. And even developers, without you saying anything, they will look into the build and say, hey, Imran, I see that you are running Bandit or Brickman. Why are we doing this? I see that we have 10 tools in this and I have 10 bugs, high severity bugs. Maybe I should go look at it. I'm not filling a build, so to say. Yeah, they're just curious because let's say if someone is in your house, you would be curious where he— where this person is from, what he's doing, why is he here.

10:15Chris RomeoYeah, so that takes you— so we kind of talked about static perspective here. Dynamic and vulnerability scanning, do you fit those into the same kind of frameworks and things?

10:28Mohammed ImranRight, right. So I do dynamic analysis as well, but by nature of these tools, it's very difficult to run them effectively, meaning that even for, let's say, you have about 100 endpoints, 100 web pages, and take an example that each of them have like 10 input points like a username, password, and all that. And if you do the math, it would at least take you a couple of hours to effectively fuzz all those endpoints to do all the scanning. But a rule which all security professionals should abide by in DevSecOps is that anything which takes more than 5 to 10 minutes shouldn't be part of your CI/CD.

11:08Yeah.

11:09Mohammed ImranSo you should be running it somewhere else as a scheduled job.

11:12Chris RomeoYep.

11:13Mohammed ImranSo what we do in SteelRun is we use something like ZapScanner, but we run only baseline scans, meaning which at least gives you a good effective coverage in terms of finding low-hanging fruit. Like for example, you have configured SSL but your cookies are not marked with secure flag.

11:34Chris RomeoYep.

11:35Mohammed ImranRight, so we take our guys from doing static static analysis, dynamic analysis, and then hardening, how you can automatically harden it, creating golden images, and then compliance as code. And before we do this, we have to then teach them, hey, how you can push code to Git. And unfortunately or fortunately, security guys don't know how to push code to a Git repository. Many of them don't.

12:02Chris RomeoAnd some of that, the problem there is security people didn't become developers, they didn't come from development.

12:07Yes.

12:07Chris RomeoSo, I mean, for example, in my story, I came from the sysadmin side. So I only learned to code in Java because I was working on a cool project to build an event correlator. We're going way back before the products existed for event correlation. I had a chance to build one with a friend of mine at work, and so we learned Java to do it. But so a lot of people are coming to security like me who are sysadmin-minded and don't code. And that's why I always recommend when people say, how do I get into cybersecurity? Well, step 2 or 3, learn a programming language. Yeah, I don't care which one.

12:40Mohammed ImranYeah, great suggestion. I would rather— I would definitely do that as well. And then, so because now I would say a couple of years down the line, you will not have a specific InfoSec engineer who does only only SOC operations or network access control, or, you know, network kind of a security, traditional InfoSec role. Uh, we will not have something like AppSec engineer or compliance person, analyst, security analyst, but we'll just have security engineer. Reason being, now everything— software is just eating the world, right? We all heard that term. And then everything is code now. Infrastructure is code, right?

13:20Yeah.

13:20Mohammed ImranYour compliance is becoming So if not now, eventually that's where we are heading.

13:26Chris RomeoYeah, it's probably 10 years in the future or so because you think of, you know, I spent 10 years of my career at Cisco. And Cisco is in some places at the forefront of DevOps and in some places like, for teams that are still building software that runs on hardware boxes, on metal boxes. You don't commit code 20 times a day to a metal box running somewhere. You can try to get that close, but big ISPs don't like to upgrade code 20 times a day. They like to upgrade code once every 6 months, and they test the next version for 6 months to go. But you're right. I mean, over time, we will slowly get to the point where DevOps takes over, completely takes over the I mean, if anybody was delivering a web app in a non-DevOps mode right now, then shame on them because web apps are designed— I mean, they are primed to operate in this fast mode. So what other tools do you think are crucial in the DevSecOps world? What other tools do people need to know if they say that they're DevSecOps-wise?

14:40Mohammed ImranYeah, so I would say you definitely have to learn a little bit about Git, like how do you add a file to it, how do you commit it. That's central to what we do, like everything as code philosophy. That's definitely one of them. And then the other tool you have to learn is how a CI/CD system works.

14:58Chris RomeoOkay, which— so I mean, is there a particular one that you're teaching and you're focusing on?

15:04Mohammed ImranWe are primarily focusing on GitLab because it's free and open source and you can share with others.

15:09Chris RomeoOkay, now is GitLab— can you do similar things like Jenkins and Kubernetes? Those are all synonyms for each other in this world.

15:19Mohammed ImranBut you can do it much nicer and easier way in GitLab.

15:23Chris RomeoIn GitLab?

15:23Mohammed ImranYeah, it's because it's just simple YAML file.

15:26Chris RomeoOkay.

15:26Mohammed ImranInstead of you clicking a bunch of buttons, it's just one file with simple steps. Like you say, hey, I once you are done with your deployment, I want you to run this script. And the way you say is script colon and hyphen whatever command, whatever you run in your CLI. Let's say you want to run Nmap scan once you deploy to make sure that the ports are not exposed. You just say nmap -ox and save a file, the output of the file, and then give an IP address. Okay, it's very simple to use.

15:54Chris RomeoSo GitLab is a good foundation for those that are starting out Yeah. In this world of DevOps? Because I mean, it's, it's possible for people to learn about DevOps on their own. They can just set up an environment and go.

16:06Mohammed ImranI learned on my own. No one taught me.

16:08Chris RomeoOkay, so GitLab is where— is a good place to start though for people?

16:11Mohammed ImranYeah, yeah. Okay, pretty simple to use, but you also have to understand the concepts behind what CI/CD is trying to achieve and why, why it exists, right? And once you do that, then you should learn configuration management system. For example, Ansible, which is again open source, is pretty easy to understand and get started with. Okay. And once you do that, and bunch of our traditional security tools, for example, static analysis, dynamic analysis tools, right? All those tools. Now, if you can do these 4 things, you're 90% of your job is done by just these tools.

16:52Chris RomeoAfter the break, we get into what is DevSecOps Studio and why would you create it. The Application Security Podcast operates with support from Security Journey. A security belt program provides the 3 pillars of successful AppSec training: learning, application, and experience. Visit us on the web at www.securityjourney.com to learn how you can teach and empower your developers using a new kind of security training. Now we'll pick back up with Mohammed asking the question, what is DevSecOps Studio and why would you create it?

17:35Mohammed ImranOne of the reasons why I created DevSecOps Studio was that I realized we don't understand as a security, we don't understand how GitLab can be used, a CI/CD system can be used, or how a configuration management system like Ansible is used, right? If you do these, then you can embed security as part of CI/CD system, and your mean time to remediate or fix a vulnerability can go from, let's say, 6 months to just a couple of minutes because developers have immediate feedback.

18:06Okay.

18:06Mohammed ImranAnd they know what's going on. And if you want to, you can fail a build, you find a vulnerability, vulnerability, a serious vulnerability, a critical vulnerability. Based on, again, criteria you can decide, whatever, based on your company policies.

18:18Chris RomeoOkay.

18:19Mohammed ImranNow, and then I showed this to my boss saying that, hey, see this, I did this, and our mean time to remediate went from 6 months to just a couple of minutes, which was pretty impressive.

18:30Chris RomeoYeah, and then you should get a raise or promotion at that point.

18:33Mohammed ImranYeah, definitely.

18:34Chris RomeoI think so. I vote yes.

18:35Mohammed ImranYes. And then, uh, and then he asked me, hey, why don't you teach our guys, right? And then I thought, hey, okay, let's go ahead and do it. And then so maybe there is already an OWASP project which does this. I Googled around it.

18:49Chris RomeoFamous last words. Maybe there's an OWASP project that does this.

18:52Yeah, exactly.

18:53Mohammed ImranAnd there was none. So then went ahead and created all of that. So what I do in the project is that we simulate an entire CI/CD pipeline Okay, so this is virtual—

19:02Chris Romeoyou're giving me virtual machines that I can use?

19:05Yeah.

19:06Chris RomeoSo I don't have to set all these things up. You're going to give me a reference architecture implementation.

19:11Mohammed ImranRight, exactly, with all these tools. So you have all CI/CD tools in it, and at least it covers at least one category of like Git server, CI/CD server, configuration management. Not only that, we also install all the important tools which you as a security professional You use Nmap, you use Gauntlet, you use BDD Security, and all of the tools. Even we have Metasploit installed.

19:35Chris RomeoOkay, just in case.

19:36Mohammed ImranYou name it, yes. And then we have something like Inspect, which is like a compliance as code tool, so you can do that. And then using Ansible, there's an open source project called DevSecHardening.

19:48Chris RomeoOkay.

19:49Mohammed ImranWhat it does is it gives you ready-made scripts you can use to harden your Ubuntu machine. Red Hat machine, and all of that.

19:56Chris RomeoOkay.

19:56Mohammed ImranRight, not only that, it also gives you scripts to check for compliance using Inspect. All of those are there in your DevSecOps Studio, so it's easy to get started. So if I have to summarize, I would say DevSecOps Studio is designed to create a platform, a distribution, a virtual machine for you to learn and teach DevSecOps principles.

20:20Chris RomeoNow, did you use this in your class that you taught here?

20:22Mohammed ImranYes, exactly the same thing. And it's open source and it's free, so you can download it and just run. And we are using the same concepts you would use in your day-to-day job to create the environment as well. It's not an ISO. You just run 2 commands. You say vagrant up and it's up. It will create— it uses infrastructure as code, the technique which DevOps use, and So you're actually practicing what you preach here. Yeah, exactly.

20:49Chris RomeoYou're not just giving me a bunch of ISO files that have already been pre-configured. It's actually going to use all the tools that you tell people to use to build up this environment. And so then I, as a user of DevSecOps Studio, I can try all these tools out. I can figure out, I can commit code, I can make changes to infrastructure services and things. And just, it's a playground for me to figure out how these tools work and to try and maybe break stuff and see what happens.

21:20Mohammed ImranOkay. And more importantly, embed security as part of that, right? This is not a DevOps distribution, rather DevSecOps.

21:28Chris RomeoYeah, so you got security baked into that as well to let people see how that works. Yes.

21:33Mohammed ImranOkay. And oftentimes you have to show people, hey, this is how you do it. then they realize, oh yeah, then now I understand.

21:39Chris RomeoYeah, right. No, I think it's— I think that's a valid teaching technique to provide all the pieces and then let people start. Because I mean, let's face it, that is a very daunting task for somebody who's new to DevOps, just to set up one of the pieces we talked about here. If you're not— now, I mean, I told you, I mean, I came from the sysadmin background, so I enjoy, you know, I would think that would be fun to put those pieces together. But most people are going to look at that and say, this is terrible trying to get Ansible installed and working on my VM somewhere and then getting it to talk to the other pieces and things. So yeah, I think it's a great idea about pulling it all together. I love the fact that you have security built into this so that there's really no option— it's not an optional thing, it's built into this, this process that you have. And then I guess if we come back around, the other project you work on, DevSlop, Now it's all about a broken application that uses— so does DevSlop use any of the pieces from DevSecOps Studio, or are these completely separate?

22:40Mohammed ImranOkay, so DevSlop started as a project to build vulnerable microservices so we can teach people how to do it in a modern infrastructure, right? And modern infrastructure has CI/CD built into it, so you can put DevSlop into DevSecOps Studio.

22:55Chris RomeoOh, okay. You can run it in there.

22:56Mohammed ImranYes, exactly. And then teach people, hey, you know, if it's— even if it's a monolith or a microservice, the pipeline is going to be same. There are obviously some differences in terms of how you deploy it, right? But the thing is, it has to go through a pipeline, right?

23:12Chris RomeoIs there any way to include like a reference architecture for— I know we have like Azure, we have AWS, we have Google's cloud. Platform. Is there any way to connect that into kind of the DevSlop or DevSecOps Studio hierarchy?

23:31Mohammed ImranYeah, so no matter which stack you're using or, you know, cloud provider you're using, the basics and fundamentals are same. For example, you have a service like CodeCommit in AWS, right, which does a job like GitLab. which is like a version control system.

23:49Chris RomeoIt's AWS's version of it.

23:50Mohammed ImranVersion of it. Then also they have like CodeDeploy, which does your continuous deployment.

23:55Chris RomeoOkay.

23:55Mohammed ImranRight, even though the names are different and the way they work are a little bit different syntactically and then by design, but the concept is same. So, and that's what we do in our class as well. We teach tools which you can use on both on-premise setup, like for example your financial industry, you cannot use SaaS by compliance reasons, because of compliance reasons. Then you can use the same set of tools not only to do on-premise but also on cloud. Because fundamentally you're just— let's say you want to start an EC2 machine. Once you start it, it's fundamentally the same. You just log in using SSH key inside.

24:29Chris RomeoYeah.

24:30Mohammed ImranJust like a bare metal box. But the thing which we're doing before is important, which is to spin it up using APIs. But fundamentals remain same.

24:40Chris RomeoFundamentals remain the same.

24:41Yeah.

24:41Chris RomeoOkay, so where, where do you recommend somebody go to get started in this? I mean, you've got DevSecOps Studio. I mean, obviously not everybody can get to your class, but, um, you know, if, if you offer that again, that obviously sounds like it'd be a good place for people to start. What about those people who won't get to your class? Where do they, where do they get started in this world of DevSecOps Studio? What do they do first?

25:02Mohammed ImranYeah, uh, go to, you know, DevSecOps DevSecOps Studio project, there you have a wiki which explains you how all of this is set up and then how you can embed security as part of that.

25:13Chris RomeoOkay, and is the, the integration of DevSlop into DevSecOps Studio, is that documented or is that something that people have to figure out?

25:20Mohammed ImranYeah, yeah, it is documented. Okay, so I can find a lot of pictures and then you can look at it.

25:25Chris RomeoI like pictures. Pictures are, you know, they make it possible for me to understand complicated things. that y'all are explaining to me. So, okay, so Imran, great. Thanks for sharing this. I didn't even know before we sat down, I told you I didn't even know DevSecOps Studio existed as a thing.

25:40Uh-huh.

25:40Chris RomeoSo I'm glad to hear about it. It sounds like a very good project for people to learn. I love these type of environments where you help people build it up and then they can do whatever they want and play with it and push it and poke all the buttons and see what happens. So I think you're going in the right direction there, and so thanks for sharing it with us, and enjoy the rest of your conference.

26:02Mohammed ImranThank you for having me, and I appreciate your time.

26:04Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast. Podcast, or on the web at www.appsecpodcast.org.