Mazin Ahmed -- Terraform Security

0:00Chris RomeoMazin Ahmed is a security engineer that specializes in AppSec and offensive security. He's passionate about information security and has previously found vulnerabilities in Facebook, Twitter, LinkedIn, and Oracle, just to name a few. Mazin is the developer of several popular open-source security tools that have been integrated into security testing frameworks and distributions. He also built fullhunt.io, a next-generation continuous attack surface security platform. And he's passionate about cloud security and has done dozens and continues to do dozens of experiments in the cloud security world. Mazin joins us to introduce Infrastructure as Code and Terraform, and to discuss the security benefits of IaC and what those benefits are for your cloud environments. We hope you enjoy this conversation with Mazin Ahmed. You're about to listen to AppSec Podcast. When you're done with this, be sure to check out our other show, High Five. Hey folks, welcome to this episode of the Application Security Podcast. This is Chris Romeo. I am the CEO of Security Journey and also co-host of the podcast. I'm joined by Robert Hurlbut. Hey Robert, how's it going today?

1:12Robert HurlbutHey Chris. Yeah, it's Robert, Threat Modeling Architect, and looking forward to another topic in application security today.

1:19Chris RomeoYeah, we're going to get into something that I think a lot of people feel like they understand, And we're here to challenge the status quo and say, you probably don't understand infrastructure as code. But hopefully, by the end of this session, you'll understand infrastructure as code. You'll understand the security benefits of it, which are really the thing that interests us from an AppSec perspective. So, our guest today is Mazin. And Mazin, we always start with security origin story. Our guests are on the edge of their seat. They want to know, how did you get into application security?

1:55Mazin AhmedFirst of all, thank you very much, Chris and Robert. I'm really excited to be here. And let's start with my story. I started in security when I was in high school, I think around 14 or 15 years old. When I was 14 or 15 years old, it was only driven by curiosity, how things work, how things are supposed to work. Supposed to work and how things are not supposed to work. And at that time, I was just reading online in forums, taking courses, anything that I can find to read and learn, and doing my normal high school days. And, uh, and after that, after a couple of months, I learned about Bugcrowd. It's a company that our platform that hosts bug bounty programs. At that time it was quite new, but now it's quite popular. And I started doing bug bounty hunting at that time and I was really focused around in 2014 or 2015. I was really focused on that. I used to be in the top 10 of Bugcrowd. And after that, I started doing penetration testing projects and AppSec projects for customers around the world. And then things went from there.

3:24Chris RomeoSo, were you doing bug bounties while you were in high school still?

3:27Mazin AhmedYeah.

3:28Chris RomeoOkay. There's a whole story to be told right there. It's like I'm envisioning, it's like a comic book. It's the superhero. By day, Peter Parker went to school, by night he was Spider-Man. That's kind of what I'm envisioning here. So how did you learn the things that you needed to know as a high school student to be able to be successful in bug bounty?

3:50Mazin AhmedThis was really hard. The internet has so many things that you can use to learn, but being self-taught was really hard because you wouldn't know what to learn and what's actually useful. And is this knowledge that you're reading correct or valid or applicable. The only way that, like, until you find a mentor— when I started, I did not have a mentor, which was bad, of course. And I totally recommend if you are starting security, try to find a mentor. I was just reading everything that I can and try to apply it on labs or different setups. And then I would be spending or wasting actually hours just trying things that appears to be non-useful until I find something that is useful and I build up from there.

4:45Chris RomeoVery cool. No, that's neat. It's neat to hear. And for those, I don't know if we have any teenage listeners that are in high school on the Application Security Podcast. I certainly hope we do. But if we do, or if people listening have kids in high school, Like, they can become security professionals too. I mean, we always talk about how there's this giant void of security talent, and we can all argue about what the total number of people is, but we know there is a void, right, in the amount of security talent that we have in our industry. And we need to get more young people that make their way and become application security professionals. So, that's really neat though, to hear your transition from high schooler to, and bug bounty person to now having a career in application security.

5:30Robert HurlbutSo curious, Mason, that you have mentioned pen tests and working on testing applications and so forth. But what I also understand is that you've sort of transitioned from internal pen tests to cloud security pen tests. So tell us a little bit about that, some of your experiences with pen testing in general, but also What made you switch and how did that sort of happen for you that now that's probably the most important that you're looking at right now?

6:01Mazin AhmedYep, for sure. When I started, my focus was in web application security. It's still something that I'm interested in and I was solely doing web application security penetration tests and assessments. And from there, I was interested in how networking works, how, uh, how would you break into a network, how would you compromise the entire infrastructure without really exploiting remote code execution on web application. And that's why where I started learning more about internal penetration testing and how would you do internal pen test from zero to the end. And I was doing this for a while for different clients here until I reached the moment that I found that there is something that is way more dangerous than inter— like the security of internal networks is cloud security. Cloud security is a thing that Companies are just starting to notice how important is it because of all of the breaches that happened in the past year or so after all of the pandemic and transitioning into remote work environments. And let's say that you may have the most secure application running on your infrastructure, but if you misconfigure your cloud environment, you're done. You're breached, and next week you would be in the news. Do you—

7:50Robert Hurlbutjust a follow-up question. When you said that, you know, more companies obviously are going to the cloud and starting to look at cloud security, do you find that a number of those companies initially think that it's all taken care of? for them that they just think, well, I moved to the cloud, it'll take care of that for me. It's already secure. It's all going to be done for me, essentially. That's the benefit or something like that. Do you find that sometimes in terms of approaches or thoughts about cloud or cloud security?

8:23Mazin AhmedYeah, 100%. And there are, like, 2 different views. One view is, the entire security is being managed by your cloud provider, whether it's AWS, Azure, or Google Cloud. And the second, like, view is, yes, the security is done by, like, by cloud, by the cloud provider, but we have to do minimal security checks for, I don't know, S3 or something like that.

8:59Robert HurlbutYeah.

8:59Mazin Ahmedjust the most minimal thing that can be done. But the reality is it's a lot of the things, there are a lot of things that you would have to manage and secure and scan, monitor, and configure just to be in a typical secure environment, not the highest security.

9:22Chris RomeoSo when you think cloud security issues, I'm curious as a pentester, What are your top few types of issues that you see all the time? Because we always hear about, you know, public S3 buckets or public storage buckets. That's one that's always on people's top 3. But what else would you add on that list if you had to give us a couple of more outside of storage bucket public nature?

9:47Mazin AhmedYeah, for sure. Okay. Let's talk about a scenario where I'm not I'm an external attacker. I do not have access to your cloud account or to your network, and I'm assigned to attack your network or your cloud environment. One attack vector that is like, I'm seeing a lot of success on it for the past years are server-side request forgery, SSRFs. And the, like, the way that it works is I would be adding an input or injecting an input or inserting an input into an application that is processing a URL or a value that is fetching resources from other places. And if this is not being checked correctly, then a person can insert a URL that would redirect or fetch resources from the MITRE CSIX server on your cloud provider, like in AWS. Let's take AWS as an example. And when this happens, you would be able to assume roles that this resource is having and then gain access as the person or as the resource that is being assumed. This is one scenario that is quite common.

11:16Robert HurlbutYeah.

11:17Chris RomeoAnd that's SSRF. If you think the OWASP Top 10 2021, the final edition has dropped now. SSRF stays or is now on the number 10 spot on the list. So even though the data said— the data, I think, from what I understood, didn't show an overwhelming number, we still know that attackers are using SSRF in a lot of different environments for great success right now. So I'm glad that that made it to the OWASP Top 10 now as a potential issue.

11:47Mazin AhmedYep. I think it was a part of the injection category, and then this year they moved it into its own separate category, which is great because with cloud, this is a major attack vector when you're coming from an AppSec into CloudSec perspective. Yeah.

12:06Chris RomeoSo let's change gears a little bit, but we're not really changing gears that much because IaC, infrastructure as code, does have a direct mitigating factor on what's going to be found in your cloud pen test. Because if you get infrastructure as code right, you're going to lock your stuff down. But let's define infrastructure as code. I'm fond of saying, explain it to me like I'm a 5th grader for infrastructure as code.

12:33Mazin AhmedYeah, for sure. How do I start?

12:39Chris RomeoOkay.

12:40Mazin AhmedBack in the days, cloud providers and platform providers had their platform or console that people would go in and they would deploy their EC2s, their VMs, their, any other resources that they would like. And they would just go in. If they need to deploy 10 S3 buckets, they would do it one by one, or they would write a script or something like that to do that. It's not really as neat or maintainable, but typically they would be wasting time in, and— Oh, without having real supervision by doing all of the clicking on the UI. And then what came to the scene is IaC, like infrastructure as code. And what it makes— made the difference is you can write all of the configurations and the requirements that you would like instead of just writing it on a note. You can write it on actual code. And then this code would go in and would be deployed the same way that you configure it. So if you would like to deploy a VM with a specific security group and then attach it to X and Y and Z and allow it to have 1 and 2 and 3, you can just write everything in as infrastructure as code. And then you can have it being reviewed by your teammates, and you can have it like, as far as you have it as like written as code, you can also have it scanned automatically in your CI pipeline. So many things that are being opened now with using Infrastructure as Code compared to the really old way where you would write all of your requirements in a document, and then you would send it to the infrastructure team, and then they would read it, and they would have a meeting. This is all changing because of infrastructure as code and everyone.

14:46Chris RomeoSo, we call it infrastructure as code, but is it better to call it— I mean, is it really— it's really infrastructure as config, right? Because isn't it YAML files? Isn't infrastructure as code, it's a series of YAML files that define an environment that's going to be built. And then there's an interpreter that's interpreting those YAML files and then doing particular actions on a cloud provider. So I think infrastructure as code is a catchier word, but is it really code? Because like the YAML file isn't executable, it's not compilable, it's more like it's an interpreted— am I going down the right road here or am I completely missing the boat?

15:30Mazin AhmedWell, you're right in one part, and but, uh, like, there are, um, like, cloud, um, cloud orchestrators that are, uh, totally relying on YAML and configurations that are static, and there are provider orchestrators that take, uh, the code that you write or the config or the file in general And then they would dynamically parse it, validate it, and execute anything that should be executed to get into the final stage of deploying.

16:07Robert HurlbutOkay.

16:10Mazin AhmedSo it's like some of them are config and some of them are code. But yeah, you're right on the part of infrastructure as code as a word is much fancier as comparing to config.

16:23Robert HurlbutWell, I think it also sounds like it helps people understand who's doing this, right? It could be a coder, it could be— well, in a DevOps type of— it could be a developer, it could be operations, it could be a combination of the two. But essentially, it's some kind of code or some kind of script or some kind of— and config as well and so forth. That's sort of managing this is what it sounds like to me. So very cool.

16:48Mazin AhmedYeah.

16:48Robert HurlbutVery cool. What are the security benefits of IaC?

16:54Mazin AhmedYeah, for sure. What I really like is once you have, let's say that you have your entire environment be using IaC for like infrastructure deployment, what you can do now is you can have all of that archived through Git, like, and have it uploaded to GitHub or GitLab or the Git provider that you're using. Let's take GitHub as an example. And once you have that, then you have a full archive of everything that happens into your cloud environment. By who, who reviewed it, where is the PR that changed it, and what were the points of security or considerations that were there. This is one thing, like manual review. And of course, the other part of having it archived. This third part is once you have it on GitHub, you can add it to your CI pipeline and you can add the CI pipeline to scan it for security and compliance checks. No, please do not apply a change or an addition of an S3 bucket if it does not have versioning enabled, or if it does not have, or if it's being public, please do not expose it or do not apply it. And all of these things, you can write like 20, 40, or 100 rules, and then you can have it checked every single time someone pushes code on a feature change so that they want to deploy it. Not only that, you can use ready-made tools, uh, like I'm talking about Terraform now, for tfsec and Checkov. And I think there is another scanner that I forgot its name, but tfsec and Checkov are the most popular 2. And these tools have already built-in modules that are really, uh, like checks that are really useful out of the box. And once you have them, you have a full CI pipeline that is scanning in addition to your security team that is helping you.

19:17Robert HurlbutYeah.

19:17Chris RomeoWhen I think security benefits, like if I step back and say, from the 10,000-foot view, what is IaC in general providing for me from a security perspective? I'm from the days of old when we used to have hardening guides. And we used to have like a checklist. Okay, go onto the Unix host and make sure the file permissions in this directory are set to this. And then make sure these accounts aren't able to log, like, you know, in those days. And what happened is inadvertently somebody would miss a step, not even on purpose, it was an accident. There were 129,000 steps we had to do manually. And so somebody would miss something and then there'd be a machine that would get compromised and we'd go back and do an investigation and say, oh, we forgot.

20:01Robert HurlbutYeah.

20:02Chris Romeogot to do step 75. And so, when I think about the security benefits of IaC and, you know, as we get into Terraform here, it's we can pre-harden, we can define what we want from a hardened configuration and let the tool deploy those resources in the hardened state. And you don't have to have a person that's going through and doing those checklists. You know, I kind of miss the good old days of going through 1,000 steps and You know, what'd you do today? Well, I hardened 1/4 of a Unix machine because I still have another 750 steps to do. So that's what I see as the big benefit.

20:37Robert HurlbutDo you really miss those, Chris?

20:39Chris RomeoI don't. I do not at all. I would much rather not be sitting there kind of typing all day long and then trying to, you know, and then you got in trouble when you're like, oh, I remember all the steps now. I don't need the guide. Oh, whoa, look, we got another host compromised. Who hardened this one? Well, it wasn't me.

21:00Robert HurlbutSo, so, uh, Chris mentioned Terraform. Uh, tell us about that. How does, uh, Terraform, uh, play into IaC?

21:07Mazin AhmedUm, Terraform is the most popular cloud orchestrator today in today's world. I think they have millions, uh, of downloads being used by Like, I don't know how many companies, but, uh, I'm pretty sure like all of the companies that are focused on infrastructure as code, the majority of them are using Terraform. And the architecture, the way that Terraform works, how smooth you can write plugins, you can write the modules, you can write checks, or something that it's not really seen in other orchestrators. That's why I'm really focused on Terraform.

21:51Chris RomeoAnd when I think of Terraform, one of the ways that somebody was explaining to me, one of the big benefits of Terraform in general is that it is cloud neutral. So, you have some of the same capabilities in AWS, Azure, GCP, but Terraform allows you to potentially say, hey, I want to move my infrastructure from AWS to Azure. I use my same Terraform and—

22:18Mazin AhmedYeah.

22:19Chris RomeoI swap out whatever's running underneath it and I can build the same environments. Probably got to tweak it a little bit, but I can come pretty close to not having to redesign my whole architecture. I get some of that neutrality from a cloud provider perspective so I can bounce around to another provider without feeling like, oh, I couldn't leave my provider because They're so entrenched in how I do everything, it would take me years to move to another provider.

22:43Mazin AhmedYep, for sure. And speaking to this point, I think before Terraform, the popular one was AWS CloudFormation. And this allowed you to only deploy, of course, in AWS. It's still a good product, but when you have a single tool that you can write the configurations and resources for AWS and Azure and GCP and another, I think, 300 providers. Uh, they go from the smallest— the biggest provider to the smallest ones, and you can have them all covered within your code. I even saw one company that is doing, uh, GitHub management, like for, uh, for creating repositories and managing applications and everything through Terraform.

23:37Chris RomeoWhat are the attack vectors on Terraform environments then? So I'm curious about this from 2 different perspectives, but the first perspective is from how can you as a pen tester attack Terraform itself? Like, are there interfaces that are exposed that you can go after, or does this happen so far behind the scenes that you can't touch it? Or what, you know, do your eyes light up as a pen tester when you hear Terraform is being used, or do you like, are you discouraged like, oh, Terraform, nothing for me to attack there.

24:07Mazin AhmedFor me, I kind of get interested when I see Terraform or know that the company is using Terraform because before I jump into like the vectors, let's talk about how it works. The way that Terraform works is, let's say today I want to deploy an S3 bucket and then I would write the code. Would then have it applied through Terraform, and the server or the machine that is running Terraform would have all of the permissions required to deploy to the account or to deploy the resource to the account. Like essentially it should be admin access. So if like this, and then you can say that this, like, we are using Terraform to also deploy to multiple accounts or multiple providers. So this means that the same machine would have access to all of them. So if you happen to land into the machine that is running Terraform, then probably you won the battle, you have access to everything. And one thing that— another thing that they have is the way that it works. Like if I deploy today like a storage bucket and then tomorrow I want to deploy an EC2 or a security group, then Terraform would go and check what's the state of the infrastructure by checking the current things that are being deployed through a state file that they store all of the changes on. And this would have all of the resources and everything that is being deployed through Terraform. So let's say you deploy an RDS through Terraform, what would happen is it would be deployed, but, and the output, which is like the database password, would be stored in the output of the Terraform state file. And anything that is a secret or a key or anything that is being retrieved after the deployment to be reused should be there in the state file, in the typical cases, I mean. And if you get access to that state file at any time, this is a huge win. You would find database passwords, Uh-huh. Like if someone deploys an IAM user and the keys, then you would find that too there and everything.

26:51Chris RomeoYeah, that seems, it's almost like a similar problem to having keys in your code, you know, hardcoded keys to get checked into Git. Do you check that? Does that state file get checked into Git sometimes on accident as well?

27:05Mazin AhmedI haven't honestly seen this before, but I don't think that this would not happen. I'm pretty sure this happened before. to different teams and different companies.

27:16Robert HurlbutYeah.

27:16Chris RomeoSo as an attacker then, if you can get into the infrastructure that the company's using, wherever they're running Terraform from, sounds like there's a lot of interesting data and information that you can then manipulate to— if you can get that state file, you can get passwords to all the things that have been built and deployed on their cloud infrastructure and potentially find a a backdoor from the perspective of, you know, they've created a backdoor scenario, not on purpose, but just by being careless with the way that they run something like Terraform and how they protect that. So what do you recommend then? What should we do?

27:57Mazin AhmedBy the way, better yet, if you don't find a backdoor, you can even create it because you would have admin access. You can create a new admin IAM user and that would have admin access. admin access, or you can append your, like, or you can create a secondary key for another user. And then unless someone is really paying attention, you wouldn't be detected. Yeah.

28:25Chris RomeoSo what do you do? What's your recommended mitigation against this? How do we prevent this from happening?

28:31Mazin AhmedThe first thing to start with is learning about all of the attacks and all of the vectors. all of the threats that using Terraform can bring to the organization. Terraform has a lot of security features that would, like, security gains that you would get when you use Terraform. But still, just deploying Terraform and run it in your infrastructure does not mean that it would be secure. You have to do a lot of work there. From the time that you are using, or like Terraform has to store a state file or state backend somewhere. From the moment you are trying to secure the storage place that is being used on the deployment to the time, to the part where you are securing the machine or the VM that is running Terraform.

29:34Chris RomeoYeah.

29:34Mazin AhmedTerraform, whether it's Enterprise or like the version, like the Atlantis, it's like a fork or a project that is built on top of Terraform that you would use for your Terraform automation. So there are so many things that you would have to make sure to lock in your network before you can say, hey, Terraform is awesome. Cool.

30:00Chris RomeoNow that's helpful to understand how, you know, what IaC is, security benefits, what Terraform is, and then the attack vectors that someone can use against Terraform, and then the mitigations or the importance of protecting your infrastructure that you're using to run Terraform because it effectively has access to everything. That's how it can create the things that it needs to do. So what would be your call to action here, you know, as we kind of wrap up our conversation? Like, what would you like our listeners to do? as a result of this information you shared with them?

30:34Mazin Ahmed2 things. One is cloud security is not as easy as it looks. There are so many things hidden in the background and so many knowledge and learning that has to be done in order to work into building a security program for the cloud environment. And second thing is, security tools or security products are not the golden solution to solve your security problems. Even if you have the most expensive solution on the market, this— as far as you are not configured, working into configuring and securing and monitoring your cloud environment, your network won't be as secure as Mazin, thank you for enlightening us, explaining some of these things for us and for our audience.

31:33Chris RomeoI know I've got a better understanding of IaC and Terraform and its capabilities now. And I know you've got a startup that's focused on this as well. That's Full Hunt. I'll just throw the name out there as a quick plug so people can check out what you're doing with Full Hunt as far as Being able to scan and look for some of these types of attack surface challenges. So I'll throw that in, as well, you know, on top of the things that Mazin shared. Have a look at Full Hunt and see if that's something that could potentially help trying to get your arms wrapped around what this problem is. So, Mazin, thank you for taking the time to be with us today and for sharing this knowledge about IaC and Terraform. Thank you.

32:17Mazin AhmedThank you very much, Chris, and pleasure to be here.

32:19Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast and on the web at www.securityjourney.com/resources/podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, with application security, there are many paths, But only one destination.