Omer Gil and Daniel Krivelevich -- Top 10 CI/CD Security Risks

0:01Chris RomeoDaniel Krivelevich is a cybersecurity expert and problem solver with 15+ years of enterprise security experience and a proven track record working with 100+ enterprises across multiple industries. He has a strong orientation to application and cloud security. Daniel co-founded Cider Security as the company's CTO. Cider is a startup focused on securing CI/CD pipelines, flows, and systems. Omer Gil is a seasoned application and cloud security expert with over 13 years of experience across multiple security disciplines. He's an experienced researcher and public speaker, and he also discovered the web cache deception attack vector in 2017. And Omer leads research at Cider Security. Daniel and Omer joined us today to talk about the Top 10 CI/CD Security Risks initiative. They put together this list of this new top 10 It has all of the things that you need to understand to know what's the risk in your CI/CD pipelines, things like insufficient flow control mechanisms and dependency chain abuse. And so they take us through why they made this list, what was the process for building it, and then what are the things that you're going to be able to get out of it as a developer, as a security practitioner, as a DevOps person. So we hope you enjoy this conversation with— Daniel and Omer.

1:21You're about to listen to AppSec Podcast.

1:25Chris RomeoWhen you're done with this, be sure to check out our other show, High Five. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo, CEO of Security Journey and co-host of the podcast. I'm also joined today by my good friend Robert Hurlbut. Hey Robert.

1:44Hey Chris. Yeah, Robert. You know, Principal Application Security Architect at Acquia, and really glad to be here to talk about this topic. This is something I know that we've been thinking about lately in the news with supply chain attacks and all kinds of interesting things going on. So a lot of fun.

2:03Chris RomeoYeah, it seems to be one of those topics that is getting a lot of attention right now. And it just happened that our 2 guests had, and via their company, had posted this new top 10 CI/CD security risk thing. And as I saw it went by, I was like, wow, that's so cool. I want to learn more about that. I want to understand what this project is about. So, we're joined today by Daniel Krivelevich and Omer Gil. And guys, we want to just jump right in with your security origin stories because that's what our audience likes to have an understanding where the people are coming from. So, you guys can decide who wants to go first, but tell us how you got into this crazy world of application security.

2:43Yeah, I'll let you start.

2:45Cool. So I'm the CTO and one of the founders at Cider, and my origins start way back in the army. In Israel, army service is mandatory, so I served in 8200, which is kind of an equivalent of the NSA in Israel. So I got my first exposure to the Cyber domain in 8200, spent about 6 years over there. And then I jumped back and forth between defensive and more offensive positions between the defender's perspective and the attacker's perspective. I did a few years doing penetration testing, which is where I met Omer. And after that, I led application security with LivePerson, a big American SaaS company. And I spent about 4 years with Cygnea, which is an Israeli company doing incident response. And over there, I led the application and cloud security disciplines. And my job was primarily to work with organizations on identifying their crown jewels, their critical applications, understanding what they look like from a development and deployment perspective, and then understanding how an adversary would be able to abuse those processes and apply controls to optimize the posture of these processes and prevent the next incident. And of course, over there, I got a very deep understanding of the challenges that organizations are facing on the CI/CD front, which is one of the primary reasons we set out to build Cider Security and focus on CI/CD security.

4:30And then, hi, I'm Omer. I, the same as Daniel, my background starts from the army. We served in different units, but we got to know each other later when we did security consulting penetration testing at Ernst Young. Worked there for like 4 years, was a team leader there. Afterwards, I moved on to Magic Leap to their cloud security team. Been there for nearly 4 years, managed the cloud security team.

5:07Wow.

5:07That was my last role there. Cloud security, application security, this was my main focus in Magic Leap. And then Daniel founded Cider Security, which he already kind of represented. So this is where I joined, like in December '20, more than a year ago. I'm leading research at Cybersecurity, and my team tries to find the big next— the next big attack vectors and threats in the domain of CI/CD, which has a high focus today in the security world.

5:52Daniel and Omer, so here's the main topic today. What is the Top 10 CI/CD Security Risk Initiative? another interesting list that you created out there, but what is it?

6:05So we drew a lot of inspiration with this Top 10 CI/CD Risks initiative from other initiatives that we enjoyed as defenders in the past, like the OWASP Top 10 and the Serverless Top 10, which was published by PureSec. And going into CIDR, Obviously, we understood that there's this whole new domain that's becoming more and more popular, both with defenders and with attackers. I mean, we saw in 2021, or even earlier in 2020, with SolarWinds, with dependency confusion, with the Codecov hack, with so many other high-magnitude hacks, that attackers are gradually shifting focus to CI/CD. And defenders are asking more and more questions about how they can protect CI/CD processes, systems, environments. And I think traditionally when defenders, when AppSec practitioners were asked what it is that they spend their day-to-day doing, the primary area of focus was the code, was implementing scanners, that find flaws in the code and doing secure design or secure architecture reviews for the applications being built. But then CI/CD, that whole process of getting code from the developer's workstation, from the SCM to production, was for a long time, I wouldn't say neglected, but outside the scope of what application security practitioners were thinking about. And it was also outside the scope of what was considered to be the organization's attack surface, which for many years made sense before CI/CD kicked in, before, you know, so many automations replaced manual processes, before there was such a thing as infrastructure as code and automated deployments. But CI/CD has really reshaped organizations' attack surface, has created endless new opportunities for attackers, endless new ways to run malicious code in the CI, to get to production. And so we witnessed, even before CIDR, but definitely after we started our journey with CIDR, we witnessed that there are so many unknowns for defenders about what the attack surface looks like, what this new flavor of risks looks like, what are the new questions defenders need to be asking themselves, and what are the new areas that defenders need to shed the spotlight on. And that led us to the understanding that, you know, our goal as a company that's focused on CI/CD security is not only to build a technology that helps defenders create more secure CI/CD environments, but also create the knowledge and, you know, help defenders, help the InfoSec community stir the discussion and spark the discussion about—

9:08Mm-hmm.

9:09The measures, the controls, the thought patterns that we need to adapt in order to create more secure CI/CD environments. And for us, it was obvious that, you know, this type of artifact is something that we have to create to, you know, to bring our knowledge forward and to stir that discussion. The format of top 10 is one that we chose simply because it already exists. It already makes sense to people. It's something that, you know, is precedented. It's not in the domain of CI/CD. So that's pretty much the reason why we chose to do this.

9:44And curious, I know we're going to be talking a little bit about how you arrived at the 10, but, you know, you just mentioned about, for example, the OWASP Top 10. That's changed in the order over the years. Is there a particular order of severity for this list or just really put it together, and these are the things we saw.

10:06Yeah. So the top 10 risks, as we eventually set the order of them, they're set by their impact. Eventually, we believe that the first one posed the biggest impact on most organizations because the consequences of exploiting these kind of vulnerabilities and flaws pose a great impact to this kind of environment or organization. And the probability that it's going to happen is higher than other risks in the list. So yeah, we think that like the first one is the most impactful. And we also know that there are other risks besides the top 10, but We know that this world of CI/CD security is so premature at the moment in most organizations that you need to start from somewhere. This is why we get at the top 10 list, something that everyone can focus on and start protecting their environments from this spot.

11:17Chris RomeoSo if I'm a developer, so we have some people in our audience that are new to application security. developers, maybe they just got out of college, or maybe even if I'm a seasoned application security person, what's the value proposition for me in checking out this document? Like, what am I going to take away? What am I going to get from it? And what's the value to me to examine this document and learn and absorb all that information?

11:44So, it's a good question. I think it's interesting that you pointed out developers as the audience because, you know, that was our intent as well. Intuitively, the target audience is security practitioners because they are the ones that need to focus on security risks. But I think, you know, with shift left and with the whole shift in mindset of the relationship between security and engineering teams, a lot of the responsibilities and the ownership of, you know, securing engineering processes is now with developers, with DevOps. And so, you know, I'm glad that you pointed it out because our target audience is both security and engineers. And I think as far as, you know, the insights that engineers could have around this document are, first of all, understanding that CI/CD, the systems that they're working with, the code that they're shipping to production, has a significant importance or weight in an organization's attack surface. So engineers and DevOps practitioners have a big responsibility in making sure that they're equipped with the right knowledge, the right tools, the right capabilities, the right relationships with security to make sure that, you know, they have the appropriate security guardrails and that they're not making mistakes and that they're not, taking measures that might have the potential to increase the attack surface. Now, as far as the actual measures that they need to take, I think, you know, this document is a very good resource in helping them get acquainted with how the same building blocks that they work with on a day-to-day basis, the SCM repo or the CI pipeline or the package in npm, or, you know, it's those building blocks that they know and use on a day-to-day basis but didn't think of the security aspect of using them. They now understand what it is I need to think about when I define CI pipelines, when I configure permissions on a repo, when I configure my package manager client, my npm client to work with npm.

14:08Chris RomeoRight.

14:10And the more they get acquainted with the mindset, with the security building blocks, the better they're going to be eventually in instilling security as part of their day-to-day. So I think it's not just about the actual technical bullets in the document. It's more about reading through the document and kind of getting a better understanding of the attacker's perspective. and kind of sorting in their heads those building blocks that will allow them to think about security in a more effective manner.

14:45Yeah.

14:48Chris RomeoAnd so, you mentioned SolarWinds CodeCov in the context of CI/CD systems. And so, I'm curious, with both of you coming from a pen testing kind of breaking background, Why do you think that the attacker's mindset has switched towards engineering systems? You know, from, you know, I'm from the early days of security in the '90s when you could get into a Windows domain in like 5 minutes, like anybody could. They just download a script and like, oh, look, I'm the domain admin. Great. Like, but, but what's the, what's the world like? What's changed in the world to make attackers focusing on these engineering systems?

15:29This is one of our favorite questions. Go ahead.

15:31And actually, you just mentioned one of our favorite examples that mainly Daniel likes to mention. Yeah, we believe that today, similar to what you just said, that CI/CD today is like Active Directory as it was 10 years ago. As you said, like, around 10 years ago, an attacker that got access to a domain user, and it could be any domain user, not necessarily domain admin, he could reach, he could take over the domain, obtain access to a domain admin account, probably in a few minutes or hours. I'm not saying that it doesn't happen today. In Red Team assessments, or in actual breaches, it still happens. But today, this whole domain got a lot, much, much, it's much more mature, sorry, than it was a few years ago, both from the vendor side, also from the perspective of some application security vendor, sorry, security vendors in general that try to protect against these kind of threats. And today, for attackers, it's much harder to gain access to control the domain or to gain access to other sensitive assets in the network because also defenders have a lot more tools and knowledge to protect against these kinds of threats. When this trend is— when we see this trend happening, attackers need to find their path to sensitive assets and to production where the data is, where eventually the money is. And they see that it's easier to get there today comparing to more— to older methods like attacking or targeting domain environments. So we think that today attackers know that if they get access to an SCM user account or access token or SSH key or compromise an npm package, they can easily get to production in a matter of seconds or minutes if they know what they're doing. And what's even more important is that defenders don't know how to identify these kinds of attacks.

18:10Right.

18:10don't have the tools or methodologies. So it's not just that it's easier to carry out for attackers, it's also reasonable to think that the defenders won't even know that the attack occurs at real time or even when they try to figure it out after it happened.

18:32I think, yeah, I think what's not changed And what probably will never change is that attackers always choose the path of least resistance. And like you said, for many years, the path of least resistance was to gain a workstation in the domain or a user in the domain, find the groups.xml file, and, you know, become local admin on every single workstation in the domain, including the domain controller. Within minutes. But, you know, that's not the path of least resistance anymore, and it hasn't been so for a while. Defenders have a wide array of products and methodologies to detect, prevent, hunt down these types of scenarios. And what we've witnessed is that, you know, even the most mature organizations with a SIEM and a SOC and a threat hunting team and, you know, the organizations that are extremely proactive in building their cybersecurity posture and finding the most sophisticated attack vectors, when it comes to simple and trivial tasks like shipping logs from the CI system to the SIEM or looking for potentially malicious deployments to production, there's nothing there. There are giant blind spots.

19:52Chris RomeoYeah.

19:53And I think what's also unique to the context of CI/CD is that the amount of opportunities for attackers over time is growing exponentially with infrastructure as code and GitOps and all the automations. And the ratio between how fast the domain of opportunities for attackers is growing versus how quickly defenders are adapting is not linear. And think about something like dependency confusion. Dependency confusion allowed white hats, but also probably hackers for a long time to run malicious code, run malware on millions of CI environments without even having any sort of access to the environment with nothing, simply by uploading a malicious package to a public package repository. Yeah. And then that's it. Millions of CI systems are running malware. And so the ease of exploitability of some of these vectors versus how well defenders are equipped to know about, to identify, to prevent these types of vectors is really what's shifting the focus of attackers to CI/CD because there's so many opportunities over there and so little or, you know, a relatively low level of maturity with most defenders.

21:26Chris RomeoYeah, I think when I, when I consider the state of our industry right now, I would say to your point, you know, SIEMs and SOCs and all those things like InfoSec has reached a certain level of maturity. organizationally, from a CISO down to the response and all of the pieces that go with threat hunting, the things you mentioned. I see the same thing currently in our industry where things that are AppSec and product security focused are very immature across, you know, even the largest of enterprises because you'd think, oh yeah, this gigantic enterprise, they probably have this figured out. No, I don't think anybody— I'm sure there are some outliers somewhere that have a really good handle on this and are getting a better handle, but I just think there's a lack of maturity right now on more of the AppSec side. And there's a giant opportunity to provide improvements into that space.

22:20Yeah. I also think, I mean, it sounds negative or it sounds, what we're saying sounds like the AppSec domain or the defenders that are part of the AppSec domain are are far behind, but I think it's just, this is the evolution of InfoSec. I mean, this is exactly where people that were responsible for protecting cloud environments or Active Directory environments were 10, 15 years ago. They were far behind the attackers, far behind the opportunities for attackers. And then over time, they closed the gap. And this is exactly what's going to happen with CI/CD security. This is part of our motivation to As I said, not only build the technology, but build the knowledge that will empower other groups to build strong technologies and controls. And, you know, over time we'll close that gap. But, you know, attackers are always a little bit in front of defenders.

23:16Chris RomeoYeah, your least path of resistance comment just makes me smile because, you know, I've probably said that 100 times. Like, you know, especially when you're talking to people that maybe aren't as familiar with security, it's like, hey, listen, so Attackers don't work like in the movies or on TV. They don't go find the most eloquent way to break into a system. They just brute force it. If they find an easy thing, they're like, woohoo, victory, I'm in. It's not like, ah, they're poring over their terminal until 24 hours later they find the eloquent hack. They could have taken the easy one, but they went with the more detailed thing. It's not the real world, right? It's least path of resistance. When there's a door, they go through it.

23:54Right.

23:56Chris Romeoa maturing in the CI/CD space, which y'all are helping to solve that problem, a maturing in that space is going to result in the least path of resistance is going to move somewhere else. And who knows where it'll land next, but I don't even want to try to predict that.

24:10So, let's dive into the top 10. What was the process for selecting these? You mentioned a little bit about, you know, in terms of severity and so forth, but how did you arrive at these particular 10?

24:25So when we initiated the process, we had more than a few risks in mind that we knew that should be part of the top 10, or at least to be candidates to be included in the final top 10 list. But we wanted to do a comprehensive job finding them and consulting with as many experts as we can to eventually set up the best list that represents the actual biggest risks in this domain. We started with brainstorming, Daniel and myself, brainstorming on all of the public hacks, all of the hacks that went public in the recent few years. Many of them happened in 2020 and 2021. But there were some CI/CD hacks that went public even before that. There are more than a few. I mean, I think it was dozens or even hundreds of public attacks, and I'm sure that there are a lot of attacks that never went public. But we read each one, analyzed them, and started categorizing them to the different risks that we believe led to these attacks to occur. Afterwards, we saw, after we analyzed all of these attacks and also added our own risks from our own experience working with environments of customers from our experience, we had a big list of risks that we saw which ones were more popular in the biggest, in the high, in the more high-impact attacks. and which ones were less impactful. And this is the place where we gathered the first draft of the top 10 list, started writing it down. But then, you know, we did, I think, a comprehensive walk, but we wanted to consult with security experts. This is where we started approaching— Yeah. People from around the world, from CISOs of huge companies with lots of experience from their roles, and also engineers, security engineers and DevOps engineers that we believe that we can, that we definitely should consult with. And eventually we had, I think, around 15 people that wanted to join our reviewers list. You want to mention some of these reviewers? Yeah.

27:17So yeah, we were fortunate enough to collaborate with some of the world's top experts on application security. We worked with the CISO of Atlassian, with the person that leads application security at Netflix, CISO of Lemonade, CISO of Relativity. There's a long list, you can find it in the top 10, but I think the general sentiment or notion behind all of this is that we wanted to make sure that if we're making this big contribution to the InfoSec community, obviously we have a wide perspective and gained a lot of experience over the years, but we wanted to make sure that, you know, it is as relevant as possible to as many different contexts as possible. And this is why we were extra diligent in, first of all, identifying and reading and seeping through as many anatomies of public hacks that were exposed. And you can see all the references and links in our document, but we analyzed these hacks one by one, understanding what went wrong, what capabilities the attackers had, what capabilities the defenders had. And then, of course, had those discussions with the application security leaders, the CISOs, around the concerns that they were facing, around what they had experienced from their own personal experience. And obviously, We also incorporated the dozens or hundreds of environments that we analyzed as part of our tenure at Cider. Again, we're fortunate to work with very collaborative organizations that, you know, we help secure their CI/CD environments. And we talk a lot about the architecture, the design of the environments, the type of challenges defenders are facing, the type of—

29:16Yeah.

29:17incidents that these companies had had. And when we put all this together, we had a very, very large data set that we analyzed for a long, long time, which ended up compiling those top 10 risks. And it took us a while to get to a point where, you know, we're in consensus because putting together, I think it's 17 different opinions, is not trivial, but we all realize that it's worth the effort because the final result is going to be very reliant on a very, very wide dataset and is going to be relevant to a very, very wide portfolio of organization types and levels of maturity. So I think, yeah, that was the process in a nutshell.

30:11Chris RomeoYeah, Robert and I are familiar with that as well. We worked on a project called the Threat Modeling Manifesto, where we got together a bunch of experts and debated, discussed, had a lot of fun working together to build a collaborative product that we were— we ended up, we got to the end and we were all proud of it. And we all were like, we'll put our name on it. But certainly understand that's a— it's an interesting and fun challenge to try to put that together.

30:38Yeah.

30:38Chris Romeowith a group of people who come from different backgrounds and perspectives. But at the end of the day, if you can pull it off, and you guys have, you end up with a list that, you know, nobody can go and look at that and say, ah, that doesn't have— there's no real— those people didn't know anything about the real world. Well, that list of people you have, they're running the real-world internet right now, dealing with the threats that a lot of people and companies don't even really know the scale and scope of what they deal with. So, I want to dive into talking about a couple of these since we've kind of set the stage about, you know, where the project came from and how it all came together. But what I want to do is have an opportunity to talk about the first one on the list, insufficient flow control mechanisms. Let's go ahead and start there. Let's just give us a high-level view about what this particular item is. And then for me, I'm always curious about the mitigations. That's always where I land. Like, once I understand it, I'm like, all right, now what do I got to do to fix it or get away from it? So let's start there with the first one on the list.

31:38Okay.

31:40So the first risk on the list is insecure flow control mechanisms— insufficient, sorry, flow control mechanisms. And it's around the ability of an attacker that has got control or obtained control over one of the systems involved in a CI/CD environment, could be the source control management system or the CI, the ability to ship code or artifacts to production without any control or review that stands in its way towards production. Obviously, with the purpose of shipping malicious code and artifacts, obviously. The point of CI/CD is eventually to automate the process of shipping code and artifacts to production. When you write code that you want to run in production, when you push it to your repository, to like a feature branch, it usually goes through through a set of tests, could be security tests and unit tests and integration tests and reviews by other developers and the build process of this code. There are more than a few stations on the way until the code eventually reaches its destination. And it's all automated between different systems and pieces of code. And attackers can take advantage of it. And ship their own malicious code to production once they obtained their access into the environment. We can talk about several, like, opportunities for an attacker, for attackers in this case. Let's say that attackers obtain access to an access token of a repository for an application that hosts application code that eventually runs on Kubernetes, in a Kubernetes pod, or as a Lambda function, or whatever in the cloud, then you can add your own code. And if it's automatically deployed to production without any review or scanning on the way, then an attacker would be able to act the same as a developer and ship code directly to production. This is something that obviously you want, you would want to prevent. So, from the one hand, yeah, you want to give the, to provide the freedom to developers to be able to ship code fast to production. This is the whole purpose of CI/CD. That's correct. And you eventually don't want to prevent developers from working fast, but still, you need, we all need to understand the impact.

34:35Yeah.

34:36of a case where an attacker obtains access to one of the hundreds or thousands of developers in the company. And he could— a developer can have his— I mean, an attacker can obtain access to their username and password or access token or SSH key or compromise the GitHub app used in the GitHub organization. So once the attacker obtains just one of authentication method to a repository. He can use it to— if he's able to use it to ship malicious code straight to production, that's probably something you want to prevent. And we can talk about some mitigations for this. I mean—

35:19Chris RomeoYeah, let's talk about— let's get into the mitigations now because I feel like I have a better understanding now about insufficient flow control mechanisms and, you know, It's almost a generic category where a lot of different things, a lot of different components in the architecture could fit into it. It all comes down to the attacker being able to put some type of malicious code and drop it somewhere through the CI/CD pipeline and in somewhere in production where they can get to it. So yeah, I guess, what do we do? How do we help ourselves to not have this particular issue?

35:55I think just, I mean, Before the actual tactical measures, I think, and this goes back to your point about what type of things developers need to think about when they're reading the document, I think one of the special things about the mindset that we've adopted at Cider and just generally the place where the industry is going as far as CI/CD security is that we're adopting the understanding that The source control system, the CI systems, the systems that were perceived for many years as outside the scope of what security teams need to be concerned about, they are now the path to production. And they are now the best avenue for attackers to get to production. And we need to consider, for example, GitHub or any other source control system. They are for many years now, no longer a version control system that is just a system where developers collaborate together on code with versions and so on and so forth. No, it's a gateway to a series of automations that ends up in an artifact being deployed to production. This is the front door to production. And once you begin understanding that the source control system, or definitely the system that does the deployment to production, is a system that contains the credentials that are ultimately used for deployment to production, so credentials that allow accessing production, we need to treat them in the same way that we treat any other system that stores credentials to production. We need to treat them like we treat the domain controller or the way we treat the vault, the CyberArk, or whatever PIM/PAM solution we're using. And, you know, once we adopt that mindset, once we understand that our GitHub or whatever source control system we're using or our CI systems are systems that we need to be as diligent around security as we are with, as I said, the domain controller or any other crown jewel, then there are a lot of measures that we need to be proactive in understanding how to implement around how to secure a repo, how to prevent, you know, how to prevent a reality where one single set of permissions One push to a repo is sufficient to ship malware to production because then that's the same as one set of domain permissions, one low-privilege domain user being one event away from being a domain admin. It's very similar. So that mindset, once you adopt it, then actually implementing the measures, once you're comfortable with taking those measures and taking those strides to protect these systems. The settings, the configurations, they're available. The shift in mindset is, I think, what organizations find the most challenging. But let's talk about the actual tactical measures. Yeah.

38:54Chris RomeoBefore we get to mitigations, I just want to share what you just said. Something just dawned on me that I can't believe I didn't have this thought before. But All of those years of us saying, oh, that's just internal. This is a system that's just internal. Like, it's never going to be on the internet. It's never— we can have a different standard of security for those things that we only keep on the inside of the network versus those things that are on the outside. And it sounds like what you're saying now is we're in a CI/CD world where those things aren't on the inside anymore. And so they can't, you know, there is no physical isolation anymore between those things. They're engaged in various internet components and whatnot.

39:32Yeah.

39:33Chris RomeoAnd so that's to your mindset point. It just dawned on me there. I was like, I just had a moment. I was like, wow, I didn't— I hadn't thought of it like that. But that's the reality of the world we live in. So let's talk mitigations now. How do we solve this first challenge?

39:47So there are many different measures that you can take to prevent against a direct push of malicious code or artifacts to production. One common practice is simply branch protection rules, for example, in the source control management system side. Something that we see that nearly any organization implements, at least in the most important repositories and branches that are eventually used in production, are branch protection rules preventing engineers from directly pushing code to sensitive branches. that are then used in production. For example, you have a repository with the main branch. When you push code to the main branch, there is a pipeline being triggered, building, testing the code, then shipping it to production. Usually, most engineers or all engineers can't directly push code to this sensitive branch, but they have to push code to a feature branch and then create a pull request, get approvals from doing the peer reviews, and only then they are able to introduce and merge the code to the main branch, which will ship it to production. I think that this common practice, which is eventually a manual review, not necessarily, there are some other practices there. You can also add security scans in this phase of of creating a pull request, but this phase of manual review, when people on your team review the code that you introduced, this gives a hard time for attackers introducing their code that will eventually be pushed through automation to production. So making sure that any branch that is eventually shipped to production is protected and you have to receive at least one review from your peers in order to merge it. That's a common practice that many organisations use. But of course, you need to make sure that you have this practice in place on all of your sensitive branches, in all of your sensitive repositories in your organisation. And this could be a challenging task when you have thousands of repositories and hundreds or thousands of pipelines. In a medium-sized organisation, it could be hard to achieve. But that's a common practice today. Another example, a quick example, is that let's say that you protected your sensitive branches and you have security scanning. And when the pipeline is triggered, let's say that it builds a container image and stores it in a container registry. Then, this image is fetched and deployed to production by a deployment pipeline. You can say, okay, there are reviews and scans in place to make sure that this image was reviewed and there are no vulnerabilities inside when this image was stored on the container registry. But what happens if some user that the attacker obtained access to has direct access to this container registry, and the attacker can directly store a malicious image there that will be then fetched by the deployment pipeline and deployed to production. In this case, this whole pipeline and protection there become worthless because an attacker can just bypass the pipeline restrictions and put malicious— and create a malicious artifact in the registry which will be deployed by another pipeline later on. So you need to think of all of the moving parts in a pipeline from code to production. It's not just the first phase of triggering the pipeline or the connection between the repository and the CI. There are many moving parts and it looks different in any organization. So it's challenging for an attacker to map all these, different attacks and understand how it looks like in their organization. But it's possible. And this risk, the insufficient protocol mechanisms risk, details various types of attacks that need to be considered.

44:17Chris RomeoYeah, I mean, it's an argument for threat modeling the CI/CD pipeline. And the top 10 that you've put together is a great input To people like Robert and I that are threat modeling fans to be able to absorb. Another pitch for people to go out and examine this document is, this is now input for you as you're looking at a CI/CD pipeline to start looking for what are the threats, how are we going to mitigate these? I'd love to spend literally the rest of my day and go through all 10 of these. I'm not joking. I would love to literally stay here and do an 8-hour podcast where we go through all 10 of these. But I know we don't have time to do that. And so, I just wanted to— we wanted to go through one of these so that our audience gets a feel for what they're going to find when they go. There's 9 more of these that we haven't even talked about yet. So, this is an advertisement for the Top 10 CI/CD because you saw one, now you can go absorb the other 9. I really had 2 more questions. I know we're getting towards the end of our time here. I wanted to make sure you got a chance to talk about CI/CD GOAT. Because, Daniel, at the beginning when we were talking about, you know, the impact for developers and how that comes together, tell us your kind of the philosophy that you have about what CI/CD Goat's going to do together with this top 10 list to really help developers understand this stuff.

45:35Sure. So I think, you know, we realized as defenders before Cider that if you want to be a good defender, You have to adopt the attacker's perspective and you have to be familiar with the way attackers think. Otherwise, what you're doing is never going to be relevant to your risk landscape. And I think beginning to adopt the attacker's perspective with CI/CD is, you know, learning the theory. understanding the building blocks that comprise the way attackers look at CI/CD as an attack surface. And for that, I think the Top 10 is a great artifact. If you don't know the way attackers think and what type of risks attackers are looking to exploit when they look at CI/CD, then probably reading the Top 10 will be very beneficial in beginning to think like an attacker, but obviously theory is never enough. You have to practice the tactics and really try to think like an attacker when you're working in these environments, and then fix these environments hands-on to understand what are the measures that organizations— like what measures really make a difference And this is precisely why we created the CI/CD Goat. The CI/CD Goat is a deliberately vulnerable CI/CD environment. It's a full-blown environment that can be run locally on the machine of anyone who clones the GitHub project with a source control system, a build system, a build agent, and kind of a a full-blown environment that enables users to basically deal with 10 challenges of varying levels of difficulty. Each one represents different types of risks within our top 10 CI/CD risks. And so once you've gained the theory with the top 10 CI/CD risks document, Then practicing those risks, trying to think like an attacker, trying to enact these scenarios, and then trying to fix them as the defender and understanding how the measures I take as a defender are now preventing those same attack vectors that I just enact as an attacker is what we believe something that is going to make a significant leap for defenders, not just security practitioners, but also DevOps engineers, anyone who's interested in securing engineering environments. So that's going to make a very significant leap in the knowledge and understanding and experience these people have with securing CI/CD environments. You want to add anything?

48:44Yeah, I mean, the GOAT is meant for everyone. Not just for hackers or people that like to capture the flag challenges. There are different levels of difficulty between easy to hard, and there are hints that you can use. And it's mainly super fun to solve. You can easily, like in 2 minutes, you can run, you can have the entire setup on your machine. It's all Docker-based, so it's really easy to start working with it, to start working with, and I encourage you all to try it.

49:27Yeah.

49:28Chris RomeoLove it. Love it. So, I guess the call to action I'm going to throw back to our audience right now is, you've heard about the top 10 CI/CD risks that Daniel Nohmer have put together via Cider, and so go out, check check it out and read it, absorb it. You know, you heard a number of reasons why developers, security practitioners, DevOps, everybody that's involved in this process needs to absorb this information and really gain this awareness because we know, as we talked about, least path of resistance right now is the CI/CD pipeline. So, Daniel and Omer, thank you for sharing this with the industry and sharing your knowledge with us today, and we look forward to following along with all the other cool things y'all do in the next, uh, next year along the way. So thanks.

50:13It was very fun talking with you guys.

50:14Yeah, thank you.

50:16Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast and on the web at www.securityjourney.com/resources/podcast. You can also find Chris on Twitter @EdgeRoute and Robert @RobertHurlbut. Remember, with application security, there are many paths, but only one destination.