François Proulx - Arbitrary Code Execution 0-day in Build Pipeline of Popular Open Source Packages

0:00Chris RomeoFrançois Proulx is a senior product security engineer at Boost Security, where he leads the supply chain research team. With over 10 years of experience building AppSec programs for companies like Intel and various startups, he's been instrumental in the DevSecOps movement, making numerous responsible disclosures to organizations such as AWS, Google, Red Hat, and ChainGuard, and speaking at conferences on the topic. He's also a founder of the NorthSec conference in Montreal and was a challenge designer for the NorthSec CTF. François shares the story of how he and his team have discovered a class of vulns within build pipelines. This is the side of software supply chain that people do not realize is full of threats. The Application Security Podcast is brought to you by Security Journey. Our training includes theory and immersive learning that teaches the skills and knowledge needed to create a security-first mindset across your organization. Learn more at securityjourney.com. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo. I am the CEO of DaVinci, that is the threat modeling company, and joined by my good friend Robert, who I spend a lot of time in podcast recordings with, but We love every minute of it. Hey, Robert.

1:23Robert HurlbutHey, Chris. Yeah, Robert Hurlbut. I'm a principal application security architect and threat modeling lead at Acquia. And as you said, always glad to be here again to talk about application security in a podcast.

1:35Chris RomeoYep. We can't, uh, can't beat it. There's not really anything else we could do a podcast on. I realized I was like, could we do something about the weather? No, I don't know anything about that. I can look out the window, but that's not really great radio right there.

1:49Robert HurlbutSo.

1:50Chris RomeoWe're joined by Francois Proulx, second-time visit slash visitor to the podcast. And so we've already heard your security origin story, Francois. So now you get a new question. And that is, what do you like to do that gets you away or takes you away from computers and technology?

2:10François ProulxYeah, I heard Tanya's answer some podcasts back and I could relate. So I'm not gonna not gonna copy Tanya's, but I do relate to her gardening thing. So it's very much something that I do in the summer that just gets me out, like, you know, completely in a different mood. But I would say something that I do more regularly that I do enjoy a lot is cooking. So I do enjoy watching, you know, YouTube channels about cooking techniques and getting quite deep into it. So that's why— If I had a book recommendation, it's O'Reilly Cooking for Geeks. It's a very good book that came out about 10 years ago that really goes into the geeky side, like the chemistry of cooking, like how temperature affects the changes of flavor and things like that. So I do like a little bit of geeking out with my cooking, but otherwise just like plain good flavors.

3:12Chris RomeoYeah. All right, that's cool. So now I'm going to give you a scenario. Robert and I invite ourselves to your house for dinner. We give you one day to prepare. What would we expect if you were— if we said, François, cook your signature dishes? What am I going to get for— what are we going to enjoy for an appetizer, for a main course, and then for a dessert?

3:32François ProulxOh, well, you kind of caught me off guard, but But I do enjoy as a— if I'm gonna have like a nice meal with people, I do love some nice shellfish like oysters. I don't know if you guys are into that, but love that.

3:49Chris RomeoYeah.

3:49François ProulxLike cannot go wrong. A nice little— and a nice wine to go along with that. And for, you know, main course, I don't know, like it could be some nice grilled meat on a barbecue or something like that. Just something as simple as that. But in recent months, I got into more like kind of vegan or vegetarian. I'm not like, you know, per se a vegetarian or a vegan in any way, but I do like to try to find ways to reduce consumption of meat and still find extremely good flavors. So I have like some good books. I don't have at the top of my head some ideas of recipes in there, but there are amazing ones that just I couldn't imagine to have such great flavors with basically just vegetables. Just really the cooking techniques make such a difference.

4:39Chris RomeoIt always, I mean, it raises your game to have to cook a vegetarian, vegan dish and not be able to use meat in any way. I think it just, it's a challenge, right? As a chef to try to create something that tastes incredible. Not that vegetables on their own don't taste incredible, But it's just, it's more of a challenge, right, for you to create it then. Do you, so do you enjoy the creative side of cooking? Is it like, what's the draw?

5:07François ProulxYeah, I think I like to kind of fine-tune, just like change flavors, just to experiment a little bit, but still kind of stick to the core recipe. I'm still not like, you know, chef or in any way, like I wouldn't do cooking contests or— but I just, I just love just the, the experience just to be in the kitchen using the utensils, just like, you know, experimenting in my own way.

5:38Robert HurlbutYeah.

5:40Chris RomeoAnd it does get you away from technology because, you know, you're using old school technology, knives, cutting boards, pans, you know, various spices and salt, pepper, all those things going in. So Excellent. No, I love to hear that. I love to hear stories of how people are using hobbies to get away from technology and computers because we all spend way too much time in front of screens working until the middle of the night sometimes because that just goes with the territory. But I love these stories of people. You know, we had Jeff Williams and his basketball league. We had Tanya with her her gardening, and now we've got Francois as our resident chef in train. We'll call it chef in training, right? Because you said you didn't want to be referred to as a chef at this point. So chef in training. But, um, yeah, excellent. Thank you for sharing that, uh, that story with us. Uh, in case you thought you tuned into the cooking podcast, we in fact are not. This is the Application Security Podcast. Robert, why don't you tee up our AppSec-related topic for us here?

6:45Robert HurlbutOkay. Hey, Francois, I know you talked recently at a thing at a conference about a particular issue with an open source tool or library and supply chain. And essentially, you were able to find a zero-day, if I remember correctly. And I wonder if you could help us understand the background of that, you know, set up that story for us. You know, what was the package, the vulnerability, the chain of events? You know, what happened? You know, how did you find out about this and so on and so forth? Let us know.

7:16François ProulxYep. So in fact, I gave that talk a number of times already and I'll be giving it next week again at Black Hat SecTor in Toronto. But so this year I'm kind of going around, but the first time when I created this slide deck, the talk was for the Linux Foundation Open Source Security Foundation Summit in Seattle. So they had that, and I really am a big fan of that. We kind of contribute to OpenSSL. So it came out, the backstory is really, and that's what I tell in the talk, is that like overall, when we hear about supply chain, we mostly are talking about the risk management of known vulnerabilities, so CVEs. People talk about SBOM and SCA and things like that, and that's, you know, quite mature and getting quite boring by now, like you're just like sick and tired of it, that's fine. So we're not talking about that. Neither are we talking about supply chain in the context of malicious packages that are popping in npm and PyPI, Docker Hub, and all that. 'Cause most of the time when you hear about it, it's like typo squatting, you know, like this, a package, very popular package, someone submits it, a malicious version with just a small typo that's very easy to mistake. You just, you know, pip install and just one letter away, and then suddenly you install something that is not quite the OpenSSL, right? It's just OpenSSL with an extra S and you're owned. No, we're really focused on the build pipeline because that's an area that we think is not as much on most people's radar, while it's still very much the topic of the OWASP CI/CD Top 10, which I know we talked even in the past together, and you had the folks that created it some time ago. But in my mind, it's a very good resource, but it really lacks kind of in the sense of pragmatic remediation guidance. It's a bit more of an academic kind of exercise. Yeah. And to be fair, I do love what they did. Like, and kudos to those guys, very big fan of their research, but it feels sometimes like they stretched it a bit to fit the top 10. Like they kind of, you know, made it fit a list that has 10 items. So yeah, anyway, but the point is that when we looked at, we started to do our own net new research, my team, like I lead a small security research team, we started to focus more on those build pipelines. And what we found quite curious is that when you look at CVEs for build components, the only one that really has like a category that's easy to find CVEs is GitHub Action. And I'm not kidding, there are no more than 20, like literally 20 or 21 CVEs for GitHub Action while there are tens of thousands of GitHub Actions. So our intuition was just, it really couldn't be the case that there's just 20 or so vulnerable GitHub Action or GitHub Action workflows and things like that out there. So we, and of course, we didn't invent the core research, like people had done that in the past. So we just, so, okay, let's kind of—

10:48Robert HurlbutRight.

10:49François ProulxDo our own research, like get to really get up to speed with it, and then like bring something new to the table. So we went on and kind of created the infrastructure to gather that data at scale and really get a sense for like, is it still the case that, you know, vulnerable build pipelines are out there on open source packages? And the answer is absolutely yes. Today, like, to this day, I still find numerous, very trivially exploitable build pipelines on very prominent, very important open source packages. Just last month, I reported 22 vulnerable repos on a very large company that you guys all know, and...

11:38Robert HurlbutGitHub.

11:40François Proulxa month or so before, the same thing for another very large company that basically just does open source. So you can, you know, make your guess. So it happens and I keep finding, and those I would consider still low-hanging fruits. Like there are things we've built the infrastructure to scan at scale, millions of repos on GitHub. and do our own. In fact, in the course of doing that, we built our own open source scanner that we call Poutine because us, we're based in Montreal, Quebec, where poutine was invented. And if you know the backstory of why poutine is called poutine, because in French it kind of has this meaning of something messy, right?

12:26Robert HurlbutYeah.

12:26François ProulxAnd we think of build pipelines sometimes as kind of messy, kind of slapped together pieces of scripts. Um, so we, you know, say, hey, like, if, if there's someone who can call a CLI tool called Putin, no one had done it. Like, can you believe it? Like, there's no CLI tool called Putin. So we did it.

12:43Chris RomeoSo let me, let me read this back to you and make sure, make sure I'm tracking with you about where, kind of what we're describing. So what you were doing was looking at the build pipelines. So it's not the packages themselves. It's the build pipelines that they're using to create the versions of the packages. And this is where you found this gaping hole, because I feel like people are running some type of checks, I hope, against major open source components. Like they're running scanners and things in those pipelines to check the component source code itself. But what you're saying is you have laser focused on the build pipeline tooling that exists to build the versions of it. And that's where you're finding these 20s of issues that you're reporting, you know, on a weekly basis. Is that— am I tracking with you?

13:41François ProulxBut to be fair, often it is in the pipeline where you run those linters. And I found tons of pipelines where people run SonarCloud, not to say it Semgrep, whatever, that are vulnerable. So that is my Trojan horse into compromising the rest.

13:58Chris RomeoSo they're using the security tool. So there's, there's known vulnerabilities in the security tool. And instead of grabbing the latest version, when the pipeline kicks off, they've hardcoded a particular version of a particular tool that they're grabbing every time, including in the build pipeline. And That's how, that's where the door's opening?

14:21François ProulxSo in fact, when we did that research, we started to notice certain patterns and we had to in fact create an inventory of those that we ended up calling the Living Off the Pipeline project. So very similar to living off the land, right? Same idea. So you're in an environment, you get some kind of RCE, what binaries are already there that you can use to your advantage to avoid pulling some extra stuff and you get the job done, right? To pivot further. So same idea here, like living off the pipeline, just use the linters. In some cases, they bring very powerful plugins to just have RCE by design. I mean, I'll, again, like people can look up this living off the pipeline project. You will see very well-known linters, static analysis tools that they know, And many of them, you can modify their behavior by putting a config file at the root of the repo. So I'll just take Chekhov. Chekhov is a well-known thing to look for infrastructure as code, right?

15:29Chris RomeoYep.

15:30François ProulxIf you put a .chekhov.yaml at the root of your repo, then you have RC. So you can modify it and dynamically add a plugin with like raw Python script. Then if Chekhov happens to have access to other secrets to, you know, scan your infrastructure. Well, that's it. You pivot to that, thanks to Chekhov.

15:52Chris RomeoSo, so make sure, I wanna make sure I understand here. So what's the attack vector? Do I have to get a PR?

16:01François ProulxYes.

16:02Chris RomeoSo I have to be able to, to, so, but, but I guess most of those open source tools, I can create a PR. without anybody doing any, without any other, I can get the build pipe, you can fire the build pipeline without anybody doing any type of approval because they don't approve it until after the PR has already gone through. So that's how the door is opening to you. There is no control on the front door.

16:22François ProulxExactly. So like in most cases, the most common kind of scenario, attack scenario is a public repo. So in fact, in the talk, I show an attack tree and it looks something like this. Like, The conditions that need to be met is you have a public GitHub repo. It could be on GitLab, you know, but basically some kind of public repo that's running a pipeline automatically as soon as someone commits, like opens a PR, for instance, like so automatically triggering the pipeline to run linters, for instance, right? And then if those tools that run automatically run with an elevated context, a privileged context, which is not always the case, mind you, right? But often, too often, they have access to secrets for whatever reason. Maybe they need an API key to pull something, or they need to write the results of the static analysis tool somewhere. So, they need access to secrets. And for the pipeline to have access to secrets, the secret store kind of, you know, the vault, well, they need to use the pipeline in privileged mode.

17:31Chris RomeoMm-hmm.

17:32François Proulxwhich then if there's an ability to have RCE in some way, which living off the pipeline is one way, then you can dump the memory and dump the secrets and then pivot to do all sorts of stuff, which could be compromise the repo itself, compromise infrastructure where it's storing the Docker image that is being built, for instance.

17:53Chris RomeoGot it, okay.

17:59François ProulxSo that's that, you know, and we've built the infrastructure to find that at scale. So it's really, again, still low-hanging fruit, unfortunately.

18:09Chris RomeoSo if we sum that up though, a lot of it is poor pipeline design because they're not, like, there's a way you can do secrets where you minimize the time that the secrets vault is open and what can use the secrets. And I've seen people do it with like multi-stage container builds, for example. Like you get what you need. Like you don't, first of all, you don't drop secrets till you get to the very end and you're kind of staging your container build and dropping things. So you're not just, it's not just like everything has access to the, to everything that's happening in the pipeline. You hit a phase where you're like, okay, everything looks good. We're green. We're we're gonna go ahead and drop secrets in right before we ship the thing out. So, is that, so is that, so there are architectural ways then in the pipeline that you can, sounds like there's probably a list of things that you could do, but one of them would be to better architect your build flow so that you end up not leaving secrets exposed from the time the pipeline starts till the time it finishes.

19:13Robert HurlbutAbsolutely.

19:14François ProulxSo, it really comes down to isolating, sandboxing, Each step, and and it starts by knowing threat modeling. It starts by knowing well first again, like least privilege. Just like as you just said, like you you want to only keep the secrets to the part that you absolutely need it. So do you need to expose them in memory? Even if you know, like oh, they're just in memory; they're not you know written to disk like somewhere. But if they're in memory and there's RCE, well.

19:45Robert Hurlbutgame over, right?

19:46François ProulxSo you want to run the parts where you're touching arbitrary input, right? The code that's coming from the PR, that's, you know, from someone that just forked it and opened the PR, you need to consider like, you know, toxic waste. And if you run a linter and it has RCE, well, it should not, you know, pivot to something else. It should, at best, maybe output to a text file, the output of the linting, and then that text file is loaded up in a completely different job. Like, for instance, let's take GitHub Action. You can separate it in jobs. So each job is basically a virtual machine that is fresh, like complete ephemeral. So that's the right way to design it. It's multiple stages that connect, you know, pipe and filter, only the thing that you've kind of vetted as trusted, but at the beginning of the pipeline, it's very dangerous.

20:51Robert HurlbutYeah. So in terms of looking at this, is there a worst-case scenario or even just understanding how prevalent this particular vulnerability is in practice.

21:05François ProulxYeah, so, well, worst case scenario, I guess it really comes down to how popular the package is and the distribution mechanism, how it's being consumed. Let's say we assume it's on a package manager, like a package ecosystem that's very popular, and people will automatically update to the latest without much, you know, thinking. And the build pipeline is basically pushing to that. So it has secrets to push to it automatically. And there's a way in through either a pull request, but that's not the only one. We can talk about other ways to get in, but they're a common one. Then, yeah, the worst case is getting those secrets and then using them to plant a malicious version before anyone notices. And, but There are other things, as I said, like pivoting, like often those environments go and push to, let's say, an AWS or Google Cloud environment where they run various tests to kind of, you know, some kind of end-to-end test, but that environment might have access to many other things. So it's like, again, kind of isolation and all that. So it could pivot much farther. So, In the case of self-hosted runners, which in the case of public repos is very dangerous because that you basically don't use ephemeral machines that are hosted by, say, GitHub Action directly, but you provide your own VMs, then you're responsible for hardening it and isolating and all that. And in that case, like pivoting further into the infrastructure is very possible.

22:48Chris RomeoSo the endgame though, for an attacker is to use the compromise of the pipeline to deploy a malicious package. Yeah.

23:00François ProulxRight.

23:00Chris RomeoThat's the primary endgame here.

23:02François ProulxYes. Most, most of the time.

23:03Chris RomeoOkay. And then, yeah. So from there, the sky's the limit as far as what you can do. I mean, we've seen plenty of supply chain attacks. We're not going to mention them because people have talked about them ad nauseam. But people put, uh, we've seen supply chain attacks where packages were modified. They get rolled up to a binary, they get deployed across an entire fleet of applications. So is this really a vulnerability or the type of problems you're finding here? If I'm using like a CircleCI or Jenkins, I don't know if anybody uses Jenkins anymore, but if I'm using kind of a non— open-source hosted kind of cloud model for this, is there any risk or is this really tied to the public repos of GitHub and GitLab and places like that?

23:55François ProulxSo I would say all CI systems are vulnerable to that as long as they can get triggered Especially here, okay, let's step back for a moment. If we're considering only in a threat model, external attackers versus insider threats, big deal, right? Big difference, right?

24:19Chris RomeoRight.

24:20François ProulxSo we're just talking about someone out there on the internet that is not part of your core, like, direct maintainers community, because then we get into the XZ kind of scenario where, like, an insider to the project that is in fact, like, a core maintainer, then it's just slipping you know, flipping sides. So that, what can we do about that? It's just computers, who's in front of the keyboard. But if we're talking about like someone on the internet, well then those things, when you just open a pull request, code starts to run. This is the same as you have a server with open ports and you accept, you have command injection, you know, accepting query string parameters and then that's it. basically more or less the same, right? So you accept people typing some code, like these files, they push it, you run code, like based on processing that file.

25:17Chris RomeoOkay. So it seems like when I think about solutions, and I already jumped the gun on this talking about architecting pipelines, I jumped right to the end, but I couldn't resist. It seems like there's 2 different groups that need to make changes as a result of this. There is the— or could be, I don't know, you tell me if this is, if this, if you're in line with this thinking or if I'm off base somewhere. There's the group we've been talking about, which are the owners of the pipelines who at some point set up a pipeline for their open source package and apparently are not doing the proper care and feeding of it. over time. I think that's probably what happens a lot of the time. But it seems like the other group would be the infrastructure providers, the CI/CD providers themselves would be able to do some things to help people at least understand these issues or provide some guardrails. It seems like there's room for some guardrails here or even paved roads, both in a pipeline, a hosted pipeline that I provide to people. Like I could provide some some paved roads and guardrails that would help to prevent against this. So, let's take the infrastructure provider first. What do they need to do to prevent these things? And then we'll go back to the open source package person that's out there.

26:40François ProulxWell, let's take GitHub, because they're the biggest, like, out there, like, for, like, you know, the vast majority of open source packages now are hosted on GitHub, right? So, they have a big potential to help there. And I would say based on our numbers, it's like almost all of them use GitHub Actions because it's free on public repos. You get GitHub Actions for free. If you look at their documentation, they tell you how to write those pipelines correctly, right? They tell you if you accept pull requests, you need to treat it like untrusted input and you need to use the workflows to segment them and use the right type of events like Pull request, there are 2 types of events trigger the workflow. One's called pull request and one is called pull request target. The second one is dangerous. The first one is sort of, you know, safe by the design in the sense that even if you accept arbitrary code, it will not have access to secrets. It's just like by design. So that's pretty safe. The other one though, it gives access to secrets. That's what you need, and the documentation says be careful, but many people find good reasons to need to have access to secret, as I mentioned before. Maybe you need access to some kind of API key to pull or push the results, right? So, as soon as you need to do that, then it's kind of like you get access to all secrets, or at least you get access to maybe more secrets than you really need. So segmenting, isolating those pipelines is at play here. So infrastructure providers, well, one thing that GitHub is, I don't wanna say too many bad things, but sometimes, you know, GitHub, since it was acquired by Microsoft, tends to play a little bit the kind of old Microsoft card of backward compatibility. So they're very scared about breaking, you know, breaking, making breaking changes. Mm-hmm. So whenever they introduce improvements to GitHub Action to harden the default behavior, that goes towards that exactly, to make it easier, batteries included. Those defaults, often, if you created your repo or your organization before the new defaults came in, they don't even tell you that the defaults have changed, and they don't, they certainly don't change it for you. So you need to kind of monitor the changes, the new best practice, and make the effort to go in the settings and to follow the recommendation.

29:21Chris RomeoYeah.

29:21François ProulxOtherwise, you know, most projects have been created well before those new defaults have came in. And that means that the majority of projects, unless they went out of their way to read and, you know, learn about all that, They have insecure defaults.

29:40Chris RomeoYeah. I'm going to stick up for Microsoft for a minute here. Can't believe that's happening at this moment. But I would say the same problem exists across all cloud deployments, right? Because if you think about, I just happen to be more familiar with AWS. You think about where AWS has, how it's matured. You could say the same thing for Azure, Google Cloud. They've all matured in the last 5 years from a security perspective. If you built an environment 5 years ago and you built one today, the one today is going to have lots of new security default and, you know, built-in kind of capabilities that the previous one, they don't retroactively apply them. So I wouldn't say that's a Microsoft— it's not a unique problem to Microsoft in this context. It's just nobody's really figured out yet, how do you have a button or a countdown that says in 30 days you're getting security whether you like it or not? Nobody's figured out how to do that without breaking. I'm sure they've tried. I mean, somebody probably thought that'll be easy and then realized what happened when it breaks the entire person's cloud environment. Nothing works anymore, right? Because they've launched all these better secure defaults. So that's, I think that's a problem. That's just, that's just a technology problem that I don't think I haven't seen. Let's, maybe I've been under a rock, that could happen, but I haven't seen anybody that's solved that problem. The general generic problem of how do you retroactively apply security over time.

31:05François ProulxYeah, but even if we're not talking about doing it automatically, it's more about communication, and they don't do that. Like, they don't even— they're scared to even tell you that there is new, better defaults that you should consider. That's the thing. It's like, I think it's transparency. And what they could do without doing that, they could just proactively notify in a pull request if they see you're using those more risky and they see your configuration is more you know, vulnerable, and then you just received a new pull request from someone that never contributed before, maybe they should have like a little warning that says, hey, like, they can run the same static analysis tools that we do. We run it at scale. Couldn't they do it like live? So, yeah.

31:50Chris RomeoAnd I think they'll get there. Like, think about MFA now. MFA on sites where you don't have, like, Microsoft is great at Setting a deadline for a tenant where you have to migrate to MFA. You don't have a choice, and you get the countdown, and eventually you get to the point where you're locked into your accounts locked until you go through the MFA setup process. So they figured it out for something as crucial as MFA, which which makes me think we just need more time. I mean, we we've had MFA since I got started in security. Like it existed as RSA tokens. back in 1997. Yeah, around that time.

32:31Robert HurlbutRight.

32:31Chris RomeoAnd here we are, 27 and a half years later, or 28 years later. And now MFA has this default stance that can, can kind of migrate its way into the account.

32:42François ProulxSo I wish we had it today.

32:45Chris RomeoBut I think there's hope for the future that, that somebody will figure out how to— and I agree with what you're saying, that you can always communicate better. But we can say that about everything, like Everything I have ever experienced in my career, I could say they could have communicated that better to us. And nobody ever does. No one ever— you never have a situation where you're like, they really overcommunicated that to us. It was great. I knew all about it. Like, and there's, there's lots of psychology reasons for why that is, because it takes 72 times for somebody to hear something before it actually clicks. And they're like, oh, I understand now. This makes sense to me. But that's a whole other tangent. So, um, so we talked about the infrastructure side. How about the open source folks that are out there that are just, uh, and I love to always set this up with, they're not getting paid for what they're building, a component that the internet is probably relying upon. Their salary is zero. They're spending their nights and weekends making this work. Like, what are some things they could do to just get some quick wins on improving their pipeline?

33:46François ProulxYeah, I think that's why we built this open source, mind you, to kind of go, you know, with the community scanner, which not that there was nothing out there existing before we wrote that scanner, but we wrote it because we thought that there was still an opportunity to have something that was indeed more kind of developer-focused, because many of those tools that existed before we did Poutine were very much like security kind of like hardcore security researcher tools.

34:22Robert HurlbutYeah.

34:23François ProulxOr even like literally kind of what I would call like security research workbench, like very much like super advanced static analysis tools. Like no developer would just use that. So we made it batteries included, very easy to get results and get very good remediation. So we spent a lot of time with the documentation to give very good, best and bad practice, kind of good and bad examples across many things, GitHub Action, GitLab, Azure DevOps, Tekton pipelines. So we did all that and we're speaking, you know, I spoke at OWASP AppSec USA and at OpenSSF. So doing that kind of to reach out more proactively to developer audiences.

35:13Chris RomeoOkay. So just to quickly touch on Poutine, if I am a developer of an open source component, let's say I'm a project lead or whatever, what do I do with Poutine? Do I just grab it and then point it at my repo, like give it a URL where my repo is and it does the rest, it does the magic? And then is that, tell us a little bit more about how that works. And then what do I get as a developer? Do I get a punch list of problems? With some solutions suggested, or do I just get problems? Like, what do I— how does Poutine work?

35:42François ProulxYeah, so you can use it as a Docker or like using, if you're on Mac, Homebrew or whatever. And you indeed just point it to an entire GitHub org. By default, you need to give it just like read-only credentials to that it'll scan public repos. But if you want to scan private repos, you can do that as well. But most people just scan the public repos because that's the attack surface you want to focus first, right? They just give it like read token to GitHub, give it an org or a specific repo. We've tested it on orgs that have upwards of 5,000 repos, and the results come in within just 5, no more than 5 minutes for this kind of scale. So most orgs that have couple dozen public repos. It's extremely fast. And again, the remediation guidance for the results, we've made them as good as it can be. It doesn't do auto-fix or things like that. But what we did also, we wrote kind of a capture the flag training program that we called Messy Poutine, which allows people to play with exploiting those things in practice. So it's a GitHub org that has dozens of vulnerable pipelines. You can go in and get flags to prove that you successfully exploited them. So I'm doing the rounds of like those kind of capture the flag kind of training events and things like that to teach people about this kind of thing, which, you know, to be fair, by now most people know that you should not concatenate SQL strings and whatnot, gets kind of boring. So we need to move on maybe.

37:25Chris RomeoYeah, something new to push people a little bit. And this is, this is, this has been an interesting conversation for me because I was just having a conversation with a journalist that wrote an article for Reversing Labs where I was talking about the intersection of threat modeling and supply chain. And one of the things that I didn't, I didn't think about the pipeline. I was interviewed now, I'd probably have a little different perspective. I'd be talking a little bit more about pipelines. But one of the things that I shared with them is that made it into the article is we never threat model the components that we rely upon. And you could say we never threat model the build pipelines of the components we rely upon. When it's, especially for open source components, it's public. We could technically threat model an open source component. We could take our top 5 components that we use across our organization and we could do our own threat model of them. Same thing with, and then we could even extend that and look at the build pipelines a little bit and use Poutine as an input into that. And so just, it kind of, and it just kind of, I guess my eyes were open the more I was thinking about it, that, you know, scanning with a tool like Poutine is powerful. Bringing threat modeling into the conversation is also a powerful way that we can make the software supply chain better. Because you said right off the top, SCA and SBOMs, are tired. Threat modeling and poutine are wired.

38:53François ProulxYeah, if I can just say something about the SBOM, like as good as the SBOM is, it's only in 99, honestly, like I think I've never seen except extremely rare cases, SBOMs only care about the dependencies that go on to the artifact that is, you know, published. But there's nowhere in those SBOM does it mention the dependencies that are in your build pipeline. So that's, you know, I mean, I was talking to, sorry, I'm blanking, the gentleman that is behind CycloneDX, Steve.

39:30Chris RomeoSteve Springett.

39:31François ProulxYeah, exactly. I was talking with him and, you know, there is something in CycloneDX just for that, but basically no one uses it. And Poutine is, to my knowledge, one of the very few tools that can expectively output nsbomb, but it's not outputting CycloneDX, but at least it outputs the kind of the dependencies that it sees. And then you could kind of recursively crawl, and that's exactly what we've done at scale. So we kind of recursively crawl all the dependencies we organically discover across those millions of repos. So not just the kind of first-level workflow, the dependencies, and then the dependencies of the workflows, So it can go very deep. So I show a graph, kind of a, we brought it into Neo4j, a graph database, and then we can see that if we find a vulnerable GitHub Action plugin that is used by, you know, thousands of workflows and those workflows touch, so it's like the attack path, the transitive attack path can grow wildly, you know?

40:41Chris RomeoVery cool. Well, thanks for sharing the story and the background and, and helping us to walk through it and see how it comes together. But with that, it's time for the lightning round. This is the second edition. So it's a set of questions you've only heard a few times before, listeners. Robert, take it away.

40:59Robert HurlbutAll right. Yeah, we have 3 questions as well as we've done before. So the first one is shift left. Is that a legitimate concept or a joke?

41:09François ProulxWell, I certainly wouldn't say a joke, but it's overused by marketing departments, I would say now. Just getting a bit tired. What does it really mean for every vendor? I mean, at its core, I do embrace the kind of concept when it was first invented, but I mean, just think about security when, you know, new threat model and all that, but like shift left, I don't know. I don't use that term personally. Like it's not in my lingo day to day.

41:47Robert HurlbutOkay. Second is, is there a conference talk that you recommend folks find on YouTube?

41:54François ProulxOh yeah. So there's one, in fact, there's very good timing, literally yesterday, Like while we're recording that, something in October, there is the DEF CON YouTube channel just released the videos for this year's DEF CON. And one of which that I love is called Grand Theft Action. Just going back to this GitHub Actions stuff, Grand Theft Actions by Adnan Khan. And so amazing, amazing talk. where he's going into essentially similar research. So we're fellow researchers, we collaborate from time to time on different things, but he did something similar to our research and found self-hosted runners on those projects and, you know, got to have access to top secret documents at Intel with the schematics of CPUs, by using the techniques that I effectively talked about. So that's an amazing talk that I highly recommend.

43:02Robert HurlbutGreat, very timely. Who is somebody that our listeners should know about in AppSec that they probably haven't ever heard of?

43:13François ProulxWell, I'm pretty sure many people in AppSec have heard of Clint Gibbler, but I'm not sure you've had Clint on your show. But I mean, for those of you who don't know already, like he makes a newsletter called TLDR Sec, which is amazing. There is nothing compares to his newsletter. That is where I find basically everything that I need to know about. So Clint is an amazing guy that just people gravitate towards. Like now he's just like the center of universe to gather all the good stuff. So a very good person to know about and keep an eye on what he does.

44:00Chris RomeoVery good. All right, well, Francois, how about a key takeaway, a call to action? How do you want to end our conversation here?

44:09François ProulxWell, there is so much to learn about, you know, securing those pipelines. We wrote a number of articles on that topic, so you can go and check our blogs, boostsecurity.io/blog. Blog. A number of the open source projects that I already mentioned also have kind of an article to talk about those. So yeah, I would think if someone that listens to this is curious about learning more, I would really tell them to look at Poutine and Messy Poutine. Messy Poutine is really going to show you how dead easy it can be to exploit those things, and then you'll get scared, like, just as much as I do. When I first, like, discovered that, and I could see that I could exploit, like, popular projects, I couldn't stop, like, you know, doing responsible disclosure, because it seems unethical not to.

45:10Chris RomeoWell, François, thank you for joining us again on the Application Security Podcast, and for walking us through this story. I know I learned a lot in this conversation. And so we look forward to your third visit to the podcast next year after you create something else that's awesome.

45:28François ProulxThank you.