Brett Smith -- Security is a Necessary Evil

0:00Chris RomeoBrett Smith is a software architect, engineer, developer with 20+ years of experience. His specialties include automation, continuous integration, delivery, testing, deployment, plus he has expertise in Linux packaging and tool design. Brett joins us to discuss why he hates security. Now, don't hang up right now. There's a lot more to his thoughts on security. He also has vast knowledge building a secure and cutting-edge build pipeline, and so he's going to walk you through the different phases and things that you need to be thinking about when building a secure build pipeline. We hope you enjoy this conversation with Brett Smith.

0:42You are now listening to the Application Security Podcast brought to you by Security Journey. When you finish this episode, check out our other show, High Five, to stay up to date with all the hot AppSec news.

0:53Chris RomeoHey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo, Chief Security Officer at Security Journey and also co-host of the podcast. I'm flying solo today because Robert is somewhere traveling the great American outdoors, solving AppSec problems here and there and everywhere. But I'm excited today to be joined by Brett Smith.

1:17Brett SmithHi, Chris.

1:18Chris RomeoI'm going to get to asking Brett his origin story in just a second. But Brett and I met at an unconference where we were both speaking, and a couple of things he said really caught my attention. I'm like, I got to interview this guy because there's a few things we got to unpack. But we're going to, and we'll get there in just a second. But Brett, people like to know where our guests are coming from. And so, if you would share your origin story and how security has intersected what's happened in your career?

1:48Brett SmithSo, um, I'm not, uh, I'm not like a straight out of college into computer science guy. I'm, um, I took the long way in. Um, when I got done with high school, I started, uh, college, got bored, quit college, and, uh, went and became an electrician. Um, I was an electrician because I was tired of working on cars with my father. Um, and then so, but the whole time I was, I've been playing with computers my whole life. Right? I mean, since I was like 12, 13, no, not even, 11. I had, I worked on Apple IIs. I mean, I had an 8086. I had, I was building my own computers all through the '90s. I was installing Linux on them, right? But I'm working as an electrician. My back's hurting one day. I go to the doctor and I go to the doctor and I go, Doc, man, my back's killing me. What can we do? And he goes, do you wear a tool belt? And I said, yeah. He says, yeah, do you carry a toolbox? I said, yeah. He says, do you go up and down ladders all day? I said, yeah. He says, get another line of work. So I went straight to ECPI and signed up. And then within like a couple weeks of ECPI, I had— I got a job because it's, you know, it's the bubble. It's '97, '98. I mean, they were hiring anybody. And I got a job as sysadmin, a junior sysadmin. And I followed this pattern in a lot of the companies. I'm the junior sysadmin, the lead sysadmin quits, and I become the lead sysadmin, and I'm not ready.

3:10Chris RomeoRight.

3:12Brett SmithSo that was the first time I ran into security because we were ripping out NT 4 and putting Windows 2000 in. And I had to do all the admin stuff and all the automation around that and the security there. And I had Macs and I had to secure the Macs, which was impossible at that time. And so that was like my first run-in with, oh my God, I've got to keep things secure. The kids, you know, in the classes would do anything to those machines.

3:37Chris RomeoYeah.

3:38Brett SmithSo we moved forward a couple of years and I joined a startup, junior admin, lead admin quits, I become the senior admin. And this time I've got Cisco VPNs and I've got, you know, I got a data center and I got to keep all that secure. And I'm, you know, running through doing firewall rules, iptables stuff. Anyway, so that, you know, was second sysadmin job. And I'm like, you know what, I'm tired of being a sysadmin. This is, This is ridiculous. You got a pager, you're getting up at 2 in the morning and riding your motorcycle all the way across town to go into the data center and go fix something that went down. I was like, this is stupid. So I go to IBM and I become a tester, right? And so testing's great. Like, I didn't have to worry about security at all. Felt relieved. Wasn't making a lot of money. Went to another startup, admin. Lead admin quit. I became the senior admin again. Security in my life again.

4:33Oh, no.

4:33Brett SmithYou know, the one where you've got the SSH tunnel to the back door of the data center just in case the VPN goes down, right? And then violation nowadays, but it saved me that day. So I didn't like that again. Well, it was a startup, it died. Most of my startups died. And I went back to IBM to test again. And then I was tired of testing and I said, okay, well, I'm gonna go get some money. And so I went and became a senior sysadmin at a company that was getting ready to go public. And that's when I really ran into security headlong, like bam, right there, right? And we had— we didn't have a lot of money and we had to build out point-to-point connections with our customers. So we built out iptables, 2 iptables servers on pizza boxes, right? We wrote a bunch of Python to keep them in sync, right? If one died, they'd flip over. And then they had a bunch of monitors back into the apps so that we could tell how to load balance it. We basically wrote our own F5. That was for security there. Then I had to start taking the production data and shipping it to Arizona into our warm standby. I had to secure that pipeline and I had to worry about all that. Then PCI DSS hit. And I had to go segment off the network. So I went and got Juniper certified, put in all these Juniper switches, built out all these ACLs. Man, I was burnt. By the time that happened, I was done with security. I'm like, I'm tired of this. I had to use a jump box just to get to the other jump box just to get in to work on stuff, right? I'm like, this is ridiculous. So I finished up the job with all the ACLs and I didn't put any notes in there at all.

6:20Oh, wow.

6:21Brett SmithNo notes, just tons of security rules. So I quit that and went to become a developer, 'cause developers don't have to care about security, right? So I'm at this company, another startup called RPATH, and I'm babysitting the Linux distributions. And we had 2 Linux distros that were RPATH Linux, right? And then we had CentOS and several flavors of CentOS, several flavors of SUSE. And so after, we had a couple of customers still using our RPATH Linuxes. So every time a CVE came out, I had to go backport the patch into our distros, right? So when there was, you know, like a really bad one, right? And so I'm patching OpenSSL one day and I'm like, oh God, security, it's back. So, but it still wasn't too bad. It was pretty automated and I felt pretty cool about it. And then that startup died. You see a pattern here. And SAS bought us, not the company. They bought the IP, and they gave us all jobs. So the stocks I had were completely useless. But anyway, so now we get to where we are now, right? So I've been at SAS for 10 years, the longest I've ever been anywhere. building out this next-gen pipeline, right? Um, and the original pipeline is, is pretty insecure. Uh, yeah. And so when I was looking at, um, SLSA, right, which is a security spec that has been put out by— from Google, uh, to help with the executive order to secure the supply chains, and I was going through all the stuff in SLSA and comparing it to our supply chain. And I was just like, oh my God, we are like open everywhere. There's more attack vectors here than there are people in this company. Right. And I was like, oh man. So then I was, as I've been the past, what, 2 years trying to figure out what's the next pipeline look like and how do we keep it secure? So that's kind of where I am.

8:29Chris RomeoAnd so you wouldn't, you wouldn't give yourself the moniker or title of security person, Dan. Is that what I'm hearing?

8:39Brett SmithNo, no, I'm architect. But you know what? Architects have to worry about security, and if they don't, they're in trouble, right? If there's an attack vector in the pipeline I designed, right, and it gets exploited, I gotta find a scapegoat or it's gonna be me. So, you know, maybe one of my coworkers will be like, He was supposed to have known.

9:00Chris RomeoI wouldn't do that.

9:01Brett SmithI would take the heat, but still, you know, it scares the daylights out of me, you know? My brother's a Marine. He works in crypto and stuff, and they use SolarWinds, okay? And they have large ammunition things, and the SolarWinds with the vulnerability that got to them through the supply chain. So I can't have that happen. We've got government contracts, and I can't, I can't let our customers be affected by something that I didn't, like, go through the trouble to secure. So, it's kind of where we are.

9:34Chris RomeoYeah, no, I mean, it makes sense. And it was fun to hear your origin story there because I'm from the same vintage as you are. And so, you're talking Apple IIs and building computers in the late '80s and into the '90s. And, you know, so I was in the same place you were from that perspective. Like, that's part of my history. And I got to application security from the sysadmin path, as well. Like, that was my, you know, and someday we'll get some developers to come on and debate. You and I will be the sysadmin group, and we'll have some developers that got to security, and we'll debate who has the better foundation. But I feel like as a sysadmin, I learned the network, I learned TCP/IP, I learned Linux.

10:15Yep.

10:16Chris RomeoWhen it came time to try to secure these things, I understood the foundational principles for how they worked. and what all the protocols did and how you wired them up and made them work, sometimes in an insecure manner. But transitioning to a security person, it was like, okay, now I put security on top of what I know about Windows NT or Windows 2000 or whatever. And then I'm able to, you know, really become a security engineer because I know the underlying pieces as well.

10:42Brett SmithRight. So one of the spots in my journey where I was running away from security was when I was a tester at IBM. Honestly, what I did at IBM as a tester set me up for what I've gotta do now, right? And I'll tie these 2 together for you. So I'm working at IBM testing SCSI systems and storage, right? And it's all mostly Linux. I'm like the Linux guy by now. Like, remember the story I said Windows 2000? That was one job. It was the last time I ever worked on Windows, right? I've been Red Hat and SUSE and Linux my whole career, basically, right?

11:17Chris RomeoYeah.

11:19Brett SmithSo, as a side hustle, I've got this whole rack of servers that are idle, right? And because we're not testing them anymore, they're out of date. So, I'm building Linux from scratch over there, right? Every day and scripting it up, right? And then I'm installing all the stuff on the SCSI and running the tests, right? And so, the Linux from scratch stuff taught me how to compile software and how to patch software, right? So, now you flash forward to our path and I'm like, patching our old stuff, right? But you take the testing frameworks, right, and you take the fact that I know how testers work and I know what all the testing does, now I am set up with a platform I can go secure it, right? I can go make this secure. Because I'm going to tell you the truth, you know, if you've got— the testing framework has to be secure too. It can't just be—

12:12Chris RomeoYeah.

12:13Brett Smiththe software that we produce, you know, the supply chain has to be secure. Yeah.

12:17Chris RomeoI mean, there is a weakest link, right? And there is always a weakest link. And that somehow is what attackers are able to sniff out and figure out wherever the weakest link is. And we've seen it in the build pipelines in the last couple of years. You mentioned SolarWinds. There's been a few other examples. Like, we know that's where attackers' heads are going. And so, Brett, you know, one of the things that really, when I first met you, that caught my attention, and I told you this at the time, like, we got to have a podcast interview about this, You were explaining to me why you hate security. And I was like, okay, this is a message. This is something that security people need to hear. And also developers that are being sent requirements and things they have to do from security, they may want to join in and say, yeah, yeah, yeah, I agree with Brett. But I think security professionals need to hear this too, because we so often say and look at things from the perspective of, Hey, we're from security. Hey, Brett, you know, I'm from security. I got all the answers, when in fact, we really don't. But I really want to understand why you've come to this conclusion at this point.

13:23Brett SmithSo, I hate security. I've had enough of it. And it's even worse now because, you know, I'm about 2 years into this journey of securing the pipeline, which is traditionally not something you secure. Well, okay. When you go out and do something, if you don't have to worry about security, it usually is really easy to go do and get done. You just go out and you blast it. Then you got your stuff and it's running and you're like, look at my pipeline, it's awesome, it's doing all this stuff. But then they're like, you got to go put authentication on that endpoint. You're like, how am I going to get the authentication? How am I going to get the password to the robot so the robot can log in?

14:07Chris RomeoYeah.

14:08Brett SmithThen you're like, okay, well, I'll give them a key. Well, if you give a developer a personal access token, the first thing he's going to do is check it into a Git repo. Then you're like, now this token has to be flipped over, changed, and I got to erase this out of the Git history. Why'd you do that? Now you're in pain and you got to get— Then basically what it boils down to is key management is hard.

14:36Right?

14:36Brett SmithAnd we talked about that. And key management is such a pain. And if I didn't have to do it, man, stuff would be so much easier. And we all want life to be easier, right? And you take something that used to take 15 minutes to go do, and now it takes you 2 days because you got to think about the security while you're doing it, right? And you can't get away from it. Listen to my story. Like, I kept running away from it. I was like, I am tired of working on security. I've had enough of this stuff.

15:02Yeah.

15:02Brett SmithI want to go somewhere like a 20-dude— I want to go to a 20-dude VC-backed startup and just do whatever I want, right? Well, if you do that these days, you're going to be unemployed. So, yeah, I hate security, but it's a necessary evil, right? It is the necessary evil. If we don't think about it upfront, it makes it even harder to implement later, right? You went and built out this beautiful pipeline, it's got no authentication on it. Well, going and putting it on it later is really hard, and then it ends up bolted on instead of being in it, Right? And so with this next pipeline, we're building it in as it goes, right? And, um, and we're shift— how do I put it? We're trying to modernize as much as we can, right? It's OIDC, it's OAuth 2, it's JWTs that are short-lived, right? The robots talk to the robots, and when the robots are talking to each other, they're using the JWT that's really short-lived. So if you, Chris, who I know is notorious for this crap, goes and grabs my JWT, you got about 30 seconds to use that JWT before it's null, right? And so I reduce my attack vectors, right? And then I'd, you know, start working on, uh, micro-segmentation of my network, right? If you don't think about micro-segmentation of your network to begin with, you can't do it later, right? It's just you can't do it later. And so yeah, that's why I hate security. My life used to be a heck of a lot easier.

16:24Chris RomeoAll right, so here's the title I have for your I don't know if it would be biography or autobiography, but why I hate security, but it's a necessary evil. It's kind of like what I see is your story. We'll call it the Brett Smith story. Maybe it will get picked up on some movie network somewhere.

16:41Brett SmithSo my nickname at work is Smitty, obviously. And we had a business plan one time. My lead tester was going to follow me around and write down stuff I said and make t-shirts It's going to be awesome. We're going to be rich. And here I am doing security again.

17:05Chris RomeoThat's a good startup idea right there. Just follow somebody around and make t-shirts. We need an app that just listens all the time to us. So, all right. Well, let's— I know a big focus of what you've been thinking about and really working on in this new pipeline has been the software supply chain and all of the number of issues that come along with that. And so, let's start walking through some of the things that have been important important/crucial to you as you're designing this new pipeline from the beginning and making it secure along the way. So, the first thing that you were mentioning to me before was verification. Let's talk about what is verification and why do we need it?

17:45Brett SmithSo, verification is one of the things that in supply chains we don't do, right? So, if you have developers, right, you have vulnerabilities. And then, if you have a supply chain, you have attack vectors, right? And if you have a supply chain and developers, at some point, those developers are gonna curl pipe bash stuff into your containers, right? And they're not gonna verify the SHA-1, and they're not even gonna know where it came from. They're just— it's something they copied off of Stack Overflow and pasted into their Dockerfile, right? So, verification, right, means you onboard all of your third-party stuff and your first-party tools first. You onboard them, you create attestations for them, right? In Salsa Level 4, you sign those attestations with a key. Remember key management? I love security. So you sign the attestation with a key and you put it in signed provenance over to the— in a service, right? And then when your robots come along to go do the work, the robots grab the signed provenance and the signed provenance says, this binary has this SHA sum, and this signed provenance is signed by somebody I trust, so I can trust the SHA sum of this binary. Now we can use it, right? So we take that, right, and then we take it farther, because now we're talking about supply chains. Supply chains are big, right, or small, depends, right? They usually cover a lot of different stuff. And then you get into what do we have to verify? We've got to verify the environment, the materials, the process, and the artifact, right? So it's not only It's not only the binary I'm checking on, right, which is the materials or the source code that I'm checking the SHA-1 of, right? It's the commands I'm gonna run with those binaries that you've gotta go check, right? So I've gotta, in that signed provenance, I have a record of all the commands I'm gonna run when I pull that binary down, and I'm gonna SHA those and make sure that I'm not running some command that's not in that provenance, right? And that's part of the process.

19:47Yeah.

19:47Brett SmithAnd then I'm gonna drag out a bunch of environment variables and I'm gonna make sure that the environment I'm running on, right, the architecture, the VM, I'm gonna make sure that is what it says it is before I do anything, right? 'Cause all of that's attack vectors. So that's what verification is. In Salsa words, it's signed provenance. And yeah, and it's hard. The tooling for it is not done. Intodo is one of the, the open source things that we could use for this. They've got a spec and they've got Go and some Python libraries. Spec's really good, you should read the Intoto stuff. Recorr from Sigstore is a transparency log that they don't really want you to store provenance in, but guess what? You could just jam provenance right in there. Because, you know, if you've got a developer and you've got a database, inevitably they turn into a blob store. I did that. We all have.

20:46Chris RomeoCome on, we all have.

20:48Brett SmithYeah, I know. So anyway, there's— the tooling's coming along, but a lot of the tooling's nascent. You know, the guys are just getting started on all this stuff. So, but verification is really important, and you should start— I mean, at least if you're curl pipe bashing something off the internet, you should at least SHA-1 it before you run it. Right? I mean, I don't know what else to tell you.

21:11Chris RomeoSo, I mean, that's kind of like a bare minimum level of a check. Okay, so SBOMs have been all the rage over the last year and 18 months or whatever. They're being pitched as the answer. And so do you see SBOMs as truly solving the problem here? Or is there— is it SBOM plus some more stuff that's going to be required to really solve this?

21:37Brett SmithYeah, so SBOMs are cool. We should do them. And the tooling for creating SBOMs is getting pretty good. And most of the big major— most of the major companies like Black Duck and Snyk and Veracode are starting to support generating SBOMs from your scans. And but the thing about an SBOM is, is that it's a static list of your dependencies, your copyrights, and your licenses. And so I build my binary, right? And at that time I built my binary, I create my SBOM, and these two are static. This is immutable and this is not gonna change 'cause this is immutable, right? The problem is that the vulnerabilities change, right? So the vulnerabilities change every day. And so there's a new spec, there's a spec out called VEX. VEX is a Vulnerability Exploit Exchange. Okay, so now that I've said it, now it makes more sense, right? What I do is I take my SBOM and I rerun the vulnerability report against it and create a new VEX, right? And then in the VEX, I can go and say, yes, there's 2,000 CVEs. Please don't freak out, okay? Because your customer's gonna freak. If you give the customer an SBOM and they run the vulnerability report against it, they get 2,000 CVEs, they're gonna freak out, dude. They're gonna call your IT desk and they're gonna be like, what is this? What did you sell me?

22:57Chris RomeoOkay.

22:57Brett SmithAnd then what I got to do is I got to go through the VEX and I got to mark things that are not exploitable through my code path, right? And then I take the VEX and the SBOM and I smash them together and I get a vulnerability report for the customer that doesn't scare them, like, you know, scare the life out of them, right?

23:14Chris RomeoHow do you get to that exploitability and being able to measure that? Because I see that as one of the big things that we need to do as an industry. We need— because we continue to go down this road of saying, Oh, there's 2,000 CVEs in this and everybody's freaking out and nobody knows what to fix and there's no prioritization there. How are you— how would you recommend being able to gauge exploitability in a codebase?

23:39Brett SmithSo there's a couple of things you can do and you can pay a nice scanning company like Sysdig And they've got a runtime, or, well, it used to be Twistlock. I guess it's Prisma Cloud now. They've got runtime analyzers, right? So you take your Kubernetes cluster and you throw these runtime analyzers on your— on the nodes, right? And then you fire up all your code, right? And then they know what's loaded into memory and what CVEs go with what's loaded into memory, right?

24:13Mm-hmm.

24:13Brett SmithThen you come around with ThreatDragon or some threat modeler, right? You know about this stuff because you were just doing this the other weekend. And you throw the threat model at it, and then you do some pen testing, and then you gather the report up out from underneath, and these 2 pieces of software will tell you whether you were executable or not, right? And then you can build a vulnerability report from that that's considerably smaller, right?

24:42Chris RomeoYeah.

24:43Brett SmithAnd that would really, I mean, that's really one of the best ways to do that. Now, that's expensive, right? So there is an open-source path, but much like all open-source type stuff, it's a box of Legos. So you have to— you got to build the Death Star out of a box of Legos. Good luck with that with no instructions. But eBPF runs in the kernel, and you can write eBPF modules that run in there and can do security scanning live while your stuff's running.

25:16Okay.

25:16Brett SmithRight? So the open-source way to do this, like if I'm a startup and I got a bunch of VC money, I'm employing 20 of my best friends for the next 2 years, is we go write a bunch of eBPF modules that do the security scanning kind of just like what I said, right? Every time there's an exec call from one of the processes, the eBPF catches it and you can log it and figure out, or it can do something with it. And then you, That's how you do it. You got— it's a mixture of runtime and your penetration and your threat modeling is the way I think we could do this. And oh, and we're gonna do this against the supply chain, not just the software, right? Because the supply chain's got the same— got more holes in it. So that's kind of what I'm thinking, and that's kind of where I'm moving towards.

26:03Chris RomeoAnd with eBPF, this is gonna be more from like, Linux-based applications that are running on top of an operating system and not web, or is there a web component to what eBPF could do for us?

26:17Brett SmitheBPF's kernel level, right? So anything that sits on top of it, on top of the Linux kernel, has to make calls through the Linux kernel, and that's where you catch those. No, it's probably not gonna catch like the WASM stuff that's running in your browser, right? So, like, from an AppSec type thing where you've got this, you know, you download all this WASM to the guy's local browser, that's probably not going to help as much there. But, you know, I don't care about the developer or the guy with the web browser. I care about my supply chain, Chris. Come on now. Yeah.

26:53Chris RomeoNo, yeah, I get what you're saying. You're focused on the supply chain side and the build infrastructure and those things. And I just took a right turn towards running web applications. I took a right turn towards what's coming out of the pipeline. And you're drawing me back into the pipeline, though, and saying, like, we're focused on protecting the pipeline. There's other stuff that we can use. And I know there's other stuff we can use, like RASP and whatnot, for runtime protection in the web app once it's pushed out of the pipeline. But I get what you're saying. We're turning the ship back and saying, no, let's look at the pipeline and protect the pipeline itself.

27:28Brett SmithRight. And then, so, like, the developers So if you're in a small shop and you're doing real DevOps, the developers need to hear what I'm talking about, right? Because they need to secure— they're the ones that are— they're building their pipeline themselves, right? Yep. And, you know, they're their own ops people. And then they're the ones that are very vulnerable to this, right? Because they're the ones that— remember when I told you I hate security? I was one of those guys. Like, I don't feel secure on this. What's the password on the Jenkins server? Password? Yep. Awesome. Did you send that in clear text? Yes. It's on a sticky note under my monitor. So, you know, those are the guys that need to hear these messages too, because, you know, it's not just big companies, right? It's the small companies need to secure their pipelines too. And again, I'm not an app developer, Chris. I develop robots that build stuff.

28:20Chris RomeoAutomated robots that are building stuff. So, you mentioned provenance a little bit in the— when we were talking about verification and then a little bit about Let's, let's talk just to just get— just talk for another minute or so about provenance so that we get kind of a working definition and understand how that fits into the bigger picture.

28:40Brett SmithRight. So imagine you're going to make chili, right? And you've got a recipe, right? And it's your grandmother's recipe, right? How do you prove it's your grandmother's recipe? Well, your grandmother wrote it on by hand on an index card, and you can tell it's her handwriting. and you know it's her, right?

29:01Chris RomeoOkay.

29:02Brett SmithOkay, flash forward into the future. You pick up your phone, you've got your grandmother's recipe, but it's typed up in somebody's, uh, uh, web browser. There's no signature, there's no pattern to the, the, you know, handwriting to it, right? How do you tell if it's that? Well, that recipe, we take it and we, um, create a piece of provenance for it, right? Provenance gives us the URL where it is, Right? It gives us maybe a SHA of all the text in the recipe, right? And then it gives us, you know, the color of the background, some more information, right? And then we take that and then we sign it with a cryptographic— we cryptographically sign it and we put it somewhere safe, right? And then so flash forward to I'm building the binary, right? I got Foo, right? Foo's got a source in Git. So, I record the hash of the source in Git, right, and maybe I check it out and I SHA-SUM the whole folder, right, or the tarball of the source, right? You SHA-SUM the tarball of the source. And then I know I'm gonna run make install, right? And then so I put those 2 things into the provenance, right, the make and make install. These are the only 2 commands I can run, all right? And then I SHA-SUM them and I Put the SHA in there, right? And then the binary that comes out of there is foo, and foo's got a SHA sum, right? And then I put that foo binary name in there, and I put the SHA sum next to it. I take that whole thing, and I sign it cryptographically, and I stuff it into my store, right? And then hopefully it's secure, like, and it doesn't have a password and password, right? And so that is signed provenance, right? Now I've created the provenance. I'm gonna build foo again, right? So I go and grab the provenance, pull it out, I go and get the source, I check to make sure the source tarball has the same SHA sum as the one the signed provenance— well, first I check to see if the signed provenance is signed and I trust it, right? And once I've done that, then I can start using the information in it, and I check all the SHA of everything, the commands, and I check all that before I do anything, then I run.

31:13Okay.

31:14Brett SmithAnd so that's what signed provenance is. And then, you know, that's— I made it sound like we could just do it, but it's not that easy. Key management's hard, and the tools are pretty new. So, but that's the basic gist of it.

31:28Chris RomeoBut at the end of the day, you have a repeatable— you can repeat a build process for a given commit in Git, for example, and you can check and verify that nothing has been modified. Through that time, and it's all cryptographically sealed and signed all the way from top to bottom.

31:49Brett SmithRight. And then we get into hermetic builds at that point, right? So now I've got, like, I know all of the materials that are going into this build. I know all the sources that are going into this build. I know all the commands I'm going to run, right? It's all detailed out to me. And then I'm going to gather all that stuff up, shut the network off, and run my build. Right? And that is a hermetic build. And then I guarantee— well, I'm supposed— that's supposed to guarantee that the binary that comes out of there is the same binary that I built the first time I built it, right?

32:21Chris RomeoMm-hmm.

32:21Brett SmithThey should match up. Another crazy idea I've run into, and you'll like this one, is I met up with a DevOps person, and she was telling me that they have 2 different build pipelines, right? I think she said one was Jenkins and one's like Tekton or something like that. And they run the source code through both of them and check the binary on the backside. And if the 2 binaries don't match, that is like out the door, right? So that was pretty interesting.

32:53Chris RomeoYeah, that's an interesting— it's a relatively simple security control to see if anything's been modified in there. It's not super high-tech or high-end, but it does get you a very quick answer. If they don't match, you know something is broken somewhere and you got to hit the brakes and stop for a minute and figure it out.

33:12Brett SmithYeah, no, it's probably Jenkins.

33:13Chris RomeoBut anyway, this might be the local leader of the Jenkins user group here, Brett Smith.

33:21Brett SmithDude, if there's anything I— if anything comes out of this next pipeline, there's no more Jenkins in my in my shop.

33:29Chris RomeoI've heard other people with that same idea. Back on the security side though, I know shift left. I think you might have an opinion on shift left and what the challenges are that come from that. So from an architect's perspective, what do you think of shift left?

33:47Brett SmithSo from an architect perspective, I want to shift left as far as I can, but there's a problem. Developers are lazy. And developers can't do all the stuff you need them to do, right? So, at some point, you start to shift everything left, right? And all of a sudden, it becomes the developer's burden to go scan his codebase, right, in Black Duck. Well, inevitably, he's going to name his project Foobar, right? And his product is Cuz, right? And you're like, wait, that doesn't line up. How do I— I don't understand.

34:24Chris RomeoRight.

34:24Brett SmithRight? And he's making his scans, he's running his reports, and then they just sit there on the server, and they're like, well, now what do I do with them, right? And so you swing around, right? And now, you know, take my instance. I got roughly 2,000 developers working at any given time, right? I've got hundreds of containers, in excess of probably 200 containers that I deliver to customers. I have hundreds of RPMs and Debian packages I deliver to customers, right?

34:51Chris RomeoYeah.

34:53Brett SmithAnd I got to go find all the scans for them, right? Now, how do I find the scans for them if the developers who, you know, we shifted so far left, the developers have to go, you know, run their own scans and they didn't name them correctly, right? So this is where the robots come in, right? And then what you want to do is, is that we want to come in with the robots, we want to automate the scanning process for the developers, right? And then this way they don't have to remember to go scan. And this time, we get to scan every time they push to the repo, right? So we tie this stuff— let's take GitHub for example, right? Pull request runs, we get an event, the event triggers the scans, right? The scans go, and if you've got newer scanner engines, they can talk back to the pull request, and they update the pull request with the results. The developer never has to leave GitHub to go see what he's got to fix. Right? Now we've shifted left, and the developer's like, hey, this isn't that hard. I think I can do this, right? So the next thing you do is you got this thing— GitHub's got Dependabot, but we were doing a sneak POC the other day, and they've got this great thing that ties back into GitHub, and it creates pull requests for you that update your libraries, right? So now the developer doesn't have to do anything but go look at the pull request and take his update, right? So now we're shifting left, and it's a healthy shift left, right? Because now the developer's not having to go out and find what the new latest library is, right? And the developer's not having to go and go to a different service to go look and see what his vulnerabilities are. And then you and I both know that that service is going to have 2,000 vulnerabilities about it, and he's not going to know where to check it, right? So that's where something like a runtime analysis comes in that can go in and, you know, the robot can go remediate the scan for the developer because the runtime information you got from the testing you were doing, right? So this is a whole— all starts to spin around like this, right? And then all the scans are named properly, right? And they've got the name, the version, and the release on them. And I can go and go, okay, I'm sending foo 1.0.0 to the customer. Let me go get the scans for foo 1.0.0. Let me go generate the SBOM. Let me go get the third-party notices, and then let me get the scans and the vulnerability report, the VEX, and then I've got it all tied together. And now if I've got 150 binaries going out, right, I can go get the 150 SBOMs, put them all together into a giant SBOM, and give this to the customer if I feel like that's what I want to do, right? It's an automated organization thing where I automatically organize stuff, and then the developers don't feel burdened because they have to go manually run these scans and then go manually remediate them. And, you know, you give them an IDE plugin that ties into your scanning software and it starts warning them before they even get to push the code that, you know, hey, you got a vulnerability here. And so, you know, that's how I feel about shift left is that, well, shift left is great, but too many people, they try and burden the developers with it and it doesn't work in a big company, right? It just doesn't work and it's not fair. And if I was a developer, I'd be like, I don't want to do that because I don't like security. Right.

38:04Chris RomeoAs we've established throughout this conversation that Brett doesn't like security.

38:09Right.

38:09Chris RomeoBut yeah, I mean, I think you've given us some different concepts and ideas to really ponder and taking security and it's almost security as a service as much as you possibly can get. But at the end of the day, you're making developers' lives easier and developers are going to appreciate that. They're going to appreciate not having to do busywork that could be automated by some type of a robot to get the results for them and put it, like you said, you know, tag everything back in their PR so that they can just live in the pull request, make the changes they need to do, everything runs again, they get the— they go green from a security perspective, boom, the PR is ready to be merged. And so I love that idea that you're leading with here about let's just keep it simple for the developers. Why make security hard for them?

39:01Brett SmithYeah. Right. If you make it hard for them, they won't do it. So like, I want my developers working on my electric sheep, okay? I don't want my developers worried about scanning the electric sheep, okay? I can automate scanning the electric sheep, and I can automate almost all of it. Now, they may have to go and help me with a remediation here or there, right? But I can automate the rest of it, right? And I put the sheep on the conveyor belt, and the sheep goes through the scanner, and then the sheep, you know, Stuff happens. Little robots run around and do stuff. Then the developer gets to do what he likes to do. Git pull, hack, hack, hack, git push, get his review, and that's his day. He gets to do what he wants to do, and I get to watch robots run around.

39:43Chris RomeoWhich can be fun to watch robots go do their work and make everything come together.

39:49Brett SmithYeah. Automation is The key in this.

39:54Yeah.

39:55Chris RomeoAnd that's, you know, that's, yeah, that's, that is sage advice for people that are not truly embracing full-on the value of the automation. You know, just in what I'm, what I'm taking away from you is just adding my SaaS tool to my build pipeline is not automation. It's a piece of automation. And you could say it's 10% of the problem, but there's a lot more that goes into making it successful than just adding a tool and saying, okay, now it's running in our pipeline.

40:22Brett SmithYeah, it's not enough. You've got to add all the tools. And then the challenge is, how do I not make the experience horrible, right, for the developer? Because adding all the tools in, usually every time you add a tool, you make it worse on the developer, right? So, you know, that's kind of the— it's The real challenge of architecting a pipeline is making it where it doesn't seem complicated and it's easy to use.

40:53Chris RomeoSo, from a key takeaway perspective or a call to action, we've talked about pipelines and verification and attestation and provenance and some of the challenges that come in shifting left if you don't do it correctly, and then even automation. But what would you offer to our audience as far as an action or something that they could do as a result of what you've shared with us today?

41:18Brett SmithSo, and this might not be the exact answer to your question, but the thing I've been thinking about, the thing I think about a lot is that I talk about all this security and securing the pipeline, right? And then what I need to do is dogfood that security. Right? And so once you get a secure pipeline, you should build all the tools in that pipeline with that secure pipeline, right? Because then you get all the stuff that you're getting for your Electric Sheep, you get free for your tooling chain, right? So you're getting your signed provenance, you're getting your scans, you're getting your remediation, you're getting your updated libraries, and then you know that your tools are being held to the same standard that your product's being held to. And I don't have developers out there building the Docker image on their desktop and pushing it to a registry, and that's one of the tools we use, that's got 15 curl pipe bash commands inside of that Dockerfile that they didn't check one of the SHA sums. That Docker image was built through the pipeline, signed provenance, hermetic build, out the door, goes back into the pipeline to help build the electric sheep. So there you go. That's what I would say is the takeaway. Dogfood.

42:29Chris RomeoYeah, and I guess I'll offer a call to action based on what I've heard in this conversation, and that is is get a handle on your pipelines, get an understanding for how you can secure these things, how you can make the developers' lives easier, because at the end of the day, that makes a more secure product application. Whatever's coming out the end of this thing is better if the pipeline is secure and set up to make developers successful. So, Brett, thanks for sharing your expertise with us about all of these things. And, you know, someday I hope to reach the point where you're like, you know what? I've changed my mind. I don't hate security. I love security. It's the best thing ever.

43:05Brett SmithNot gonna happen. We'll hold out hope.

43:10Thank you for listening to Security Journey's AppSec Podcast. You can find us on Twitter @AppSecPodcast, on LinkedIn as the Application Security Podcast, or on the web at www.securityjourney.com/resources/appsecpodcast. Find Chris on Twitter @edgerow. and Robert @RobertHurlbut. Remember, there are many application security paths, but only one destination.