JC Herz and Steve Springett — SBOMs and software supply chain assurance

0:00Chris RomeoJC Herz is the COO of ION Channel, a software logistics and supply chain assurance platform for critical infrastructure. She's a visiting fellow at George Mason's National Security Institute and co-chairs a Department of Commerce working group on software bills of materials for security-sensitive public and private sector enterprises. JC and Steve Springett join to talk all things software bill of materials. We define what an SBOM is and what it's used for. We talk threats that SBOM counters, who started it, and what the tie-in is with OWASP. JC concludes our time by explaining why now is the time that you must care about SBOMs.

0:40We hope you enjoy this conversation with JC Herz and Steve Springett.

0:45Chris RomeoAt Security Journey, we believe security is every developer's job. We work with our customers to help them build long-term sustainable security culture amongst all their developers. Our approach is to provide security education that's conversational, quick, hands-on, and fun.

1:03We don't do lectures.

1:04Chris RomeoInstead, we let the experts talk about what's important. Modules are quick, 10 to 20 minutes in length. We believe in hands-on experiments, builder and breaker style, that allow your developers to put what they learned into action. And lastly, fun. Training doesn't have to be boring. We make it engaging and fun for the developers. Visit www.securityjourney.com to sign up for a free trial of the Security Dojo.

1:27Hey folks, welcome to this episode of the Application Security Podcast.

1:47Chris RomeoThis is Chris Romeo, CEO of Security Journey and co-host of said podcast.

1:52I'm also joined by Robert Hurlbut. Hey Robert. Hey, Chris. Yeah, good to be here. Threat modeling architect. Glad that, uh, we have a couple of guests with us today. Yeah, definitely. We're gonna talk about a subject that I thought I knew something about, but now I'm realizing maybe I don't. So I'm looking forward to being educated here as well today. So JC, we're gonna start with your security origin story. It's always where we begin. We jump right in and we wanna know, how did you get into this crazy world of security?

2:20So my security journey begins in the Pentagon.

2:24Dun dun!

2:25I was working in the CIO's office to help create some policies around open source software. So, cast your mind back to the day when we were having these wars about whether open source was allowable within enterprise systems, and there are a bunch of big vendors running around trying to spread fear, uncertainty, and doubt about open source is GPL. If you use it, you'll have to put all your military code out on the internet. There was a lot of that jazz going on. So we had— we bought lawyers, guns, and money, right? So mission owners, lots of lawyers, and some funding to do an assay of, okay, well, how much open source is used in these systems? And what would we do if we had to live without it? And both of those answers were not the answers that the proprietary-only crowd wanted to hear. And we actually created a lot of policy to make the Defense Department safe for open source and vice versa. This led to this whole business of supply chain because the one thing that the open source did not have that the vendors had was someone standing behind them to say, we'll send you updates, we'll send you patches, we're going to be continuously responsible for maintaining this, or at least we say we are. The whole business of operation and maintenance O&M, right, in the geeky acquisition speak, came down to, all right, so we won the war. Open source won everywhere. It's there. Now we have to get away from this business of open source versus proprietary because open source is in everything. That distinction is no longer useful. We have to start making these distinctions about open source versus open source. How do we treat these projects as suppliers?

4:19Right.

4:20Which is really what they are, these communities of developers who are either maintaining or not maintaining their software, are suppliers. And the analogy that I like to use is that a Gemological Institute of America certified diamond and a Crazy Jewels grocery store vending machine ring are both technically jewelry, but you wouldn't necessarily want to put them in the same category. That's no longer useful. So if we are really serious about supply chain, which is in the news and an issue for all kinds of reasons related to telecommunications and other things, we really need to start to understand the quality and the maintenance histories of what we're using and whether those things are defective or whether they're vulnerable or whether they're actively maintained. And so that led to the founding of Ion Channel, which I help run, which really has to do with the continuous analysis of what's going on in the open-source supply chain so that we can help our customers who are generally in critical infrastructure, in energy, in telecom, and in defense understand not just the CVE checking, which is very cute and needs to happen, but what are some of the leading indicators of risk that tell you that maybe not today, maybe not tomorrow, but sometime you may want to not use that open-source component and use another one instead, or require that people meet certain standards for supply chain risk management. Some of this gets very bureaucratic and into these high assurance standards, but part of the work that I've been doing with Steve Springett at OWASP with SCVS has to do with how do you make all that stuff lightweight so that you have a way of assessing your capacity and your posture without having to hire, you know, 5 people to meet NIST 800-53 controls, right? Which none of the open, innovative people can really afford to do. So I think the ambition is to bring up the security posture the same way that a lot of critical infrastructure wants it to be, because we're all the suppliers, suppliers, suppliers. You're getting down to some very small organizations. but how not to crush the small innovators who really cannot afford these very heavyweight, bureaucratic, costly compliance regimes to get certified for this or that, which they're not going to do anyway.

6:51And so you mentioned the fact that Steve is also here, and so we're happy to welcome Steve back for his 3rd appearance on the Application Security Podcast, which puts him I was just thinking about some other people who are in that, Jim Manico, Adam Shostak, so it's a very distinguished group. If you want to hear Steve's previous appearances, you can hear him talk about Dependency Check and Dependency Track, that was in season 3. And in season 5, he did a discussion with us about an insider's checklist for software composition analysis. So Steve, we're glad to have you back with us as well.

7:24Thank you very much. Glad to be here and very privileged to be in such awesome company.

7:29JC, let's jump into this. When you were telling us as we were preparing, you were talking about how some of the government's own cyber assessment tools have critical software vulnerabilities. And so that's a— that was a pretty shocking statement, kind of a big— as a big picture idea here for me. So help us to understand more about that situation.

7:49So when COVID started, right, the whole lockdown started, Ion Channel began analyzing and monitoring a bunch of critical infrastructure capabilities, a lot of healthcare capabilities. And while we were at it, we said, well, let's just take some of the open source software that the government is maintaining and put it into analysis just to make sure that that's good. And I'm sure it will be. Well, it wasn't. So there were tools that the government was putting forward as cyber assessment tools. And one of them belongs to the Critical Infrastructure Security Agency. It's the CSET tool. And the other belongs to the National Institute of Standards and Technology, NIST, right? It has a supply chain interdependency tool, right? Which is a piece of software that you're supposed to use internally to put in who all your suppliers are, right? Which is operationally sensitive data. And then it will tell you something, it'll visualize for you kind of what your supply chain interdependencies are. The problem with this is that it has critical vulnerabilities in it, both critical and high CVEs, as well as a whole bunch of other supply chain risks, which people don't even govern on. And it is being made available as a binary download from a website maintained by the federal government. Now, if you're just maintaining an open source project as research and you say, okay, here's the source code, if you wanna take the source code build it yourself and run your own SAST and do your own assessment, that's fine, that's cool, that's groovy. The minute you compile something and you make it available as a binary from your own website that is not inspectable, you are a supplier and you should not be a risky one. And so Ion Channel emailed NIST to tell them about these findings. And the response that we got was, oh, well, you know, it's just a research project and there's a workaround because if you run this tool on a website on a computer that's not connected to the internet, it's not a problem. And I, as diplomatically as I could, responded with the fact that if someone downloads a binary piece of software, an executable, from a website and can install it on their computer in their workplace, it means A, there's very little in the way of system security in that enterprise, and B, it's connected to the internet. So why don't we, you know, go ahead and fix it? And you get down to it and it turns out that the Boston Consulting Group, which helped build this, hadn't updated it in 11 months. And a lot of these vulnerability issues come down to maintenance. We wrote a white paper about this. Essentially, boring is sexy, right? The biggest risk is boring. It's not about shadowy actors coming up with these custom-type Stuxnet attacks on enterprises that they've done reconnaissance and surveillance on. It's just a whole bunch of potholes not being filled. And organization— everyone's talking about backdoors, Huawei backdoors. You don't need backdoors. When you read the Finite State report on the Huawei software, there were so many non-maintained components in that system, like double-digit number of major versions behind, that you didn't need to put a backdoor or any malware in these systems. And it would be stupid to do so because if someone finds the malware, They can attribute it. It's much better from an information theory perspective to have the plausible deniability. No one can prove that you intentionally failed to maintain your software in such a way as you knew exactly how to exploit it. And yet, these are the facts on the ground. Maintenance is— no one gets promoted for it. This is the problem. Everyone gets promoted for new features. They get promoted for, we've rolled out this new capability. No one gets promoted on, wow, that's a very robust and well-maintained capability you have there. And it's not getting done. And so software ages like milk, not like fine wine, right? And these things are crumbling. And these incredibly sexy demoware projects that are not maintained— because if you run your pipeline SaaS, the tools that we've seen the federal government maintains, they have build passes. Badges on their GitHub repos. You know, the last time that build passing was done was 7 months ago, and I just found one of those at a different federal agency. So you could be passing at a point in time, but unless you're monitoring continuously, which is, I mean, vested interest, this is what Ion Channel does, but unless you're monitoring continuously, you have build passing, but you're critically vulnerable. So you're worse off than before because ignorance is not as bad as the illusion of knowledge. This is what we have to deal with. We maintain the maintenance records for all these components. There's one, Open Hospital. That's the great one. We did all these healthcare capabilities. There's an open-source project called Open Hospital. If you Google it, you'll see there's a website. It's everything you need to run a hospital in an austere environment. The website has all these pictures of these gorgeous African children. It's maintained by Informatici Senza Frontiere in Italy, like Gordon a great nonprofit. And these people claim that this hospital software is in 13 countries with 23 installations and has processed 425,000 patient records. And if you scan it today, it comes up green. There's no high-criticals, there's no viruses. We started monitoring this in April, And they had 3 Trojans in that software for 6 solid months. So you can't know that up until 2 months ago, this thing had 3 viruses in it, and all those patient records, which includes vaccinations, births, healthcare data, that was all compromisable. So the maintenance and, you know, days passing, days failing, mean time to remediation, which is a proxy for every other meaningful cybersecurity metric. If you don't have that and you're not measuring that and you're not holding people to that, you're nowhere. I think this is what the SBOM enables, is at least know what you have, like a software bill of materials, which is now required as a condition of FDA approval for medical devices. We're working with medical device manufacturers to figure out what the hell it means for them to produce one of these things and for a hospital system to consume one. But there's now contracting languages. Mayo Clinic, whose information assurance department I would stack up against any 3-letter agencies, they're amazing, they're elite. They now have terms and conditions in their contracts that say, if you want to use open-source components in your vendor solution that you're selling to the Mayo Clinic, That's awesome. None of those components are allowed to be non-maintained.

15:35Okay.

15:36And by non-maintained, we mean it hasn't been updated for a year, there's no point of contact in the package manager, and if a security issue is identified, it is not remediated within 30 days. And if you have one of these components in your vendor solution and you want to keep using it, congratulations, buddy. you get to maintain it.

15:57Yeah, a number of different thoughts and stuff as you were going through there. I was just kind of smiling a little bit like it's, you know, kind of a, I guess, a meta idea that there was a supply chain solution that had supply chain problems in the tool itself. And then their suggestion was, you should run this thing in the basement. Like, I remember, like, I've been in security for going on 24 years at this point. And I remember when that used to be a recommendation that we made, like in the early days of security, people would say, well, you know, you should just take it. We're just going to take it. We're going to bury it, you know, 10 floors below ground in a bunker, and we're going to run the computer there. So that's why we don't have to care about authentication. And there's going to be armed guards and everything else. And like, yeah, sure, if there's going to be armed guards, you're going to have all these other things. But then we had— somebody invented this thing called the internet. Like, well, then that strategy of protecting stuff was no longer possible.

16:54Yeah, and you probably had to do some work to get a piece of software onto that computer 10 levels down. Instead of just downloading it to your browser.

17:02Yeah, yeah, it's a different world now. It's like, I mean, I was involved with Common Criteria when it first came to the US. I was— I have a unique listing on my resume. I was part of the first commercial company that did a Common Criteria evaluation. We worked with the government as they oversaw You know, I mean, that would be a whole episode right there to itself about how they oversaw. But yeah, so, so yeah, it was a different world though that we lived in now. And it's like, it's amazing that that was kind of somebody's answer though, was like, let's take the guidance from 20 years ago and just try to, you know, we'll just run it not connected to the internet.

17:45Again, look, look at CMMC, right? This is a cybersecurity maturity metric for all defense contractors, so we're told. And it's a 3-year certification. Someone does an audit on you, they give you the stamp of approval, there's some holy water that someone from the Defense Acquisition Service sprinkles on you, and then you're good for 3 years. And I think one of the SCVS principles that's in the controls, right, these are real controls that actually, like, people in technology and industry in the commercial world can use, is Is this automatable? Is this a human telling you something? Is it a document drill, or did a machine process create this information? Ion Channel, we take down bills of materials that it's a 1,000-row spreadsheet with package name and version, which is the nicknames for software that some vendor is using from their repo. Part of the value proposition is solving the naming problem. To be able to actually resolve Joe's library version 1.2 to the Joe organization and the URL where this thing actually lives, and it's actually not called Joe's library, it's called my favorite library in the supply chain. Unless you can do that, you're really nowhere for vulnerability management. We see software bills of materials that are just filled with the output of people's spreadsheets. So it's beautifully structured data. The structure of it is gorgeous, but the quality of the underlying data is— it's just gibberish. It's, it's not going to match anything unless someone goes through by hand, right, to resolve, well, is this— let me check in the TAP, TAP, TAP, let me look in the NVD. And it doesn't really work either. I mean, that's the tragedy is people are burning hours on develop— let's write some Python scripts, right? That's the first one. Or, oh, the Python scripts aren't really working, let's do it by hand. But by hand doesn't work either because the NVD's data is not complete. Not only is there a bunch of CVEs that come from other places, but even the CVEs in the NVD aren't complete because there are third-party inclusions in the packages that are not, you know, Joe's library may not have any CVEs against it, but there's a whole bunch of things in a container for Joe's library or in a package in the package manager that go, that's a runtime dependency for Joe's library that are actually vulnerable. And we—

20:38That's because Joe's library is made up of component pieces. Joe took pieces of other people's open source And put them together into his own library.

20:48So that's an issue, the graph. So when we go through, we ingest the graph of everything in GitHub, everything in the package managers, all this other stuff. The problem is that if you have— this is especially an issue with containers. When you have a container, and to some degree a binary package as well, you have the piece of software that you want, the thing that is the label on the box. You're getting Joe's library. and Joe's Library has been assured, right? So you've done this independent assurance process of the thing, like Joe's Library as a name, you've looked it up somewhere and you're good to go. The thing in the box is perfect. The packing materials are toxic, right? So these vulnerability databases don't track the packaging materials. The packaging materials actually contain lead and all kinds of heavy metal and are radioactive, but they're not registered as contents. We'll look at a package, and there's a runtime dependency of the package. Essentially, it's a sidecar dependency. It's not in the thing. It's around the thing, but it comes with the thing. If you take the box and put the box in your enterprise, you have that vulnerability and the NVD won't tell you that. So there's a huge amount of false negative because we're not in our new paradigms for packaging software taking into account the fact that the packing peanuts are actually contents. And if you wanna take a box that has packing peanuts in it, you'd better assure that packing peanuts Because if they're vulnerable, so are you.

22:34So you mentioned something about— so people, in your experience, people are creating right now the list of the open source that's in their stuff manually, and they're putting that in a spreadsheet. Like, that's a common practice right now?

22:49I think that there's a bias when you get a bunch of technology people together as experienced and astute as you and Steve, right, that we live and breathe technology, and we're generally, if not on the bleeding edge, on the leading edge of what's going on. We assume that when you create a piece of software, you're going to set up a Git repo, you're going to put it into a CI/CD pipeline, and you're going to have these automated manifests, and that's just how the world works. But that is actually not the vast majority of deployed software, particularly in legacy systems. The rest of the whole world is very manual, and we even see things like manifests and requirements.txt and all kinds of dependency files, even in GitHub repos, that are made by hand. They don't actually— they're these artisanal things. The hard, boring problem that we solve is, how do you go from that list of things that someone made by hand, which could be like a— they call it a floss list, right? The Excel spreadsheet that contains your open-source components because your lawyers made you keep that list because you don't want to get crosswise with GPL. That's where a lot of this information is coming from because the enterprises that are maintaining these capabilities and selling them, and these could be global consulting companies, they have no provenance or chain of custody on their components. They have no idea where that stuff came from. They have a bunch of JAR files that are lying around. So to talk about going and forensically determining where their stuff came from, you can't do it. The data isn't there. The person who put it there hasn't worked at the organization for 4 and a half years. And that's the reality of the rest of the huge amount of software that actually exists in the world. not among the forward-leading, fast follower, or early adopters who are all on this podcast.

25:02It's a good reminder for us that we're not— that— and I hadn't even— I'd never actually thought of that, but it's a good reminder that not everybody does approach software from the same perspective that we do, meaning they're not doing those things that you talked about. So, Steve, I want to get you to come in here and talk about SBOM. So, we've mentioned SBOM a couple of times, and that's the thing that I said, that I thought I understood before we started the conversation, but I'd love to get some more perspective on kind of what that is to help us and our listeners.

25:33Yeah, it really is the list of ingredients. So when we talk about SBOMs, we are referring to a software bill of materials. And like any bill of materials, it is simply a list of ingredients of what, if I have a piece of software, what is in that thing, right? The majority of things that's going to be in there are going to be open source components, but SBOMs can also describe all your first-party components as well. So all your third-party and open source components, all your first-party components, and potentially some services that you also depend on, depending on if you actually care about including those, those dependencies as well. But really, it is the full list of ingredients. Once we actually have that, then we can do some really interesting things with that information, including a lot of the supply chain type things that JC has been referring to, right? Without the full SBOM, and we're really looking for that full, not only my direct dependencies, but my, all of my transitive dependencies and all of my runtime and environmental dependencies as well. That really should be included in the SBOM. And then once I have all that information, then like I said, you can do some really interesting things, as JC was alluding to, with different forms of analysis that you just can't get with a lot of tools today.

27:00And so yeah, I'm curious, JC, with SBOM, what would I— when I have that, what kind of threats could I counter if I have that list of ingredients, as Steve mentioned? What would that help me with?

27:12So, in the cybersecurity world, folks who focus on threat, there's a specific definition of threat, which is, you know, particular actors who are trying particular things. So, threat and vulnerability are both two sides of the same coin, right? So, if you have a software bill of materials, first of all, on the very basic level, you can figure out whether there's any known vulnerabilities associated with your stuff. So you're— look, do a CVE checker, right? This is table stakes at this point. Like, it's not even fancy. Beyond that, what you can start to do is understand what are the supply chain risks associated with each of those components. And, you know, there's different kinds of analysis you can do. You can go all the way from like a fully automated platform that's looking at maintenance histories, which we do, to, you know, by person researching who are these developers, right? In some cases, that's pertinent. One of the things that we do is flag change of control events. So if you have an open source component that's in your stuff, and you know it's in your stuff because you have a software bill of material, and you're using a, you know, a third-party capability like ours, or you're doing it yourself to look at, well, has this component changed control? Is there someone new who is now maintaining it? That's not necessarily a bad thing. Oftentimes, it's a great thing. You have a project that becomes an Apache project. There are some graduation exercises for open-source projects that become important enough that are great. However, sometimes you have a broadly used component with a maintainer who doesn't care anymore, who's bored with it, and, you know, some nice person offers to take it off his hands. And that is a security-relevant event that if you are maintaining an infrastructure that needs a high security posture, you should absolutely be aware of as soon as possible. And so that's the sort of monitoring that you can then do. The other more sophisticated stuff is, you know, we're looking at leading indicators of risk to say, What are the components that are going to have CVEs 8 to 12 months from now? Because the whole CVE process, vulnerability management as CVE remediation, I call it, and this is a technical term, the insane whack-a-mole gerbil wheel because the CVE by definition is a lagging indicator of risk, and there's no way you can ever win that. It's a Red Queen game. So, what we're looking for is, based on a lot of machine learning, if you look at the publication of a CVE, which that could also take 8 months, right? Because someone has to develop an exploit.

30:04Right.

30:05They have to document it. They have to submit it. Then the vendor or the maintainer has to do a fix before it's published, right? That's a long thing, right? There's a lot of exposure time. So, if you roll back the tape, what are some of the indicators or combinations of indicators in the supply chain? that tell you that there is a high likelihood that an exploit already exists for this component that will be revealed to the world in 8 to 12 months, because that's what you really wanna do. Because if you can do that, then you don't have to do the insane whack-a-mole gerbil wheel of CVE remediation because those components are not in your system. And that's ultimately where people should wanna go And it's where a software bill of materials sort of starts help getting them started. And I think on a first pass without using anything, you know, ultra elite, you can just start to look at your level of technical debt. Because even aside from which of these components are gonna have CVEs or not, or Ion Channel tells us this is this bad juju in here, you can see, well, if this thing, has a vulnerability, how easy is it going to be to fix it? If there's so much technical debt in your enterprise system that it's going to be hard for you to refactor it because the security update is going to break it, that is the first clue that you should be doing something different. From a threat perspective, I think we all have a lot to learn from the sort of left pad scenario. Right, is you have a broadly used component that for one reason or another is taken offline, no one can update, and while they can't update, while the shock and awe of that, all these other vulnerabilities are exploitable, right? So that would be, you know, a smart threat actor's approach to compromising systems wholesale. Unless you know that there's a left pad in your system— and why are we dependent on a library for indent again? This is supply chain risk management. This is not vulnerability CVE checking, although you have to do that. That's the crawl, walk, right, before you can run. If you don't have a software bill of materials that tells you what's in your stuff, you really cannot then analyze what your supply chain risk is the same way that a manufacturer would. So if I'm a manufacturer of a physical product and I have a critical component coming from a single factory on the tropical coastline of a politically unstable country, the part might be fine, but there is risk there. That's supplier risk that I need to understand. And so I think what, what the software bill of material gives you is a starting place to say, okay, so this is in this thing that I'm buying or this thing that I'm making, how can I then start to look at the known vulnerabilities and all those transitive dependencies that Steve mentioned, as well as some of, some of the other risks that I can either monitor or buy a service to monitor or do by hand or understand some other way?

33:30I want to switch gears and ask the OWASP question in just a second, but JC, I'm curious from your perspective, Why is now the time for SBOM? Like, security, application security has been around since the early 2000s, and we've had this problem since as long as software has existed. I think you could argue all the way back to the beginning of software, there were depend— as soon as someone invented a library, then you had this dependency, you know, vulnerability potential problem.

33:59Chris RomeoSo why is now the right time?

34:01Is there some flashpoint or something that's happened, or what's the driver for SBOM?

34:06I think it's a combination of things. One is that the threat landscape has changed dramatically. If you look at the notion of maintenance and you look at the ransomware attacks that have happened because people do not maintain their systems or because products aren't maintained, that's real to people now in a way that it wasn't before when you have the city of Baltimore or school systems or The other thing is we've moved from these attacks that have to do with compromise of confidentiality, right? So the Target attack, the Equifax attack, even as horrible as it was, it was a bunch of data about consumers that was exfiltrated. They compromised confidentiality. It did not compromise availability or reliability. When you start getting into these other -ilities, especially in things like electricity and power and utilities, that's a different realm, right, of where people are worried about. So, you know, my personal motto as a professional is preventing the zombie apocalypse every day, right? So, that is different. The other thing is that there's just regulatory pressure. So, when you have to submit a software bill of materials to get FDA approval for a medical device, That's roughly a kajillion dollars worth of industry that now has to pay attention. We're seeing the same requirements play out in, really, a lot of critical infrastructure, in energy, in anything nuclear, in defense, and in finance. I mean, in finance, the banks already will require a software bill of materials if you want to sell them some awesome AI/ML package for their trading desk.

35:51Yeah.

35:52If you don't give them one, there's one bank that will immediately require a 40% discount to reflect the higher cost of assurance if they buy your product at all. There are certain libraries that, if you have in your stuff, you give them an SBOM, certain libraries, if they're in your stuff, you are not going to sell your product to JPMorgan Chase. It's not going to happen because these guys are not willing to assume that level of third-party risk. When some of these large, highly security-aware organizations begin to exert their requirements on their suppliers as a condition of procurement or as a condition of acceptance, how people get paid, that's when it starts to get real. The thing about these requirements is that they tend to flow down. Now you have to figure out how to exert this level of situational awareness with regard to your Eastern European contract outsourced software developer or your Bangladeshi Mandalorian app developer, right? Now it's your problem. That's how supply chains work. They are multi-tier. When the people who are the downstream consumers who have the money start to exert these requirements on their first concentric layer out, well, those people are then forced to at least attempt to exert those requirements on that next layer down. Now, the fortunate thing for open source is it's open. The more open source that we actually use, especially open source that is responsibly maintained, the less opacity that we really have to deal with. I mean, the things that I find concerning is not— there are so many of these vendors throwing up all this fear, uncertainty, and doubt. Oh, open source is bad. Sonatype, Black Duck, all these guys are like, you should be scared. We can protect you, right? This ridiculous FUD. The real problem is when you have a subcontractor who's folded in some licensed proprietary third-party component that he doesn't want to tell you about because he views that as his competitive advantage. It's the unknown unknowns. With open source, even if you have a first-level dependency list, Ion Channel, we resolve those all the time to full transitive dependency analysis. There's a whole bunch of open source tools that can do that too. But when there's something that is not disclosed because it's a license component, that is, that is the one, that's the thing that's gonna kill you. So I think that monitoring open source components and having them in our SBOMs, that's all very important. But I don't think we should let a bunch of SCA vendors try to scare us into thinking that open source is the problem because we all have the tools that we need to become aware of it and to manage it. It's time to just grow up and not be cowed by these requirements, but to actually equip ourselves for the task. Just make the cultural decision that, yeah, we're going to have to slow down the delivery of new features. We're going to have to maintain this stuff, and that's the hardest thing of all.

39:14Yeah, it's always a challenge just trying to slow things down, but it sounds like now is the right time. And so, it's probably been the right time forever for this particular problem, but there's enough attention, there's enough things that are happening that people are dialing in. And like you said, we always follow the money. So, when the money drives, the need for these types of things, then that's where— that's when people start to take a lot more action.

39:41And I think that Steve, I mean, you speak so well about the requirement for automation. I would love to have Steve lay out the case for SCVS as, you know, a metric for automation and, you know, to assess the quality and the comprehensiveness of your data, because I think that's very important.

40:00Yeah, indeed. You know, as JC mentioned, there's a lot of manual work being done in legacy systems, especially for, you know, legacy languages and whatnot, C and C++ and assembly, right? Device drivers are part of the software stack that we care about, but a lot of the things that, especially around SCVS, which is a verification standard, the majority of requirements in there can be automatable. And this is really important because it allows modern development shops, it'll actually allow any shop to create some tests that can continuously monitor an organization's maturity in regards to what they're their supply chain capabilities are, and that's really, really important.

41:00So SCVS is a relatively new OWASP project, right? That's within the last year. And what's the adoption look like for that? Are people starting to lean into it and seeing that as something that, hey, I can bring this into my company and we can start to use this? Is this a standard that's trying to influence regulators? Like, what's the purpose of SCVS?

41:21That's interesting. There's a lot of related OWASP work that's out there, right? You've got SCVS, which is the Software Component Verification Standard. You've got Dependency Track, which is a way to analyze SBOMs, also an OWASP project. OWASP is not a standards body, so CycloneDX, for example, one of 3 SBOM formats, comes out of the OWASP world but is not an OWASP project. So there's a lot of related things, and a lot of the commonality between a lot of these efforts is really about shifting the focus to the whole whack-a-mole thing that is not a strategy into thinking in terms of supply chain, right? The OWASP Top 10 has A9. A9's been in there since 2013. It needs to be— it needs to go away, right? A9 is a subset of a much larger problem. I hope the OWASP Top 10 folks actually take that into consideration. I created a ticket for their GitHub repo, I don't know, a year and a half, 2 years ago, to hopefully bring A9 into a more category type of thing. So we're talking more about supply chain risk, But a lot of the OWASP projects that, you know, I'm involved with, whether it be DT, Dependency Track, or SCVS, they're really trying to shift the focus towards supply chain. SCVS being the most recent one where we give close to 100 requirements, I think. I think it's like 90, about 90 or so requirements. And like all verification standards, right? all of OS verification standards, it's pretty much 3 levels, right? Level 1, I actually, you know, know how to spell security. Level 2, I actually care about security. Level 3, oh my God, this stuff is really important. And we try to make it approachable and lightweight, right? We want organizations to be able to measure where they're currently at and work on improvement plans, right? When I would tell historically, when I would tell people in the security space, other technologists, whatever, when I would tell them that one of my former employers, it took us 2 years to secure Maven, right? 2 entire years from going from, you know, running Ant and having dependencies checked in a version control system to using Maven and having the same level of assurance, it took us 2 years to get there. And a lot of the activities that we, that we had to undergo are actually requirements in SCVS, right? So it's more about the holistic supply chain thing. It's not, you know, if you use a lot of these things out of the box, they will work, and that's the intent of them, but they're not going to work securely. And SCVS is a way to kind of shift the focus to talking about supply chain things and make it approachable so organizations can adopt it and improve over time.

44:46So we're just about out of time for our conversation for today. So JC, I'll start with you, and Steve, I'll come back to you as far as like What's a key takeaway or a call to action? What do you want our listeners to do as a result of our conversation today?

45:02I think the 2 things would be software bills of materials. Can you produce one for your system, whatever you're developing, if you are a developer? If you're a consumer of software and you're not developing, are you requiring a software bill of materials from the people whose software you're using? So that's number one. And then number two, I think it's worth looking at the OWASP Software Component Verification Standard as a benchmark to say, well, where are we here? Are we on— do we have half of the level 1s, a quarter of the level 1s? Let's pick something and work on it as a way to just start on the journey, right? So from the Shire to Mordor, where we're all implementing all this stuff at speed and scale, it's worth looking at because it's a very rigorous and yet not 480-page, which is what NIST does, way of looking at to say, okay, where are we at? And then how can we start? Because everyone starts in a place and everyone can get to a better place. And we can all be constructive about this without without like the emotional baggage of shame and guilt because all our systems are garbage.

46:16Yep. Steve, did JC take all the good key takeaways or?

46:21No, you know, I will second the whole SBOM adoption. You know, SBOMs are, they're already important and they're going to be increasingly important in the near future. It is a sign of organizational maturity, organizations that can produce one through automated means. have a certain level of, you know, development maturity. That kind of maturity is going to be an expectation going forward. So adopting SBOMs is in your strategic advantage, regardless of actually how you actually produce them. Being able to do so is going to have many technical and economic advantages to you.

47:03All right, well, JC and Steve, thank you so much for educating us About supply chain risk and SBOMs and SCVS. And I learned a lot of different new things today, and we'll definitely have to do another conversation in the future to continue this because I feel like there's a lot more things we can talk about. But thank you for your time today. And listeners, you have some homework there. Get going with this SBOM thing, check out SCVS. You know, you've got your assignment. So, thanks for being here today.

47:34Thanks, Chris. Thanks, Robert.

47:36Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, security is a journey. A journey, not a destination.