--- title: "David Habusha -- Third Party Software is not a Cathedral, It’s a Bazaar" url: https://appsecpodcast.com/david-habusha-third-party-software-is-not-a-cathedral-it-s-a-bazaar/ date: 2018-04-13 duration_seconds: 2222 guests: ["David Habusha"] topics: ["OWASP Top 10", "Software Supply Chain", "Vulnerabilities and Exploits"] audio: https://www.buzzsprout.com/1730684/episodes/8122687-david-habusha-third-party-software-is-not-a-cathedral-it-s-a-bazaar.mp3 transcript: true --- # David Habusha -- Third Party Software is not a Cathedral, It’s a Bazaar *April 13, 2018 · 37 min* with [David Habusha](https://appsecpodcast.com/guests/david-habusha/) on [OWASP Top 10](https://appsecpodcast.com/topics/owasp-top-10/), [Software Supply Chain](https://appsecpodcast.com/topics/supply-chain/), [Vulnerabilities and Exploits](https://appsecpodcast.com/topics/vulnerabilities/) [Audio](https://www.buzzsprout.com/1730684/episodes/8122687-david-habusha-third-party-software-is-not-a-cathedral-it-s-a-bazaar.mp3) ## Show notes An application can inherit serious vulnerabilities from code its developers never wrote. David Habusha, then a product leader at WhiteSource, explains the problem behind the 2017 OWASP Top 10 category on components with known vulnerabilities. He brings a product-management perspective to the discussion, connecting dependency choices to business outcomes and the realities of modern development. Chris and Robert ask why static and dynamic testing may miss this problem, what software composition analysis actually identifies, and how accurate component matching can be. The conversation uses Apache Struts and the Equifax breach to examine inventory, notification, and remediation. David closes by describing open source as a bazaar rather than a cathedral, where continuous awareness and maintenance matter more than expecting any component to remain permanently safe. The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/). About Security Journey Security Journey provides application security education for developers and everyone in the software development lifecycle. → [Learn more about Security Journey](https://www.securityjourney.com/) Connect with David Habusha: → [David Habusha on LinkedIn](https://www.linkedin.com/in/davidhabusha/) Mentioned in this episode: → [OWASP Top 10 2017 — Vulnerable Components](https://owasp.org/www-project-top-ten/2017/A9_2017-Using_Components_with_Known_Vulnerabilities) → [WhiteSource — now Mend.io](https://www.mend.io/) → [Apache Struts vulnerability CVE-2017-5638](https://nvd.nist.gov/vuln/detail/CVE-2017-5638) Chapters: 00:00 Understanding vulnerable third-party components 01:30 David’s security background 04:09 A product manager’s perspective on AppSec 08:48 What components with known vulnerabilities means 12:28 How much software comes from dependencies? 14:54 The consequences of vulnerable components 18:14 Why SAST and DAST are not enough 20:35 What software composition analysis does 24:02 Accuracy and component identification 27:29 What the Equifax breach teaches about SCA 32:58 Why dependency security is continuous work ## Transcript *5,093 words · assemblyai* **0:02 Chris Romeo:** On this episode of the Application Security Podcast, Robert and I are joined by David Habusha, and we talk about the OWASP Top 10 A9, using components with known vulnerabilities. David explains this for us and also goes into the software composition analysis market. David is the first product manager that we've ever had on the AppSec Podcast, so I asked him a couple of questions about how product management and security work together. We hope you enjoy. The Application Security Podcast. Here we go. Hey folks, welcome to this episode of the Application Security Podcast. On this week's episode, we explore yet another piece of the OWASP Top 10 2017, and this time we are looking at A9, the use of known vulnerable components. And we're joined by David here today. David, I'm going to go ahead and ask you, why don't you just go ahead and first Tell us your name and tell us kind of who you are and where you're coming from. **1:30 David Habusha:** Hi, happy to be here. And, you know, my story and my background with security actually started through an acquisition. I was working many years for enterprise software, a company called Veritas Software that was acquired by Symantec about 10 years ago. Convergence of IT and security began, and that was the actual reason for the acquisition. And there we started exploring mutual use cases of IT and business impact of vulnerability breaches, which was really interesting to see what happens if all the protection systems, firewalls, and what have you fail and hackers actually breach into your systems. What is the impact on the IT systems in terms of downtime and more importantly with costs. It was kind of premature at the time. After that, I worked for a company that provided database security systems for small and medium-sized databases. It was really interesting to see this industry evolving from providing database firewalls that actually provided IDS and IPS systems to preventing SQL injection attacks and actually controlling privileged access to sensitive data in databases. These solutions eventually evolved into also providing dynamic data masking solutions. In the last 5 years, before actually joining a company called Whitesource, I started my own company which is really interesting because it started from a personal privacy solution, basically letting end users know about all the applications and websites that can access any piece of personal information, whether on social networks like Facebook, Twitter, Google, what have you, as well as on mobile devices. But very quickly it evolved into a kind of a consumer security solution kind of identity management protection and protecting modern systems like mobile and web browsing. So all in all, I kind of moved between various aspects of security, but this domain is fascinating. And definitely the thing that I'm working on right now has a direct attachment to OWASP A9. **4:09 Chris Romeo:** And so you, right now, your job role or job title is a product manager, right? So you're coming to you're coming at this application security problem perhaps thinking about it a little differently than other types of folks that we've had on the show before. You know, we've had CISOs, we've had technical leads, we've had senior developers. This is the first time we've ever had a product manager that's actually been on the show here. So maybe just talk to us a little bit about the role of application security from your perspective as a as a product manager, knowing that you've had a bit of a long background in security? How do you think about security as a product manager? **4:52 David Habusha:** Well, it's really interesting because, you know, running a product management team for a company that provides application security solutions allows me to look from the outside on security organizations, but not only on security organizations, but on businesses as a whole. And what I see right now happening in the industry is kind of a shift in which the SecOps and security teams are becoming more and more developer-friendly because they acknowledge that to actually change the processes and policies, they need a much better collaboration between the various teams because security teams are usually the ones who find the issues, whether in production systems or earlier in the process when, you know, scanning code, etc. But developers are eventually the ones who need to actually fix the security issues. And what happens is that fixing the security issues, as much as important, is not really the most interesting thing to do when it comes to developers. So we find more and more companies either moving developers to the security teams and establishing what's called SecDevOps and converging security with DevOps, but also trying to provide them with kind of friendly tools and awareness to security so they can start much earlier in the process to be aware of the risks associated with using code components and actually maintaining clean code. **6:44 Robert Hurlbut:** Yeah. **6:46 Chris Romeo:** And so, I had a chance to work with a lot of product managers in a previous job. And I guess I experienced a number of product managers, and this is going back 10 years now, that were not very security-friendly. Only because they didn't understand. I think they just didn't understand really why they needed to care about application security. So, when you're looking across the industry right now and when you're talking to potentially maybe other product managers that are out there, are you seeing more people, more product managers becoming more security savvy in general these days, or is it still just focused on features and shipping product and revenue? **7:26 David Habusha:** That's also very interesting because I know I joined the company almost a year ago and I wrote a blog post about why did I join the company. And then I got so many requests from peer product managers on, you know, we have these application security issues, can you help us with that? And what do you say about this? What do you say about that? So You know, even consumer companies or project management companies. And, you know, a lot of my peers approach me just to ask me about security issues associated with their products. That was really amazing to see something that I, let's say 2, 3 years back, I wouldn't have seen because they would focus on functionalities and user experience and performance of their products. But now you can see that coming more and more often. earlier, even when designing the product and the product management processes. So awareness of security is definitely on the rise in that regard. **8:31 Chris Romeo:** Yeah, that's great to hear. And, you know, I could ask you questions about that for the whole time, but we're supposed to be talking about A9. So let's jump in and transition and talk about— so OWASP Top 10 A9, using components with known vulnerabilities. **8:48 David Habusha:** What— **8:48 Chris Romeo:** so let's just start with the basics, like what are components with known vulnerabilities? What does that even mean? **8:54 David Habusha:** What does that mean? I would call it a reality because at any point of time, probably any large organization runs a component with known vulnerabilities. There was a study, I believe 2 years ago, you know, ran across the Global 500 companies that found that more than 50% of them use vulnerable components at any point of time. open source components like Apache Struts, they are being widespread, they're used in many organizations and there's so many associated vulnerabilities that organizations don't even make it on time to patch it. And when they patch it, a new vulnerability comes up. So I would call it a reality. And what it basically means is that your code, any tool that you run to actually make sure that your code is safe, you know, creates kind of an illusion of how secure your application is, but you're using a lot of third-party code, whether commercial or open source, out of which the open source community is— has open channels of communications, and all these components are being hacked by usually white hats. And these exploits are being actually published to the public. And then you find yourself using third-party components that everybody knows about the vulnerabilities associated with them, whether you want it or not. But that's a reality because organizations today focus on the customized, personalized business logic And anything that they can use that was already written and tested by the community is being used to save time, to get faster to the market, to save costs, and to be competitive. Otherwise, you can't succeed because you will deliver a much more expensive and heavier software. So to me, that's a reality. **11:11 Chris Romeo:** Yeah, and Robert's actually coming to us from a Solid development background. Robert, from your perspective, I mean, do you just grab— as a developer, you just go and grab libraries and things to provide the functionality that you need? You don't even think twice about that, do you? **11:30 Robert Hurlbut:** Well, it depends. **11:33 Chris Romeo:** You might because you're a security developer. I might. **11:36 Robert Hurlbut:** Exactly. **11:36 Chris Romeo:** But the average developer out there? **11:38 Robert Hurlbut:** The average developer may not. I mean, it depends on where they are. If they're in a company that manages repositories of components, and so they're pretty much vetted for the most part, then only those are the ones that they could pull down and use in their applications. And so hopefully they've already gone through some kind of, like I said, a vetting process, checking for the latest version, making sure they're there and so forth. But if not, and there are many companies that do this where it's pretty much open season, a developer says, hey, I need a new library, I need a new component, Hey, this looks good. I just did a search. Great. Download it, install it, and off we go. And so that's a reality, as was mentioned, that can happen for developers as well. **12:28 Chris Romeo:** Yeah, and I've heard that— I've seen this number thrown around, and I'll just throw it out, and both of you can either confirm or deny it for me, but the number that I've heard is that the on-average an application consists of about 90%, if you were to do this by lines of code, about 90% of an average application is coming from third-party and open source, and only about 10% of an application is actually custom code that's been written by the actual developer. Does that seem reasonable? Is that off base somewhere? **13:03 David Habusha:** Yes, that's in line with what we're seeing as well. And if you look back 5 years ago, this number would be around 60%. 3 years ago, it was around 75%. Last 2 years, it was 80%. And now in the industry, speaking about 90%, as you mentioned. So you just see it on the rise. It's amazing. **13:25 Chris Romeo:** And the vulnerabilities that exist in these third-party components, this open source, this is a range of criticality of vulnerability as well, right? So Some percentage of these are going to be low, but some of these are going to be a criticality of 10.0, meaning they allow remote code execution and all kinds of other bad stuff, right? So there's a range of problems that we're dealing with. Is that correct? **13:49 David Habusha:** Oh, yeah. Each of these vulnerabilities is associated with something called a CVSS score. And 10, as you mentioned, is kind of the riskiest. And there's a full variety of range of CVSS scores. If you look at the major breaches like the Apache Struts breach that led to the Equifax saga, and that was a CVSS 10 breach definitely, but you see anything from 2 to 10 and it's widespread. So, I mean, the most common components, usually you would find a lot of vulnerabilities with a variety of ranges, but actually the kind of mid to low usage components, you would usually find the highest risk of security vulnerabilities because otherwise it's not worthwhile finding these vulnerabilities, I assume. But this is kind of what it is in the industry. Yeah. **14:54 Chris Romeo:** So, what's the worst-case scenario then? If— so, let me give you a what-if situation. So, I have an application. I've chosen some libraries to provide functionality at random. It turns out they have some major security vulnerabilities that exist that are now being embedded within my application. What's the worst thing that can go wrong? What should be— what should keep me up at night as the worst-case scenario? **15:21 David Habusha:** It could be anything from breaching siloed parts of your application with insignificant data to actually hitting your master database with all the records and taking down your systems. Because, you know, third-party components are so widespread that people put their frontend systems based on third-party components like application servers and even SSLs and VPNs. **15:51 Robert Hurlbut:** Yeah. **15:53 David Habusha:** So even the most critical parts of your app are usually based on open-source components that are usually the source of many of the known attacks and known vulnerabilities. Therefore, it's the highest risk possible. **16:10 Chris Romeo:** So then we've— so the things that we're really concerned about, so there could be remote code execution, buffer overflows that could exist. Could be almost any— many of the things in the OWASP Top 10 could be potential challenges here, right? So cross-site scripting, SQL injection, all of the— and all of these things could then result in what you talked about from, you know, compromising a certain piece of an application all the way to a major data breach, and which could ultimately result in an attacker shutting your systems down completely or causing some major damage behind the scenes. **16:47 David Habusha:** Yeah, that's correct. And, you know, handling open source third-party components is very different from handling proprietary code because with all the good— all the benefits that you can actually get by using open source components that I mentioned earlier, everybody sees these components, everybody has the access to the source code of these components. And therefore everybody can actually try to find vulnerabilities in these components. And these vulnerabilities are also public. So when they go— when these vulnerabilities are published, there is a window of time from the publishing date of the vulnerability until you actually fix it in your systems that you are very vulnerable because A, everybody knows about the vulnerability. B, everybody knows that it takes time to actually fix that vulnerability. And they have, they know the source code, they know how to actually run the same attack because 2 days after you can find samples of, you know, how to write scripts that demonstrate the attack on YouTube or on any other website. And you have to act fast actually to fix these vulnerabilities. **18:11 Robert Hurlbut:** Yeah. **18:12 David Habusha:** It's a really rough story. **18:14 Chris Romeo:** So, I'm trying to fully understand the problem space here. And so, I'm thinking about this and I'm thinking, I'm back to thinking about my average company here. They have a secure— a process for security. They have tools like static analysis and dynamic analysis. Why aren't they finding these problems? Why aren't they providing and why aren't they scanning these third-party components that are part of their applications in the same way that they are testing their own code? Where's the breakdown here? **18:48 David Habusha:** I think that's the key. And as we see a lot of organizations starting to realize that third-party open-source components need a different approach. It's not that there's a known set of attacks that these components are vulnerable to. And it's not that if you scan your code once and you haven't changed your app since then, you're safe. It's not, it's not the situation like your proprietary code. You have to run continuous scans of code within, starting from your development time to the build time to the deployment time. And that's not even enough because you first need to actually understand what you're using. If you're using Apache Struts or Apache Spring, there's probably 30 or 40 different underlying open source components that are included with these components that may be also vulnerable, but you don't even know about that. So you have to first be able to understand what you're actually using and then be able to continuously scan them and then be always on top of all the security advisories that publish known vulnerabilities. And just looking at the NVD and MITRE is not enough because there's so many other different security advisories out there that you need to constantly scan to know that you're not vulnerable when using these components. So that's, that's a different type of challenge. It's not your proprietary code. It's something that everybody has access to, including the people that you don't want to have access to. **20:35 Chris Romeo:** Yeah. So, what does the technology space kind of look like here now as far as tools to find these types of problems? I've heard this, I've heard a new term thrown around, software composition analysis or SCA. And I've seen people talking about this as a new category. that's going to— I've seen it thrown around in the same discussions as like IAST, Interactive Application Security Testing. So, what is, I guess, what are the tools, what are the types of tools that are available to help us solve this problem for our average company? **21:17 David Habusha:** Yeah, software composition analysis is a set of tools that actually started a few years back. With the realization that people use a lot of third-party open-source components, and the challenges that I just described can actually be automated. So the first and most important thing to understand is first, what do I have in my system? What are my developers using? And you cannot just maintain an Excel sheet or ask your developers Kindly tell me, you know, what are the open source components you're using? Because even if they tell you, you would find a different picture because, as I mentioned, every open source component has dependent other components which then have transitive dependencies. So you need to constantly map these dependencies and stay on top of what you have in your systems. Then you need to ensure that you put clean code earlier in the process. You can start as early as actually when selecting the component. So when you ask Robert, you know, about the awareness of selecting open source components earlier in the process in the development time, and that's where people should start actually looking at using safe components, using the latest version of these components, using components that have a low number of vulnerabilities. And using, if possible, kind of standardized component across the organization so you don't have to kind of use 10 different components for SSL or for any other purpose. Then you need to make sure that this taps into your CI/CD process. So with every build, with every deployment, you need to rerun to make sure that nothing has changed in terms of the components that are being used and in terms of the vulnerabilities that were actually found in the recent days. But also you need to constantly monitor the security advisories. And even if you haven't changed anything, haven't built anything new, there might be a new vulnerability coming that affects your third-party components that you need to actually handle and fix. So these tools essentially automate all of these processes and provide you with a real-time understanding of what you use, how vulnerable you are, you know, with the levels of vulnerability, and what could be done to actually mitigate the risks. **24:02 Chris Romeo:** So how good are these tools then at this point? That's always where That's always where, as a practitioner, always where I land because I know there's a lot of tools that do a lot of things and some of them are really very good at what they do and others are not so good at what they do. So, how accurate— what's the error rate on these types of tools today as far as what's the percentage of things that are going to slip past a software composition analysis tool? **24:37 David Habusha:** Yeah, so that's a question on, you know, how the tool operates and the technologies behind the tool. There's various approaches starting from actually running code scanning, which, you know, generate a lot of results. Some of them may be actually false positives because they're looking for, code snippets, you know, matching them between open source components and your code, trying to find similarities. **25:09 Robert Hurlbut:** Okay. **25:09 David Habusha:** And there are some misses. Other tools are using hashing technologies to find exact matches of open source packages in known repositories and known, like GitHub or Maven Central, what have you. Against your inventory. They may not present any false positives, but they may have some low number of inaccuracies in terms of the detection of open source components. They need to constantly be on top of every new component on the web out there. So that's very hard to do as well. And definitely, if you look at the combination of technologies and finding the right approach that matches your organization, depending on the languages you use, the package managers, and so on, then you can find the right tool that would give you the confidence in actually giving you a real-time analysis of the risks associated with the components you're using. **26:22 Chris Romeo:** So on average, what would you— what do you think is the error rate that these tools— how much are they going to miss? 5%? 50%? Are they pretty accurate? Are they going to, you know, is it only going to be a 1% kind of problem where they're going to get 99% of the stuff right as they go? What's, what's been your experience? **26:40 David Habusha:** My experience is that these tools got much better in the last 2 years, and we're talking about probably less than 5% of inaccuracies. **26:51 Chris Romeo:** Okay. **26:52 David Habusha:** Yeah, and then, you know, there's always these new programming languages and package managers. So the idea is to actually— I think that, you know, the first point of actually running the right discovery engine is the key to providing you with the security vulnerabilities associated with your usage of open source components. Once you have the most accurate discovery, then mapping it, mapping the inventory to the known vulnerabilities is an easier job. So everybody's focusing on, you know, providing you the right discovery. Okay. **27:29 Chris Romeo:** So we've mentioned Equifax a couple times here. And so I'd love to explore for a minute or two how software composition analysis could have prevented the Equifax breach. And just for— just in case anybody's been on a media vacation for the past year where you haven't used your phone or your laptop or anything, Equifax was breached a number of months ago, and the— basically their entire data set was stolen, including all of the credit records and things for everybody in the United States. And so it came out that The Equifax breach pointed back to an Apache Struts vulnerability that, as David mentioned, had a CVSS score, Common Vulnerability Scoring System score, of 10.0, so it was critical. So, David, could software composition analysis have prevented this problem for Equifax? And how could, how could this have had a different outcome as a result of perhaps proper Sure. **28:38 David Habusha:** So if you look at what's called in the software composition analysis world shift left, there were multiple points at which the attack could have been prevented. So if you look at the kind of the sequence of events of the attack, somewhere in the beginning of March, the exploit code started appearing in the NVD. Okay. Few days later, it was actually published. It had the CVE of 2017-5638, I believe. I'm just using this example a lot, so I know the number by heart. And the fix actually was published on the same day. So whenever there's a CVE that goes from reserved state, which means that somebody found something and now the open source owner has a window of opportunities to fix the code, but nobody knows what the, breaches, usually there's 90 days to fix the vulnerability before it goes public. Then it happened. It was in reserved state, so nobody could have actually known how to use it. And on March 10th, I believe, it was out there. If at that point Equifax had a software composition analysis tool, they would immediately receive an alert that they're using a component that a new vulnerability has been published and is actually affecting the way they use the component. And then they have a window of opportunity to actually patch the system. By the way, the fix was fixing a few lines of code in Apache Struts. And they published a new version of Apache Struts on the same day the vulnerability was published. But the problem in this sense, and we also see that, and we actually, you know, there's an industry survey that says that very large organizations need 12 to 18 months to actually patch a system and update a software component version of a component they're using. So it means that hackers have a window of opportunities of 3 to 4 months to actually attack your systems. And that's because the things I mentioned earlier, if you're using Apache Struts, you're using 40 different components, and then you have kind of a ripple effect. You have to now update a lot of different components. Now you have to build your system, make sure that nothing breaks, then you have to test it on a sandbox, and you have to make sure that it's backward compatible. Then you need to put it on a staging environment, and then you deploy to production, and that takes a lot of time in organization. So with software composition analysis, you could have found it earlier. They would point you to the remediation advice much earlier. There's some new technologies evolving that can actually give you, you know, traces and lines of code in which, in the way you actually call the vulnerable components, that maybe besides of actually updating the component itself, you could have bypassed the problem faster than actually upgrading the component. But what happened in Equifax is more interesting because Even in this window, it took them from March till August to actually understand that they had this problem. So it took them 2 months just to realize that they have the problem and then 3 more months to actually fix the problem, which is a huge window of 5 months to actually being vulnerable to this attack. And actually, you know, the whole thing was published or discovered end of July and went public somewhere in August. So definitely with software composition analysis tools, it could have been prevented much earlier in the process. **32:57 Robert Hurlbut:** Yeah. **32:58 Chris Romeo:** Yeah. Okay. So I guess to kind of wrap up the conversation here, and I've been asking this same question of everybody that we've been talking to about a specific OWASP Top 10 item. What do we have to do to make this problem of software with known vulnerable components disappear from the OWASP Top 10 in 5 years from now, whenever they do the next version? What has to happen for us as an industry so that when that OWASP Top 10 is updated, this issue's no longer gonna appear on that list? **33:33 David Habusha:** It's kind of a philosophical question because Most of A9 is about using third-party open-source components. And open-source components, we'd like to say that it's not a cathedral, it's a bazaar. And by that we mean that we take something that is kind of a public thing that is maintained by a community and everything is kind of open to everybody, and you want to model it as a proprietary entity with clear processes, with direct control and stakeholders, which is very difficult in the case of something that is widely used in public. It requires a cultural— there's a lot that can be done actually. I don't know if disappear is the right term, but definitely we can do a lot to make it better and a lot as an industry in terms of collaborating between various companies and the open source owners to alleviate and definitely maybe remove it from the list of the top 10. I'm not saying go away or disappear, but you need a cultural change and the cooperation between enterprises, white hat hackers, developers, the community. You need to educate people about the awareness and the way open source components work. What are the risks associated with using these components? And try to make that as early in the process as possible, starting from selecting the components, from actually putting them into your build environments, and then from shipping them and constantly monitoring them. And I think that the key is to share knowledge between the various stakeholders in the industry. So CVE and MITRE and NVD is only one point of, you know, source of truth. But there needs to be a way for any organization to immediately understand how vulnerable they are and what they can do. And if we share these best practices, processes, education, and tools, I think that the industry can be in a much better place. **35:54 Chris Romeo:** Awesome. Well, David, thank you for taking the time to be with us today and sharing your insights as a product manager and your insights into A9, as well as the software composition analysis. This has been very beneficial for me. I've learned a lot. So, thank you very much. And we'd like to— I'd like to invite you to come back in the future, and I'd love to spend a whole episode talking about your experiences as a product manager because I think that's a whole other interesting topic that I'd love to talk about. So, we'd love to have you back in season 4 to talk about that. Thank you for being with us today. **36:28 David Habusha:** Thank you guys, and I'm happy to come again. Thanks. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org. --- Source: https://appsecpodcast.com/david-habusha-third-party-software-is-not-a-cathedral-it-s-a-bazaar/