Devin McMasters -- Bug Bounty with a Side of Empathy

0:00Chris RomeoHey folks, season 3, episode 19 of the AppSec Podcast. On this episode, I am joined by Devin McMasters to talk about all things bug bounty. So we hope you enjoy this deep dive into what is a bug bounty, why do you care, how do you set up a program? Devin provides lots of insight and advice from real-world experience about how to make bug bounties successful. Hope you enjoy. The Application Security Podcast. Here we go. Hey folks, welcome to this episode of the AppSec Podcast, and today we're going to talk about bug bounty. This has been a topic that a number of our listeners have had questions about, and so we're going to try to provide an introduction to this concept. And I'm joined today by Devin McMasters, and Devin's going to help us get dive deeper into this idea of bug bounty. So, Devin, we always start by asking a simple question, and that is, what is your superhero security origin story? Just like in every comic book, there's an origin story. It's episode 1. What is, uh, what's the story behind your kind of episode 1 and your, your track into security?

1:41Devin McMastersThat's a great question, Chris. And hey, thanks, uh, just at the beginning here, thanks for having me on today. Um, my story is, is a little interesting. So for me, For many years, well over a decade, I was an applications developer, mostly client applications, manufacturing tests, etc. And one day while looking for a new opportunity, I happened across a product security posting internal. So I had all of these ideas about how our company wasn't managing physical security really well and all of the things I had witnessed and observed.

2:17Right.

2:17Devin McMastersover my many years of service. So I went into the interview thinking, okay, I'm going to wow these people about all these improvements I can do to improve our physical security of devices and things going on in our manufacturing sites. And about 20 seconds into it, the hiring manager stopped me and said, you realize this is for application security, right? We both paused at that moment and he said, The good thing is I like all of the ideas you're coming to the table with about improvements and redundancy checks and security checks because if we just take it from hardware and flip it to software, I think you're our guy. And that's how I made the jump over to the application security space.

3:02Chris RomeoVery cool. Very cool. So what was your primary language that you wrote code in?

3:07Devin McMastersPrimarily, I was C and C++.

3:11Chris RomeoOkay.

3:11Devin McMastersLots of Windows application, Windows device drivers, mostly at the hardware level.

3:16Chris RomeoOkay. Now, after you got established in application security and had a few years of experience, did you have any moments where you woke up in the middle of the night and went, oh crap, I got to go back and fix this code that I wrote years ago? Any of those moments?

3:32Devin McMastersOh, absolutely. It's, it's interesting when you look back over the last 15 or 20 years worth of software development and you realize all the priorities around features feature sets and customer requirements, and not until recent time was security or secure development practices ever discussed, or hardly ever. To look back on all the software we developed and the pieces of code you'd copied here and copied and pasted just generation over generation, it's quite frightening sometimes.

4:06Chris RomeoYeah, I was talking to somebody today, and we were talking about cross-site scripting and thinking, is there an answer for this? Meaning, are we ever going to solve this problem? And my answer was, well, when all the legacy code goes away that didn't have solid frameworks implemented in it, then I think we have a chance of fixing something completely and eradicating cross-site scripting. But I don't think that's going to be for 10, 20, I dare say 30 years before we can get rid of all that stuff.

4:37Devin McMastersOh, that's, you know, cross-site scripting, buffer overflows, all of them. There's so many different incarnations. that, uh, the, the problem is, is, uh, new containers, new modules evolve and grow. It just opens up new attack surfaces.

4:54Chris RomeoYeah, definitely. So the topic we want to, we want to dive into today is this idea of bug bounty. And so I know some of our listeners aren't even going to really have any perspective on what this is, so I thought we'd start by just answering the question, what is this idea of bug Sure, there's, there's a couple of ways people think of this.

5:16Devin McMastersFrom my perspective and from the business side, bug bounties are really a way to go out and do crowdsourced security. So in the simplest form, it's going out and saying, hey, person of interest, either a hacker or security researcher, we would like you to investigate our product or application. find issues with it, and in return, we're willing to pay you for the issues you find. And our issue payout will typically be based on severity level. So there'll be a pay schedule where we pay out a very high amount for critical severity levels down to very low amounts for medium or low severities. Okay.

6:01Chris RomeoAnd so why would you want to do this? Like, what's the driver for you as somebody— you mentioned the business, so I'm going to pick on the kind of the business side. What's What's the driver for you as somebody on the business side to even do this at all?

6:12Devin McMastersWhat we find, Chris, is that we— whether you have a PCIRT function, product security incident response team, whatever your security inbox is for people to report vulnerabilities, there's always this flow of researchers or beginner security people trying to do what, you know, similar to a ransom request. There's multiple times a week we'd get requests that come in and say, hey, I found something wrong with your product. If you pay me $2,000, I'll tell you what it is. So you're left wondering, like, well, do we pay the person? Is this a scam? Did they really find something? And the monetary side of it, you step back and look, okay, this person is driven to find things wrong in applications because of money. If you look at our internal employee base, our developers are really paid to develop features, not necessarily to find security issues. So, if we flip the script around what are strengths to go out into the world of hackers and security researchers, recognizing that they are the experts at finding things and they are highly motivated based on a monetary reward to find issues that our development teams are probably not going to find or not capable of finding based on skill sets and background and experience, then it makes sense for us to go out and almost pay as we play, if you will, for people to come in and interrogate, attack, poke holes in our applications and processes, and in return we reward them for their efforts.

7:49Chris RomeoSo it sounds like your thinking is that people are gonna do this anyway, so we might as well coordinate and consolidate their efforts into a program where we have some amount of control over it versus just allowing them to operate in the wild, wild west where they may have found a problem, they may not, they may be bluffing us, they may have found a problem. And so if we can bring it under our own kind of rules, then maybe we can manage it better. Is that a fair summary of kind of the idea behind the program side?

8:23Devin McMastersThat's a great summary, Chris. And when you talk about efficiency for your development and security teams to be able to respond to issues that are found, you can spend countless hours countless days and weeks scouring the internet for all the various forums where researchers or hackers may post their findings. Maybe they tell you directly, maybe they post it to their own personal blog. But by taking the proactive approach, just as you described, absolutely, there's a huge efficiency gain to be had for the development teams as well as the company.

8:57Chris RomeoSo you're incentivizing the researchers then to come to you and receive money for what in the past they may have posted on a forum or posted on their blog, and you would have to go chase around and find it. So you— it's kind of like— it sounds like it's a better partnership between the researcher and the company themselves that's actually putting out the bug bounty.

9:19Devin McMastersIt is. It's a great way to make a partnership, and depending upon what's found, one of the, the great benefits is these Our experience anyway is that the security researchers and hackers typically find things that our development team had no idea or no way of detecting or background experience in identifying. So it works as great feedback. I mean, similar to if you'd done a penetration test or anything like that, but it's from such a diverse mix of people outside of your own organization bringing all of this wonderful feedback on how to improve your overall quality.

9:57Chris RomeoYeah, so I had a question about how— so how does this fit with what I'll call traditional penetration testing, where you're paying some company to go spend some amount of time to try to break into your application? Let's just say web application for purpose of this example. So how does the bug bounty fit with that traditional pen testing world?

10:18Devin McMastersSure, there's a lot of similarities there, Chris, and One of the differences why we see value in the bug bounty program is the diverse set of researchers that end up getting applied. So depending upon, you know, depending upon how you pay and how you service a penetration test, a service provider or a company may come in and you're not quite certain what background, depth of experience, what skills is that service provider bringing to the table for the penetration test. When we go through a bug bounty program, You know, our bug bounty program service provider is— we can be very specific. We want a very diverse set of skills. We'd like as many different attack vectors as possible. Now, that provider will go off and recruit a wide, you know, a very diverse set of hackers based on our criteria. Now, we don't even know what to ask sometimes, but the bug bounty program service provider is experts at knowing here's what they found across all these other companies, and they can take that experience and knowledge and apply it to the testing of our application or service.

11:30Chris RomeoOkay, so that, that definitely makes sense about the connections between the penetration testing. So with the bug bounty side, you're paying for results and only when there are results. And in the penetration testing side, you're paying whether they find something or not. Is that correct?

11:51Devin McMastersYeah, that's another great value proposition point to bring up. The payout schedules, you can be very specific in a bug bounty program. program, you're— when you define your scope of engagement, you can say, I— we are only interested in this set of vulnerabilities. We don't care about all of these things that come out of automated tools. We want these low-level, nitty-gritty, keep-us-up-at-night issues that, uh, that aren't going to be found by the typical tester.

12:20Chris RomeoOkay, what, what are some of the type of findings that we're gonna— we're gonna get from one of the— from a bug bounty like this? Because it's not going to be the same types of things we're going to get from our scanning tools, we hope. So what, what would be an example of something that we should expect from a really high-end researcher that we're going to pay that, that highest amount for that critical or high value?

12:44Devin McMastersSure. There's, there's 2 vectors of that. The first would be remote code executions are typically high on everybody's list. And I would agree with that. Those are our most concerning issues. Now, so the remote code executions typically involve a lot of extra programming, not just running a, you know, an automated test against the site and feeding it in a bunch of scripts. There's a lot of manual software development or coding involved to root out where the vulnerabilities might be. The other piece of it is, and what's really important, is the hackers and the researchers separate from the pen testers. They will go off and for them to come up with their attack vectors, they will scour the internet and look for their reconnaissance information. A lot of times what they'll uncover for us is a lot of inadvertent information about our application or processes that are posted maybe in— I don't want to say like criminal forums or places like Pastebin or somewhere, public forums where there's this treasure trove of information about us that we didn't even know. So that's the other benefit that we're paying for, is they're uncovering and revealing all of that data that's out there that could be used to compromise our applications that the traditional pen tester wouldn't find.

14:15Chris RomeoOkay, because they're connected into those forums or dare I say, dark web type of places that are not indexed by Google and things, and they have connections and quality, and they're in those conversations or have access to those sites. And so they can bring some of that information to you that you wouldn't even be able to find on your own.

14:35Devin McMastersExactly.

14:37Chris RomeoOkay. So when we go back to this idea of payouts, what, what are— I mean, what are standard kind of payouts? How much are people making as researchers that are playing in these bug bounty Sure.

14:53Devin McMastersSo you'll— like any other profession, you have your high performers and mid-performers, kind of a bell curve. Your typical successful people will make 6 figures regularly. Now, what they'll find is— what you'll find is you'll get people who are experts at finding a particular kind of exploit. Or such as remote code execution. So they become experts at finding their niche area. So they'll go around and look and shop for bounty programs that are out there, knowing that there's a strong likelihood they're one of the few people who's going to be capable of finding that issue, and that typically has a high payout.

15:37Chris RomeoOkay.

15:37Devin McMastersBut if you're looking to give people an idea, typical entry-level bug bounty programs start out with maybe a budget of around $20,000, and you might pay upwards for maybe, let's say, $5,000 to $7,000 for a critical or high vulnerability and maybe $200 for what we would consider medium or low level, like cross-site scripting. As annoying as they are, and depending upon where it exists and how it's implemented, It could range anywhere from that low to critical area, but that's the range of somewhere between $200 to $5,000 a bug found for an entry-level program.

16:21Chris RomeoWhen you said $20,000 budget, you meant for like a single— that's a yearly budget?

16:27Devin McMastersNo, so there's different ways to do it. Companies that are starting out, if you think you're interested in doing a bug bounty program, there's a variety of ways to engage. You don't have to have an an open-ended program that's on an annual basis. You can do short-term engagements for, let's say, a 2-week period. So you put in $20,000 into the pot, you go out and recruit your hackers, and you say, for 2 weeks, here's what we want you to attack. And it's done privately, so it's not open to the public. And at the end of that 2 weeks, or when you run out of the $20,000 if they've been really effective on the front end, that's your program. Now you can do the assessment and take that feedback and determine if you want to put together an annual budget to support an open-ended program that just runs public all year long.

17:19Chris RomeoOkay, so, and so you mentioned public there, so there's a difference between— is there public and private bug bounties then? Are there people doing bug bounties that we don't even know about at this point?

17:30Devin McMastersAbsolutely. So when it's The majority of the companies will tend to start out with a private— a privately ran bounty program because, you know, you're not sure what's going to be found. Do your— does your development team have the resources to respond to this influx of issues?

17:50Chris RomeoAre—

17:51Devin McMasterswhat's the disclosures going to be like? Can you deal with having to do so many disclosures in a short period of time or open-ended period of time? So the private programs are actually extremely popular. And the hackers or security researchers can only participate if they're invited.

18:07Chris RomeoOkay, so I guess on a maturity level for the company, how mature does a company have to be to start a bug bounty, at least in your experience, to truly get some benefits out of it and not just waste their money?

18:23Devin McMastersSure, there's a— that's a great question. The main thing is that some kind of incident response, security response program, the core pieces of it need to be in place. You need to have an established incoming process to accept vulnerability reports. You need to have a good working relationship with your legal and public relations piece because things that are found may result in having to do a disclosure. It may involve personal identifiable information, PII. So there's a lot of— there's a wide range of impacts depending upon what's discovered on what you could be required in responding. So if your program, if you don't have a good incident response program established, it doesn't have to be world-class, but you need to have the basics. You have to prep the legal team, the PR team, and the development teams that, hey, for this period of time, these are going to come in. We need to be able to respond, remediate, and potentially disclose about them. And what happens is if you don't deliver on all of those aspects, the hackers lose confidence in you. The bug bounty service provider kind of loses confidence in you, and you get this reputation out in the the researcher community that, well, they don't really have their act together. And the next time that you want to do a program, a bug bounty program, they may not sign up to participate.

19:59Chris RomeoI guess, how do they have— I'm fascinated by the fact that there's a reputational side to bug bounty on the company side. So is there a negative perception from the researchers' perspective because you're not— we're not paying? them, them in this? Or how are they losing faith in, in us as a company as a result of, uh, us not dealing with the, the, the problems that they found quick enough?

20:29Devin McMastersSure, there's a couple of things. One is, uh, response time. So if you're a researcher, you know, time is money like any other, any other profession. But let's say you've submitted a cross-site scripting vulnerability. Well, if you're the company, let's say I'm the one that's hosting the bug bounty, if I don't respond back to them and say, yes, we verified this, the door is still open for another researcher to submit that.

20:59Ah.

21:00Devin McMastersWhat happens is you run the risk of duplication of effort. Then if you get so many researchers submitting the same thing, then you have to go back and say, oh, sorry, someone else already submitted this one. And as part of the program, they would— we would already list what the vulnerabilities are found. There's the identification piece of who did what is all anonymous and masked off, but the running list of what's already been found is available. So that's where the security researchers and hackers get upset because, hey, they just spent all this time submitting something that's a duplicate.

21:38Chris RomeoAnd they're gonna get nothing then.

21:39Devin McMastersThey get nothing for duplicates.

21:42Chris RomeoSo if they, so if they end up, so if you're not fast enough in your response to validate the initial submitter of a problem, others will submit the same thing. And then when they, when those get duped, they're gonna get upset because, and they're gonna, they're gonna potentially go away from you and say, I'm not dealing with you guys anymore because you're so slow.

22:01Devin McMastersExactly.

22:02Chris RomeoOkay, that makes sense. All right. So what about So, so we talked about kind of the IR perspective, how you have to have— you don't have to be perfect in your incident response approach, but you have to have, you have to have some of these basic things kind of figured out. Um, what, what is it? What is the, the interaction with the development teams look like as far as what is— what do I have to do to be prepared to prepare the development teams to be a functioning participant in this bug bounty program?

22:33Devin McMastersThat's another great question. Really, it comes down to timing. Where are you in the lifecycle of your development process? If you're within the quarter of when your next big major release is coming out, that's probably not a good time to start your program or to host a program. You need to be able to go in and work with the development team and say, how can we keep the day-to-day operations going? How do we plan, whether you're doing Scrum or whatever process you're following, Do we have allotments for unplanned interrupts? Do we want to go ahead and forecast for bug bounty program submissions that are going to come in during this time? Now, knowing that there's going to be this additional workload come in, what does that do to the rest of the development cycle and the commits on features and such? So really, it's the developers themselves are actually, most of the time, they're open and they're somewhat excited to see what people are going to find wrong. You have to caution that with the fact that— tell people, we're going to find issues. Everybody makes mistakes. Don't worry about it. When we bring these in, don't take offense and take it personally that these people found, you know, 500 things wrong with our application. That's what we're paying them to do. Just go into it with an open mind and be very direct with people and say the point is to improve our overall quality. accept the feedback coming in, and generally the development teams fall right in line.

24:06Chris RomeoOkay, so any— is kind of switching gears a little bit here. Have you— is there anything that you've seen in bug bounty that just made you shake your head and say, wow, this is not how this should be done? Is there anything we can share with, with the audience that, that they could take away, like, as a lesson learned as what not to do? Because it's not— you've given us a lot of good stuff as far as what we should do. What should we not do?

24:31Devin McMastersYeah, absolutely. So when you look at doing a bug bounty program, it's— you're making a business engagement with the security researcher and the hacker community. So relationships are paramount for being able to do business in our, in our world. So if you're slow to respond or if you get into arguments with Sometimes there'll be a disagreement. It usually comes down to severity because the severity level is what determines the payout.

25:03Chris RomeoAnd you're the one who has the final say in that, right?

25:06Devin McMastersYeah, absolutely. So when you define your program, you say our company is the one setting, you know, we have the final call. Now you have to take that, you have to apply a little bit of empathy and say this person is trying to, probably trying to earn a living. doing this, and do you really want to get into an argument over severity level if it's really close and aggravate this person? Or as a— you know, you're paying for this service, you can just say, okay, this sounds like it should be the higher severity level, and we're going to pay it out and keep a good working relationship. So it takes a little bit of experience to know when to make that judgment call. about being empathetic and what side to err on. But by and large, the main mistake people I've seen make is to get into petty arguments over severity levels and the payouts. Okay.

26:04Chris RomeoThat's definitely good advice for everyone to consider as you're thinking about getting into or starting a bug bounty program. That's— the relationship with the researchers sounds like it's going to be crucial. And it's a way— it sounds like a way you can have a very positive relationship or you can choose to have a negative relationship based on how much you want to argue and kind of make their life difficult along the way.

26:29Devin McMastersAbsolutely. And the relationship is critical because what will happen, Chris, is with social media, any kind of bad interaction will immediately get turned around and reposted somewhere else to some other social media account. So if someone has a bad experience, they're going to go tell They'll post on their own blog or their Twitter feed about it, and then it just kind of balloons from that point. And now you've got this reputation or this one comment about you or your program or your company floating around out there when it could have easily been avoided.

27:04Chris RomeoYeah, yeah, that definitely makes sense. So I guess as kind of a way to conclude our conversation here, well, if you could only, if you could only offer 3 simple tips for bug bounty, what would those, what would those 3 simple tips— and by simple means they got to be less than about 10 or 12 words.

27:24Devin McMastersSure. Oh, let's see, some simple tips. Uh, first would be, uh, have your incident response function organized. Okay, like we said before, it doesn't have to be world-class, but organized. 2, definitely have— let's see, I think the second most important tip would be the clarity of your scope. If you go into it with a wide-open scope or not well-defined what you want them to attack or look for, you're in for a nightmare.

28:06Okay.

28:06Devin McMastersYou'll start receiving all kinds of things that you weren't even expecting. Um, and the third, I think I've hit it on already, but be empathetic. Go have a strong sense of empathy towards the other people you're working with, the security researchers and hackers, and try to understand your business engagement with them from their point of view.

28:27Chris RomeoAwesome, Devin. So thank you so much for, uh, taking the time today to explain bug bounty to us and, and for our listeners. And I want to extend an invitation for you to join us in season 4 to talk about PCERT, which I know is another thing that you're passionate about, and we didn't have really a lot of time to get into that today, so we'd love to have you back in season 4, and thank you for taking the time today.

28:48Devin McMastersHey, thanks a lot, Chris, and of course I'd love to come back.

28:51Chris RomeoAwesome, thanks.

28:54Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.