Bill Sempf — Growing AppSec People and KidzMash

0:00Chris RomeoWelcome back to the Application Security Podcast. This is Chris Romeo, one of your co-hosts and also CEO of Security Journey. On this episode, Robert Hurlbut does an interview with Bill Sempf at the CodeMash conference, and the topic of discussion is how do you grow AppSec people, or how do you convert developers into application security professionals? Fascinating topic. And they also get into another project that Bill's been doing through CodeMash called KidsMash, or How do you get 750 kids interested in the world of cybersecurity and provide an event for them at a conference? So lots of interesting things to hear about, and we hope you enjoy. This episode of the Application Security Podcast is brought to you by Security Journey. Security Journey has a new weekly publication called High Five, 5 security articles that are worth your time. We scour the internet looking for the best articles on application and product security.

1:00Bill SempfWe add in just a touch of sarcasm snark in our descriptions.

1:05Chris RomeoJust what security people and developers love. To sign up, visit www.securityjourney.com/highfive. That's /hi5. The Application Security Podcast.

1:30Robert HurlbutHere we This is Robert, and I'm actually at the CodeMash conference here in Sandusky, Ohio at the beginning of January. And with me today, I have Bill Sempf.

2:01Bill SempfHello, everybody. Pleased to be back again.

2:04Robert HurlbutAbsolutely. So Bill has been with us before. He's been on the program. If you remember back in 2018, he spoke to us about one of the topics in the OWASP Top 10. And so Bill is speaking here as well as I am at the Codemash conference. There's a few security talks, and glad to have you, Bill.

2:24Bill SempfThank you.

2:24Robert HurlbutSo tell us about, let's start off with what your talk is this week.

2:29Bill SempfSure. So one of the biggest problems that we've got in application security in general from the soft skill standpoint is we need people. And we just, I get 15 phone calls and emails a week from recruiters that are looking for someone to fill every role from, you know, basic vulnerability analysis and assisting developers on the security side all the way up to advanced security architecture roles and finding solutions to really hard problems in software architecture, like rolling old identification schemes up to today's standards. I mean, they got, hey, we've got RACF and we need to meet current PCI standards, help. It's a big problem. And the issue is that Well, what I've always said in the past, um, when people ask me, hey Bill, how did you get into application security? I probably said that in the previous show. Um, well, first you spend 20 years developing web applications, and then you have to learn what all the bad guys do. And it's, it's a, it's a totally different skill set. So what I've done over the past maybe 3 years is I've been mentoring 5 people, um, and I've— from completely different backgrounds and skill sets, and I've kind of gathered their stories together and gleaned what I could out of what we need to do to grow our own AppSec people. And it's not unlike the stories that I've given in the past, but I have learned a couple of new things about— a little bit about— I've learned a little bit about drive and like what makes people want to do things and how to find people who have some drive and how to maybe encourage some drive in people who maybe aren't performing to their personal maximum, which is weird for me because that's— this is not my area. I'm not an HR person. I'm— my social skills are lacking. I'm— but, but I've learned— I've actually learned a lot, um, about that. So, um, I, I talk about them and then I talk about what skills people need to have and then how to get those skills into your, uh, into your development group, um, and, and, and start isolating folks that might be good candidates to be an AppSec person even though they don't know it themselves. Um, so it's really just kind of a reflection on the stuff I've learned, which is, I mean, that's kind of what most of these talks are really. If you think about it, that's, that's really what a conference talk is, right? Hey, I learned this really cool thing and I want to teach you about it. This is a different talk for me because it's almost a soft skills talk. I talk about the technical stuff that people need to know, but I talk about it more from how do you teach them and less about here's how you do it, which is the way I've done it, what I've been doing for however many years.

5:33Robert HurlbutTell me about drive. You mentioned those who have that drive, but also maybe cultivating that drive. What do you mean by that?

5:40Bill SempfIt's so hard. I don't have all the answers yet, but it's unquestionable that So the main crux of the lessons that I learned from cultivating an individual in your group is what I've been doing, and I've done it successfully now at 5 different companies, a couple of them big, is you hold a training and instead of just giving them exercises or, you know, having a copy of WebGoat up for them to hack at, which is cool, you actually set up a CTF with CTFD and make it a little competitive and give them no prizes. No, nobody's gonna be reviewed. There's no quiz at the end or anything like that. That's not the point of application security training, and it shouldn't be. So don't, don't try to make it that. But those people who stay over lunch for a couple minutes to solve something, or stay when the training's over to ask questions about something that's a little bit more advanced, those are the folks that probably have that itch to learn more. And one of my mentors, Brent Houston, said to me one day— I said, I asked him, what are you going to do when you retire? And he's like, people like us never retire. We're going to be doing this in some way forever, and you know that. And you don't want people to live to work, and I don't recommend that people do, and I'm the first one to say that. I'm absolutely the first person to say that. But I mean, my RSS feed is full of security stuff, and I sit and flip through blogs when I'm having lunch, and that little bit of extra, as long as you can make the home-life balance work, which is more important, frankly. But if you can make that work, then that drive is what you're— that constant curiosity of, man, what is going on out there and how can I better involve myself in it? That's what you look for. But I don't want to be one of those people that says, well, show me your GitHub profile with, you know, 7 check-ins a day for the past 20 years. No, that's not what I mean. But you need a little teeny bit of that to do this because you're not going to learn— if you're a dev, you're not going to learn AppSec on the job. You're just not. You've got to have a little bit of that edge to want to do it on your own, the way you learn with your stuff. And that's what we're looking for, I think. I think anyway.

8:23Robert HurlbutOkay. And you mentioned CTF. For our listeners, what is CTF? What is a CTF?

8:28Bill SempfOh, so a CTF is a capture the flag. We're running a cool one here at CodeMash, actually. But what it is, is basically it's a technology treasure hunt, right? Where you've got certain problems that somehow relate to some little edge of technology. Like there's one that we have with a an SVG file that has a string encoded in it somewhere called a flag that lets us capture the flag. And you can, if you know how to deal with SVG files, you can get, you can glean the flag out of the SVG file. If you don't know how to do anything with an SVG file, well, guess what? Look it up. And that's what we're trying to get people to do, right?

9:19Robert HurlbutYeah.

9:20Bill Sempftrying to get people to learn things. Today I sat with, what, 20— what was it? 10 to 15-year-olds and walked them through a couple of capture-the-flag-style challenges in web application security using an OWASP tool called Security Shepherd. And we had to go— they had to go in using the developer tools in Firefox and resend the post using, um, after modifying the, the, um, the post data and see how the application responded differently. And that's what a CTF is all about. It's, it's to make you learn new stuff. And the CodeMash CTF that we have up, I, I've learned— I don't know if you've looked at it yet, but I've learned a ton of stuff already, and I'm supposed to be running it. And I learned a bunch of stuff. I didn't write all the challenges. that you have to reimplement RSA security for one of the advanced challenges. And it's just, it's awesome. And it makes you go and learn, go and find something that somebody else has written and learn how it works and nail it to the ground. And that's what we're looking for.

10:34Robert HurlbutOkay, very good. Now you mentioned working with some kids here. You have a KidsMASH.

10:38Bill SempfWe do.

10:38Robert HurlbutTell me about that.

10:40Bill SempfSo CodeMash is a pretty big regional developer conference, 2,500 people for the main conference on Tuesday and Thursday— or I'm sorry, Thursday and Friday. And then, oh gosh, I don't know what, 8, 9 years ago, one of the attendees just kind of said, hey, I'm going to put up a sign and if anybody who's here with their wife and kids, because we're at a big indoor water park in Sandusky, so people bring their families.

11:05Chris RomeoYeah.

11:05Bill SempfIf anybody wants to, who's not attending the conference, wants to get together for lunch, meet here at 11:30. And so, you know, 20 families met and they went to lunch and it was fun. And then they were like, oh, well, let's go to the water park at 3 together as a group. And then, oh, let's get together and do some stuff later on. And that was fun. And then it was, heck, let's have a couple of speakers come in and talk about tech topics. And that was fun. And then it was, then CodeMash. took notice and, and they're like, hey, this is kind of awesome. Can we like do this for real and put it in the schedule? And, um, feathered a little bit of a feather, but the, uh, then my wife took over, who is an extremely good organizer. And, um, it was— we have 2 young children, so it worked out very well for us. Uh, it became— what do they have now? 22 speakers.

11:57Robert HurlbutWow.

11:57Bill SempfAnd, uh, 750 registered kids. A makerspace with 10 STEM projects that they can come into anytime, including robotics and electronics and working with friction and working with senses and all kinds of stuff. Kids from 0 to 18 look forward to it every year. And we're— I mean, there are many people who post that their kids talk about the holidays, you know, oh great, it's time for the holidays, Thanksgiving, Christmas, and CodeMash. Or KidSmash, you know, the Kalahari in general. So it's, um, it's, it's, it's a, it's a big part of the, the con. And, uh, you've seen it before, the, the, all the kids that during the main part of the con, you got all these little green-shirted kids running around. It's really awesome.

12:46Robert HurlbutIt is, it is fantastic. So you get, uh, people getting exposure early, um, and then of course developers getting exposure to a lot of different technologies and tools and so forth. But what I'm really excited to see is definitely the beefing up of the security, application security.

13:04Bill SempfAbsolutely. All the talks. Yeah, Mike Woolard took over the security track and we have just an absolute slate of talks. I think there's 14. We've had many luminaries of the security industry come by and talk to decoders, which is just awesome. I mean, to get people who are, you know, really well established in the security world and have to deal with the application security side on their part get to come and talk to 2,500 developers and say, hey, this is something you need to know about, you know, whether it be logging or whether it be input validation or whether it be whatever. It's an important lesson to learn, and it's been extremely popular. We talked about origin stories a little bit ago, but you might not know the origin story of the track. I'll tell you that real quick.

14:01Robert HurlbutI'm curious.

14:01Bill SempfYeah. Rob Gillen, who is the speaker guru for the whole conference, he's the one who kind of organizes all the speaker track stuff. He and I, before he did that job, we gave the first Security Precompiler here, and it was a scattershot. I mean, I talked about encryption, he talked about Wi-Fi security, I talked about like patching binaries and doing crackmes. Full day session, 8 hours training, uh, during one of the precompilers. They had to move our room 3 times because the fire code people were complaining because we filled the room up so fast.

14:38Robert HurlbutWow.

14:38Bill SempfAnd that's when we went, you know what, I think maybe this whole thing about developers not liking to talk about security is kind of not true.

14:45Robert HurlbutRight.

14:45Bill SempfSo the next year we had a security track. We had 6 talks or something, and then more and more and more and more. And right now we get, you know, guys like you and Jumanneko in, for crying out loud, to do stuff. So it's, it's just, it's, it's really gotten big, and I'm so glad to see the developers interested. And, and they're running around doing the CTF. They have to come find me to, to collect one of the prizes for the CTF, and they're all just addicted. This is so much fun, this makes this conference even better. And I'm going, this is, that's just awesome.

15:15Robert HurlbutIt's a win.

15:16Bill SempfIt is a win.

15:16Robert HurlbutAbsolutely, absolutely. Yeah, I was really excited to see yesterday in my pre-compiler that I ran is that so many developers, I wasn't sure how many were to come to a threat modeling talk, but so many did. And how many of you know about threat modeling? And maybe one hand.

15:29Chris RomeoOh, wow.

15:31Robert HurlbutGreat.

15:31Bill SempfOh, I didn't know that, that's awesome.

15:34Robert HurlbutFirst time learning about it, try to see how they can use it and so on. So that was—

15:37Bill SempfSee, that's the only way we're gonna solve this problem is, I mean, so many of these people, they're awesome, they're great devs, they really want to improve their ability to solve development problems, but they just don't even know where to start in application security. And whether it be a capture the flag competition or having their kids do something weird and then going, huh, or going to a threat modeling session for the first time or happening to catch an input validation talk by Manico or whatever, that's going to get the path started. They're going to go, oh wait, there's a whole thing here I can follow, and it doesn't even take a tremendous amount of energy. We can just go this way. That's what I want to see happen.

16:24Robert HurlbutAnd that's kind of what we're— in terms of, for example, the Application Security Podcast, we were hoping to reach not just security people but also developers and helping them understand about application security because As you said, I mean, it's the developers who are going to be building the products.

16:39Bill SempfIt's awesome that we have so many people who come up from the general let's make things better stack that you tend to think of as operations security people, you know, the network people and folks like that who lean towards the security side that can tell you when something's wrong. But we also really need to train the coders.

16:58Robert HurlbutRight.

16:58Bill SempfAnd that's what we're shooting for here. And it's a small thing. I mean, it's You say, you know, 2,500 people, well, that's obviously a tiny percentage of the total number of software developers in the world. But you got— man, you got to start somewhere. And if each of these people tell 5 people, that number gets up there.

17:17Robert HurlbutAbsolutely. Well, I know you're speaking on the topic, you're finding new people. What else can you tell us about the topic in terms of You found them. How do you grow them? How do you continue to mentor them? Or should they look for mentors? I mean, those kinds of things.

17:36Bill SempfYeah, that's a future topic for sure. I don't have a lot of solutions that direction yet. There's an InfoSec mentor program somebody had set up online. I don't remember who off the top of my head that actually I found one of my protégés on. which was nice. But in general, I mean, what I'm telling people now is get your, you know, get your developers trained and isolate the security champions and then have their information— have your information security team get them rolling, you know. And then from there on, I mean, OWASP is the path. I think. I don't mean to sound like a religious zealot or anything, but it's a massive wiki full of the combined wisdom of 44,000 people over the past, what, 16 years?

18:35Chris RomeoMm-hmm.

18:36Bill SempfThat's where you get started, and it's getting better organized every single day. So shoot them at OASP and have them go to a OS meeting, get involved with people at those organizations, and then pick a project and get started. I'll throw a towel in the ring here. If anybody's listening who's a .NET person, I run the .NET project. We're looking for people to write articles on stuff that isn't covered on docs.microsoft.com. So if you've solved a cool problem in Microsoft security, let me know and write about it and we'll put it up there. That's what all the projects are like. I mean, well, Some of them are coding, but most of them are documentation and just learning and places for people to look things up. So that's probably step 2.

19:24Robert HurlbutOkay, very good. All right, well, Bill, thanks again for joining us. We always appreciate time with you to learn from you and hear what's going on, and look forward to maybe another time in the future.

19:36Bill SempfIndeed. Thank you very much. All right, thank you. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @appsecpodcast or on the web at www.appsecpodcast.org.

20:11Robert HurlbutThanks for listening.