Matt Tesauro -- #AppSec Pipeline as Toolbox

0:00Chris RomeoHey folks, welcome to another episode of the Application Security Podcast. This week we're joined by Matt Tesauro, a co-lead of the AppSec Pipeline project. He talks about how they got started building this project and some ways for you to dive in as well. Hope you enjoy.

0:15Robert HurlbutThe Application Security Podcast. Here we go. Hey folks, we're still at AppSecEU and we have a chance to speak with Matt right now. Matt, would you introduce yourself real quick for the audience?

0:56Matt TesauroYeah, sure. Hi, this is Matt Tesauro.

0:59Robert HurlbutAnd Matt, we always start off with our security origin story. So how did you get into the world of security?

1:05Matt TesauroUh, it was kind of through a circuitous way, I guess. I started out after getting done with university and went to a telecom provider that oddly enough was— the data center was in Texas, but all of our— the vast majority of our clients were in Europe, particularly France, Belgium, Netherlands, and a bit in the UK. So as a developer, when I nubbed it, I had somebody angry in Europe waking me up at 2 AM to drive back into the office and fix things, which if for nothing else is a great incentive for quality. But I went on from that job to start— move to the actual Texas A&M University and work for them. And I hated the systems I was writing software for, so I started becoming a sysadmin. And from there I kind of got a bit of a security bug, got my CISSP, and surprisingly, I wouldn't say this was a rational decision, but my next employer decided that because I had a CISSP, I'd make a great pen tester, which was a stretch, but I took it. And I started doing pen testing, did that for quite a while as a consultant and also working internally. And then my first real AppSec job was at the Texas Education Agency where I was a one-man band. There was me and 120-odd developers, and I just, I introduced them I had one of the QA people tell me that they loved me because I really shook up things when I showed up and started pointing out security issues.

2:28Robert HurlbutThat's a good thing.

2:30Matt TesauroAnd I just, from there, I found OWASP and have just been doing AppSec ever since.

2:34Robert HurlbutVery cool. And are you involved in the Austin— did you say, what part of Texas are you in?

2:39Matt TesauroI live in New Braunfels, which is midway between Austin and San Antonio, so I end up going to both.

2:46Robert HurlbutOkay.

2:47Matt TesauroI sometimes go to Austin, sometimes go to San Antonio. I'm about an hour to 45 minutes either direction, so it's not too bad.

2:52Chris RomeoOkay, cool.

2:52Robert HurlbutAnd the, um, the— I, I too had a little bit of a background in education, so I think that's a great playground for a number of things. And so, you know, we always have different listeners who are maybe new career people and stuff, and, and I always encourage folks, like, you know, if you can get a job early in education, it's, it's good because They don't pay, they're never going to pay you very well, but they'll give you a playground.

3:16Matt TesauroYeah. Oh, I got to do all kinds of experimentation when I was working at A&M. This is not necessarily security related, but I just got a wild hair to play around with Gentoo Linux, and that's a kind of compile-everything version of Linux.

3:27Robert HurlbutYep.

3:27Matt TesauroAnd I ran, I took some of the old beige boxes out of a student lab that we were going to retire and moved them up to a room and made a distcc cluster. And so it does, it doesn't pay well, but the fringe benefits, the ability to play, experiment, and And actually, that's where I found WebGoat. I was writing apps for the university and I somehow, I don't remember how, stumbled upon WebGoat, the first OWASP thing I ever really ran into. And I went through the examples in there and went, oh crap, I bet I have SQL injection in the thing I wrote. And sure enough, like I went back and looked and yeah, it was injectable and the rest is history.

4:02Robert HurlbutYeah, wow, that's cool that your eyes were open to OWASP right in the EDU space. And yeah, there's I mean, find a corporation for me where they'll let you just grab a bunch of computers you find in a room and build a lab somewhere. Like, nobody's going to let you do that in corporate America. So that's, yeah, that's great. So I know you're heavily involved in the AppSec Pipeline project. And why don't you tell us, start out by just telling us what is this project and what are the pieces of it?

4:31Matt TesauroSure. It was, it started out with myself and my co-lead, Aaron Weaver. Realizing, well, it started out really with us realizing there's a lot of good stuff at OWASP, but if you're running an AppSec program, there's not like a nice place to find those kind of things. You know, there's— I know because I've been in the business for a while that I can use A, B, C, and D and they're useful, but maybe if you're new to it or you've just, you know, suddenly been given the, hey, by the way, you're doing AppSec badge, you don't know what is useful and what isn't of the suite of things that OWASP has.

5:02Chris RomeoGot it.

5:03Matt TesauroSo it really started with that. idea initially, but what it sort of morphed into was an idea— at the time, Aaron and I were working together at the same employer, and we did some interesting automation around security. And it was really to make— well, to be— well, at that employer at the time, it was a very large company. They had combined multiple divisions, and they were centralizing a lot of the security. And so we had this very diverse global org that had thousands of applications and a very small team. And the very small team was sort of glued together from all these different pieces, and it was just kind of chaos. And so we kind of took a deep breath and said, how— what is the fundamental workflow of an AppSec team? And we defined that and added automation to it and called that the sort of the first gen, if you want to call it, of that AppSec pipeline. And it was really inner-focused. It was like, how do I make the AppSec team go faster.

5:58Robert HurlbutAnd this was— and this came to you out of necessity because you didn't have— it's not like you had 100 AppSec people on your team and you could distribute it across people. You had to solve it with tech.

6:09Matt TesauroWe had to solve it with tech because we had— I'm trying to remember the numbers, it's been a couple years. We had, I think, 6 or 8 people total in the AppSec team. There was, depending on who you ask, because that's one of the things App Inventory is sort of dicey, I think we had— I heard the numbers between 1,500 and 2,500 applications across the business. and the developers were in the 6,000 range.

6:30Robert HurlbutOkay.

6:31Matt TesauroSo like, yeah, those ratios are awful. And so if we're going to get any kind of coverage, we can't do it with bodies, you know, manually doing testing.

6:40Robert HurlbutYep.

6:41Matt TesauroAnd yeah, that's what drove that sort of first gen of the application pipeline. And it was really, let's figure out how to do some automation and some glue code to make our lives better so that we can do work quicker, faster. And we went from Year 1 was 44 assessments. 2 years later, we did just over 400.

7:01Robert HurlbutWow.

7:01Matt TesauroYeah, so over 2 years, we— 9— I don't remember the number from my slide deck, but it was like an older slide deck. I think it was 9.4 times over 2 years increased the throughput of the team.

7:11Robert HurlbutWow. So what are some of the OWASP tools that people would recognize that are included in your AppSec pipeline here?

7:20Matt TesauroSo that's an interesting question and a great question because at least in the second generation, the pipeline was really just a conceptual model. This is the framework in which you would actually lay out your own pipeline because that's one of the tricks. Like every AppSec application team, honestly, an AppSec team has to deal with a unique environment.

7:37Robert HurlbutYeah.

7:38Matt TesauroLike everybody has their own very snowflakey idea of how apps should be built. And so we wanted to give a very broad framework to get that done.

7:47Robert HurlbutOkay.

7:47Matt TesauroThe sort of the one that I talked about at AppSecEU just now, that's what I would guess call, for lack of a better word, a 3rd generation of this thinking. And we have a specification and a concrete implementation of a tool that I call GASP, 'cause I'm bad at naming tools. It's the Golang AppSec Pipeline, 'cause I could backronym that.

8:07Robert HurlbutOkay.

8:08Matt TesauroAnd that takes Docker containers, with predefined tools and then runs them against— runs one or more of them in a predefined order against a target, and a target being either a running application or source code.

8:21Robert HurlbutOkay.

8:23Matt TesauroAnd so that I announced today at AppSecEU, and that's probably the latest generation of it. But the sort of the other big part of that is an OWASP project called DefectDojo.

8:33Chris RomeoOkay.

8:34Matt TesauroThat is a single source of truth for the activities in your application security program. So you track engagements, which is sort of just a way to say I'm talking with or interacting with an application in some sort of way. And it might be a single thing like I'm doing a threat model, or it might be 4 or 5 things. I'm doing a threat model and looking at the static code, and I'm looking with dependency check at their library management, and I'm doing a manual test, right?

8:59Robert HurlbutAnd those activities or engagements are designed to be something that the developer themselves— is it the tool is reporting this or the developer's reporting that?

9:07Matt TesauroThe tool is reporting that for you. track it. A lot of that was driven from when I ran the product security group at Rackspace where inevitably you'd get a manager drive-by and say, hey, this is of course the critical thing, drop all your tools and go work on X.

9:22Robert HurlbutYep.

9:23Matt TesauroAnd for a while I just had to from the hip decide, well, what is that going to cost the org? Well, now that I'm tracking it, I can say, great, I can go work on X, but Y, Z, and W aren't getting touched. And you'd have a lot of interesting conversations with management at that point of like, well, actually Y and Z are really important too. but take the guy off W and put it on X. Great, we can do that. And now it allowed me to sort of have sane conversations about resource allocation on my team because, I mean, everyone's agile and things are switching around quickly and you have to shift work, but I didn't really understand the cost of that shifting until I had it managed with something like DefectDojo.

9:56Robert HurlbutSo, okay, so that allows you to quantify—

10:00Matt TesauroAll of the sort of activities that are in flight or planned. Yeah. And then the other aspect of it is there are importers for tools and also a generic importer if an importer doesn't exist to pull in results from n number of security tools and normalize them into one sort of normalized format.

10:20Robert HurlbutOkay.

10:20Matt TesauroAnd then you— it does deduping, false positive management, and those kind of things. And then from there, I can take those results and push them into a bug tracker. I can do reporting or metrics and all that other good stuff.

10:31Robert HurlbutOkay. So Defect Dojo is a fundamental piece of this AppSec pipeline then?

10:36Matt TesauroYeah. And I'm, I'm lazy. Go figure. I, and I don't do UIs very well. So the Gasp tool is a command line tool that's really just meant to run behind the scenes, but you need to sort of surface those in a usable fashion. And I use Defect Dojo for that.

10:50Robert HurlbutAs the way to get it. Okay.

10:51Matt TesauroYeah.

10:53Chris RomeoAfter the break, Matt talks more about the tools in the AppSec Pipeline project.

11:00Robert HurlbutThe Application Security Podcast operates with support from Security Journey. A security belt program provides the 3 pillars of successful AppSec training: learning, application, and experience. Visit us on the web at www.securityjourney.com to learn how you can teach and empower your developers using a new kind of security training.

11:26Chris RomeoMatt dives back in answering what other tools are found in the project.

11:30Matt TesauroWell, the specification just talks about how to make the tools, but I— we already have Zap, we have Dependency Check, uh, we have like 15 or some 20-odd doctors. I don't remember exactly how many. We have a bunch of tools, um, some of which are OWASP, some of which aren't. And actually, funny enough, the AppSec pipeline is an OWASP project and and Defectojo was an OWASP project.

11:52Robert HurlbutYep.

11:53Matt TesauroSo yeah, we're definitely dogfooding a lot of the OWASP stuff.

11:56Robert HurlbutOkay.

11:56Matt TesauroUm, but really it's, it's, it's, it is about the tools, but it's more about automating the tools. And the other thing is you, you understand your workflow very well. It's very well defined if you create this sort of pipeline idea. Like if you think about a CI/CD pipeline, right, that is I'm gonna run these things and at the end of it I'm gonna produce an artifact that I can go deploy to a server, or maybe ideally it is deployed to a server.

12:18Robert HurlbutYep.

12:18Matt TesauroSo for an AppSec pipeline, at least in the way we conceive of it, you run these things and at the end of it the artifact is findings, security findings about the application that you just reviewed. So it's— fundamentally, it's a way to sort of— well, the best use of it in my mind is to have an automated way to do baseline testing across the suite of applications that your program has. So if you have 200 applications and they're in various languages and you have a tool in the pipeline for all of those languages, I can run— like if you say have 36 Python apps, right? I can run Bandit, a Python static analysis tool, against all 36 apps and now I have a relative positioning in terms of security of all of my suite of Python apps, right? Because if this one is very noisy because it's a basket case and this one's very quiet, I now can make rational judgments. Well, This one here is a basket case, but guess what? 3 people use it. It's only available on the internal LAN. Eh, I don't really care about it. But this one is hypercritical, and it only has 2 things, but it's hypercritical. We're going to go fix those 2 things. So it's really to get a map or a landscape of your suite of applications by doing sort of a baseline check. And then from there, you can manage your program much more intelligently because you have visibility now into into your whole suite of applications. Like, I used to, I used to ask at conferences, like, raise your hand if you have too many AppSec people, right? And obviously no hands raised. And my other favorite one now is to say, raise your hand if you know all of the applications that your business have. And I did that today, and I had, I don't know, 50-some-odd people in the room, and I had one guy raise his hand.

13:55Robert HurlbutYep.

13:56Matt TesauroAnd I've never worked at a place where I knew them all. You just find them. You turn over rocks and like, oh crap, there's another application. You know, marketing launched this thing, and then somebody else launched this other thing, and that's just how it happens.

14:04Robert HurlbutI still can't believe one person raised their hand.

14:06Matt TesauroYeah, I know. I was kind of like, I wanted to go question him, but I was in the middle of a talk.

14:09Robert HurlbutProbably has one app.

14:11Matt TesauroWe have one app. We have one app, so I know all of it.

14:14Robert HurlbutIt's a big monolithic thing. It has everything else in it, but it's one— we call it one app. So when I— so if I think about the AppSec pipeline, so can I— is it, is it correct for me to think of this as a toolbox of things that I can borrow? Like, I can take these Docker containers and maybe I can assemble them in a way and an order that might be different than how you approach it.

14:33Matt TesauroYep.

14:33Robert HurlbutBut so it's, it's a—

14:34Chris Romeoit's—

14:35Robert Hurlbutso for me to use the AppSec pipeline, I don't have to follow— it's not going to give me steps A through Z.

14:40Chris RomeoNope.

14:41Robert HurlbutIt's going to give me tools.

14:42Matt TesauroIt's going to give you a bunch of Lego blocks in different colors, and if you want to do red, green, blue, that's cool, but I like blue, green, red. It doesn't really matter.

14:48Robert HurlbutAnd it's okay that we have—

14:50Chris Romeoit's, it's—

14:50Robert Hurlbutit gives us— we still get the same end result of baseline security testing that fits our organization.

14:57Matt TesauroYep, because I didn't want to be prescriptive because I've worked in too many AppSec programs to, you know, to— I just know that fundamentally none of them are the same. I mean, there's similarities, but they don't— like the internal way that apps are developed, we use GitHub, we use branches, we use tags. There's all sorts of different ways you can run an AppSec program. So Aaron and I were very conscientious about making this sort of abstract, which also makes it a little bit tricky to explain sometimes.

15:19Robert HurlbutYeah.

15:19Matt TesauroBut it also has the benefit of I'm not going to tell you, no, no, you must run them in ABC. If you want to do C, B, A, awesome. If you just want to do B, awesome. You want to do D that I don't even do, awesome.

15:30Robert HurlbutDoesn't matter. Okay, so what— how would you recommend somebody get started with this AppSec pipeline? Say we have, you know, some of our listeners who've, like me, had never really heard a whole lot about it, didn't really understand it. Now they've got a grip for what they can do with it. Where do you point them to start?

15:44Matt TesauroAh, so the project has some documentation on it that gives sort of a high-level abstract generic chart of like, this is your template for AppSec pipeline. I would say that's probably a good place to start. And then the GASP tool, or actually the gasp-docker, the one that I wrote and is up on GitHub, that one is an implementation of the— if you think— well, let me go back. I'll back up a step. There's different phases in the pipeline. There's intake, right, where you have things coming in. I have my boss is telling me I must check There's a PCI assessment, we just rolled a new app, whatever it is, there's some sort of intake, right, that happens. The next phase of it is triage. I look at what I have coming in and I need to do something with it.

16:26Robert HurlbutOkay.

16:26Matt TesauroThe middle phase is testing. I need to test it. And then the final phase is what we call delivery, and that's putting stuff into a vulnerability repository like DefectDojo and then pushing out metrics and reports and bugs into bug trackers. So the, the GASP tool is really designed for that testing phase. It's an implementation of how you can do testing automation in that middle slice of the AppSec pipeline. Dojo now is feature-rich. They've done a ton of commits in the last month that really you could use Defect Dojo to handle your intake and your vulnerability repository. The first time I did this, you know, and I'm learning, we actually had 2 different apps to do it, which was sort of cool because we could— I could on app A and my coworker worked on app B.

17:11Robert HurlbutOkay.

17:12Matt TesauroBut the problem is you have to keep those in sync.

17:14Robert HurlbutYep.

17:14Matt TesauroAnd like the application named in the first one isn't named as the first, you know, the second one, and it's syncing them, and my ID 7 is your ID 12, and it just made it complicated. So in subsequent installations that I've done of the AppSec pipeline, we've simplified it into one piece.

17:29Chris RomeoOkay.

17:30Matt TesauroCode, you know, like application-wise. Conceptually there's 2, right? You have intake and you have delivery, but If you can have one app that handles both of those, you're way further up.

17:39Robert HurlbutSo the template though is the place for somebody brand new, just pick up the template, start studying that as a way to look for, you know, what are the pieces they should add in.

17:50Matt TesauroYeah, and you can go to YouTube and put in AppSec Pipeline and a bunch of my previous talks are there. And I've been doing these talks for a couple of years, so you can, if you really are like into history or something, I guess you could watch the evolution of them or watch the latest one if you want to know my latest thinking on it. A lot of this was like, I did a talk on this, shoot, 4 years ago, and I didn't even— I hadn't really conceived of the idea of an AppSec pipeline, but I knew what I had wasn't working. And my old way of sort of doing AppSec, and I said this in my talk, you know, like I think traditional AppSec in the I have a week of testing and, you know, this is, oh, you're done, let me test it, that doesn't exist.

18:26Robert HurlbutYep.

18:26Matt TesauroIf you have a place like at Rack, we had 75 deploys a week, there's no testing window. You just test. when you can and as quick as you can.

18:32Robert HurlbutYeah, yeah. And I guess, okay, so, so that'd be a good place for folks to start. Um, so real quick before we wrap up here, you are back at OWASP.

18:42Matt TesauroI am back at OWASP.

18:43Robert HurlbutWhat is your role at OWASP?

18:45Matt TesauroUh, what is my title? Uh, I am the Director of Community and Operations. Um, so what that really means is I handle a lot of the, the backend processes, um, to try to manage, uh, chapters, projects, membership, all that kind of— the services in essence that OWASP Foundation provides to its members.

19:04Chris RomeoOkay.

19:04Matt TesauroSome of those are better than others and we're going to update a lot of them because they need some help. And then the other side of it, the community side of it, is primarily chapters because that's our sort of boots-on-ground local, you know, collections of people that help this organization be successful and get the word out, you know, to your local area. They meet usually monthly. So check your area. We have on the OWASP wiki, www.owasp.org, a list of all the chapters around the world, 220-some-odd of them. So yeah, that's, that's my kind of oversee chapters and then just the processes and the IT-ish stuff. Yeah, behind the scenes.

19:38Robert HurlbutAnd that's, that's a great advertisement for— I'm the co-lead of the Raleigh, North Carolina chapter. And so yes, if for any of our listeners, if you have not been to an OWASP chapter meeting, Find one. Like Matt said, there's 220 of them worldwide, and they're in most major cities, and it's just a great opportunity to go out to a non-commercial. So a lot of meetups these days, it's all about the sponsor, and you gotta listen to somebody drone on for 20 minutes about their product or something that they're doing.

20:06Matt TesauroRight, their magic box.

20:07Robert HurlbutNot at OWASP. That's not what we do. So it's all about education and networking and the connection side. And so yeah, that's, that's exciting. You're going to be a part of that, and I'm guessing you're going to be making some more, you know, more big improvements in the future.

20:22Matt TesauroWe're hoping to do a lot of that. And it was 10, almost a little over 10 years ago in 2008, I found OWASP, and that was when I was working as a solo AppSec guy at this, you know, 1 to 120 ratio, uh, place. And, and I tell you, particularly if you're kind of the lone wolf, like going to an OWASP meeting and sort of being— for me, I was able to sort of hang out with my tribe.

20:42Robert HurlbutYeah.

20:42Matt TesauroAnd talk shop. That was So useful because at the day job I was kind of, you know, an isolated little island of AppSec in the sea of people doing things and, you know, making apps and whatnot. And it was very nice to be able to go and, gosh, I had this thing with this, you know, app and it wouldn't do this, and I could, you know, commiserate with a friend or get some ideas from a friend. And yeah, OWASP, I love it.

21:02Robert HurlbutYeah, that's a great, that's a great point for the people that are out there that are operating on their own or in a small group. It's a great place to get different perspectives and In my experience, people are very open. Oh yeah, willing to share and talk, and there's no NDAs or anything required. People just share things, successes and things that are, you know, broken, and they're wide open. So it's good stuff. Well, Matt, thanks for taking the time here today. We really appreciate your perspectives and teaching us about AppSec Pipeline. And good luck as you continue here with the OWASP Foundation.

21:30Matt TesauroAll right, great. It's been— thank you for having me. It's been great. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @appsecpodcast or on the web at www.appsecpodcast.org.