Daniel Ramsbrock -- Web Application Pen Testing – Part 2

0:05Robert HurlbutThe Application Security Podcast. Here we go.

0:10Chris RomeoHi, this is Robert Hurlbut, and on this 2-part episode of the Application Security Podcast, Chris and I speak with Daniel Ramsbrock about web app penetration testing. This is part 2 of the interview where we focus on the process of pen testing and web app pen testing.

0:42Robert HurlbutDaniel, walk us through the process that you go through to actually run— go through a test.

0:53Daniel RamsbrockSure. So, you know, I always start out with trying to gather as much information as I can about the application and its sort of immediate environment, you know, any dependencies that it has or components that it interacts with on the backend. So, you know, and that varies. Sometimes, you know, the clients are very open and provide lots of documentation and opportunity to speak with the architects or developers or, you know, the team who's managing the application. So that's always very valuable. You don't always have that luxury. So then the next step after that is to perform some basic reconnaissance, you know, just basically, you know, log into the application. And well, that's another important point. So one of the key things that always, you know, that can be a sticking point and that it's a key difference between regular penetration testing and application penetration testing is the fact that, you know, 99% of the time you need credentials. You need at least one, ideally multiple accounts.

1:51Robert HurlbutYeah.

1:53Daniel Ramsbrockto log into the system.

1:54Robert HurlbutNow, why would you need individual credentials here?

1:56Daniel RamsbrockRight, because a lot of clients struggle with this too, because they say, well, you know, you're a hacker, you're supposed to just break in and— Yeah, I watch Mr.

2:05Robert HurlbutRobot on USA, so you should be able to just type a bunch of keys on Kali Linux and make my website explode.

2:11Daniel RamsbrockExactly right. Why do you need to log in? Well, you know, my answer to that is, well, unless you have some really bad vulnerabilities, like injection on the login page, you know, it's going to take a lot of my time to— I may eventually be able to get in, but I'm going to spend a lot of the allotted time that I have just trying to get into the application. And if I don't get in, then, you know, 90+% of the potential attack surface of the application will remain untested, namely all of the things that are behind the login page that I would not have access to without a user account.

2:45Robert HurlbutOkay, so you want that. So it's going to be better, the customer is going to get a better return on their investment if they provide you with the logins and let you actually take the same approach that one of their customers would have.

2:59Daniel RamsbrockExactly, exactly. The assumption should always be that an attacker has at least user-level access to the system, and I always encourage clients to consider also the admin perspective, right? So providing me with an admin account as well so that I can see if, if I can, for example, log in as a regular user but, you know, forcibly access the admin functionality without having those privileges. And so those are called, you know, privilege escalation attacks. And so a lot of manual web application testing revolves around those types of activities. And so I usually request, you know, 2 sets of credentials for each role that the application has. And so usually there's at least a regular user role Sometimes there's an admin role and, you know, and if the application is more involved than that, there might be other sort of in-between roles that have, you know, more privileges than a regular user but not quite admin privileges. And so what I'd like to do is to, you know, to try kind of all the different combinations of that matrix. So in a simple case where you have regular users and admins, you know, I will log in as user A and try to access, you know, the account information for user B that I should not have access to. I'll try to log in as, you know, a regular user and try to access admin functionality that I should not have access to.

4:21Robert HurlbutNow, is this during— are we still in the reconnaissance phase, or have we moved into an active attack, or is this the brainstorming section, or where are you in the process right now?

4:31Daniel RamsbrockThat's a good point. So yeah, I kind of skipped ahead a little bit. So yeah, during the reconnaissance phase, I'm gathering, you know, the— I'm just logging in with the accounts that have been provided, basically. And I'm just going through the application and clicking every menu that I— as I'm doing this, so I should back up a little bit. As I'm doing this, I'm pumping all my traffic through a proxy, a recording proxy, for example, Burp Suite or something similar, similar tool. And all the traffic between my browser and the application is being recorded. And so I just click through the application. I click on every single menu option that I see. And I fill in some garbage data in some of the forms that I see.

5:11Robert HurlbutOkay.

5:13Daniel Ramsbrocksubmit them and just kind of see how the application works and what the functionality is. And then once I've kind of cycled through all the accounts that have been provided, to your point, then I start getting into the more active phase of the testing where I start, you know, actually altering parameters and seeing what the application does and trying to access things, you know, objects that belong to other accounts that I should not have access to.

5:40Robert HurlbutYeah, so why are you recording? You mentioned this Burp proxy thing that you're using, and maybe you can tell us just quickly what a proxy— what you mean when you use the word proxy in this description, but also I'm curious as to why are you recording everything that goes through that? What's the reasoning behind that?

6:00Daniel RamsbrockSure. So, you know, proxy, as you might be familiar with, is just really a way to funnel web traffic through a particular endpoint. And it's typically used, you know, by corporate— in a corporate environment to control where employees are browsing and to record what they're doing. In this case, I'm using it locally, you know, on the machine to record all the traffic. And the reason why I do that is because there's a lot of things that go on in the background when you're loading a web page. You know, a single, you know, home page that you might load will result typically in at least a dozen or so requests in the background, and much more than that for complex applications, there's API calls that are being made, you know, images and different JavaScript files that are being loaded.

6:50Robert HurlbutYou're making me nostalgic for the '90s when I started with computers and web servers used to just return one file.

6:57Daniel RamsbrockYes, yes, those were the simple days. Yep, and so it's very important, you know, to look at all these things that are going on in the background, and a lot of times just from glancing through that traffic afterwards, I can identify areas where the application is doing things that it maybe shouldn't be doing. If it's passing things like user IDs or permission strings as part of these requests in the background, then it may be possible to alter those and mess with those from an attacker's perspective.

7:33Robert HurlbutOkay, so we've come through the reconnaissance phase where we've talked about how you're going to record some attacks. What's the thought process that you're going through to actually decide what you're going to even try in this process of trying to attack?

7:50Daniel RamsbrockRight. So a lot of it depends on— so I guess I should back up and add one more angle to this, which is that so far I've just been talking about, you know, what I do during a manual test. Now, in many cases, the test that I'm doing is actually a conjunction of, you know, some automated scanning of the application plus manual testing. And so, uh, in those cases where I have, uh, you know, so, and there's, there's products out there, uh, sort of these big commercial products that, that exist. Um, the 2 big ones that I've, you know, used quite a bit are AppScan Standard, which is an IBM product, and Acunetix Web Scanner. Um, and so, you know, they, they, um, they take credentials. You basically provide a credentials to the application, they scan the whole application, and they throw just every— everything and the kitchen sink at the application and see what sticks. Um, and so a lot of times you can use that, um, as a starting point to, to, to see which areas of the application, uh, you know, look like they might be vulnerable.

8:54Robert HurlbutAnd, you know, is that part of your reconnaissance still, or is that— so it sounds like those tools might actually kind of sit between reconnaissance and actively attack, because it sounds like they're doing some of both.

9:05Daniel RamsbrockCorrect, correct. Yeah, I would usually do the reconnaissance first or let— or kind of let the scan run in parallel as I'm doing the reconnaissance. Exactly, so they kind of sit in between when there is an automated scanning component.

9:19Robert HurlbutNow, isn't that cheating though, to use automated scanning tools, you know, to try and find vulnerabilities?

9:26Daniel RamsbrockWell, absolutely, absolutely not, I would say, because It's, for me, it's a way to, to provide much broader coverage, and this especially comes into play for large applications where, you know, it's impossible to manually cover every, you know, there might be hundreds of forms that, you know, can be submitted and filled out, hundreds, you know, each of them might have dozens of parameters, and so just mathematically covering all those combinations manually would be impossible to do in a 1 or 2 week test window. And so to me, the tools are a way to extend the coverage, to provide this much broader coverage of the application than you could provide manually.

10:14Robert HurlbutYeah, and I'm gonna guess that bad guys use tools as well, so it's not like you're doing something that the bad guys are not gonna do. You're just modeling, you're doing what they're gonna do with Phantom.

10:24Daniel RamsbrockCorrect, correct.

10:27Robert HurlbutOkay, so back to the active attack. So we've got our scanning tool that's going to give us some input, and then where's your brain come into this process?

10:34Daniel RamsbrockRight, exactly. So, um, so from that point on, you know, it's, it's really just, um, some of it I really just call it a hunch. I mean, like I mentioned, you know, I look through the, the traffic that I've recorded during the reconnaissance phase, and, and certain things just kind of catch your attention. Um, you know, when you see things, whenever you see Um, you know, user-provided input being used, uh, to, to make, you know, stateful changes to the application. You know, those are opportunities, um, to inject things. And like, a good example of this is, um, let's say that, you know, this is an invoice, uh, viewing application. Each invoice has a 5-digit ID. Um, and so if I see that, you know, when I pull up an invoice that it's providing that ID number you know, from the user to the server, then I might try providing different 5-digit IDs. And I might try fuzzing, you know, just throwing a bunch of random 5-digit IDs at the system to see if it returns me valid invoices. And so that's usually— it's— that's called a direct object reference issue.

11:39Robert HurlbutAnd so it's, you know, it's a way to see if one user can access other people's data just by knowing that Now, is that something that the tools are going to scan for as well, or is that something that requires a human being that's got more higher-end processing to be able to discover and think through that particular issue?

12:01Daniel RamsbrockYeah, exactly. It does require a human to think through it. So what the tools are going to find, if that particular ID parameter is vulnerable to SQL injection or cross-site scripting, those are things that the tools can easily check for. But the tool doesn't necessarily understand what the application is trying to do. And so the fact that it, you know, it provides a different number and still receives valid data, it doesn't really have a way to know that that data doesn't belong to the user that's currently logged in. So yeah, absolutely, there's a huge, um, a huge part of this that, that requires a human to understand what the application is supposed to do and how it's supposed to work.

12:40Chris RomeoSo those are what are I was gonna say, those are what are called— we usually call false positives, right? And that's, I think, one of the hard parts about trying to interpret, right, when you look at the data?

12:51Daniel RamsbrockYeah, exactly. So, you know, I do spend a fair amount of time, especially on the penetration testing side, there are quite a few false positives. You know, you might see a SQL injection error being flagged when really It's just a verbose error issue where the database is saying, by the way, this column value you provided is too long, but it's not actually injectable.

13:18Robert HurlbutOkay, and it looks like there might be SQL, so the tool sees something that looks like a response from a SQL Server, and it might flag on that particular text it sees in the response.

13:31Daniel RamsbrockExactly, exactly. So by and large, the tools are very complex pattern matching engines, right? And, but it can only match what it knows about, and even then it might be, you know, flagging things that are not true positives.

13:46Robert HurlbutOkay, so we've now cleared the active attack. You know, we found some problems that exist with this particular web application that's being tested. So what's our— what do we do next?

14:00Daniel RamsbrockRight, so assuming that we've confirmed those issues manually, that they are true positives, So, now it's a matter of— and to me, this is a part of the process that sometimes gets glossed over or gets skipped. For me, it's a matter of now, of course, organizing the findings and starting putting together whatever the standard reporting format is that the client uses. But most importantly is to start researching how the developers would actually go about fixing these things. And not necessarily at a high level, but at a very— in the very context-specific environment and language-specific way. So, you know, if I'm working with a Java app, I'm not going to provide, you know, a .NET code sample for how to do— how to prevent SQL injection. And, you know, if this is an IIS server, then I'm going to provide an IIS-specific, you know, configuration change that needs to be made in order to send the correct, you know, HTTP headers for caching or to restrict transport security, for example. Okay.

15:15Robert HurlbutThat's interesting to kind of see how you work through the entire process here, and you're actually providing— it's not— you're not just going out and finding, here's all the problems that exist in your website. You're providing mitigations, actionable items that the customer can take and go fix these problems, because that's what I would see a lot of people, their response after they got the testing report back would be, this is great, awesome job finding all the problems, but we don't have any idea what to do next.

15:48Daniel RamsbrockExactly, exactly. And that's what I always want to prevent. I want to prevent the PDF on desk syndrome, right, of just having this big glob of PDF results that are— it's fun to read through if you're bored, but what, you know, ultimately what are you going to do about it? And so that's— so the other thing that I always like to do is to provide— and sometimes clients have their own, you know, tracking system, which is great, and other times it's just a spreadsheet— but I also like to provide a summary of the findings in a table format so that they have a nice view of sortable, searchable, you know, at-a-glance view of all the things that I've found, and they can use that to track their remediation activities. I find that it helps especially with repeat customers. It helps them, you know, to keep track of these issues, and I see less recurrence from year to year.

16:47Robert HurlbutWhich is— that's your ultimate goal, is to— is hopefully that when you come back and test this thing again, you're not going to find the same— you're not going to have a copy of the— you're not going to generate a new copy of the same exact report with the same exact findings.

17:00Daniel RamsbrockExactly, exactly. And that's rare. Sometimes that happens. There's always a couple of repeat offenders, and, you know, a lot of times they'll let me know right up front, hey, we focused on the 2 or 3 high severity issues and some of the mediums, and we didn't get to the low ones kind of thing. So sometimes I know what to expect in terms of what's been fixed.

17:21Robert HurlbutYeah, and a lot of times those low ones are low priority for a reason because maybe it's an information disclosure, but it's not a— it's telling you what the server version is or something of the web server.

17:34Daniel RamsbrockExactly.

17:35Robert HurlbutBut it's not— so it'd be good to hide it, and why give people more information than you need to? But nobody's going to actively exploit that information banner that's being printed out.

17:45Daniel RamsbrockThat's right.

17:46Robert HurlbutWe're all the way up through the reporting and the client engagement phase. I just got to ask you this. Have you ever had a client just look at you and be like, nah, this is crazy. None of this stuff is real. This isn't going to work.

18:01Daniel RamsbrockNot quite to that level, but I have had people try to kind of argue with And especially in organizations where they get sort of penalized for, you know, they have a certain amount of time to fix severity X and severity Y. And so I certainly have had customers, you know, argue the severity of things and why certain things are not relevant in their environments and kind of try to talk their way out of it. But for the most part, you know, and I've had customers that have been very persistent about like, well, show me, I don't believe it until I see it, you know. But usually either doing a screen share or a person— in-person demo of the issue and explaining to them, you know, what an attacker can do with this is sufficient to help them kind of understand why this is important.

18:52Robert HurlbutYeah, and I want to go back to a point that you made about the— you talked about the PDF on the desk, and so I interpret that to be You did the work, you generated the report, you sent it to them, and your biggest concern is that that PDF is going to sit right on the desk and they're never going to do anything with it. I want to get Robert's take on this from a compliance perspective because that kind of smells to me like a compliance-driven culture is going to be the one where they just— they do the testing because they have to. They take the report, drop it on the desk, never to be seen again because it's under a pile of 12,000 other pieces of paper.

19:32Daniel RamsbrockYeah.

19:32Robert Hurlbutof paper. So Robert, do you agree with that assessment, or what do you think about this, about the world of compliance as it applies to AppSec?

19:40Chris RomeoWell, certainly compliance can be a lot of checkboxes. I mean, that's what we've seen where, okay, did we do this? Yes, we do that. Are we following this particular guideline? Yes, but not thinking about what does that really mean and how does that work as the base the foundation, not the end result. So let's apply that to the pen test. Oh yeah, we got a pen test, it's done, we got a report, move on. I like especially, Daniel, when you were talking about how can you take the report and turn into something that's an action item. And, you know, again, related to the compliance, it can't be something that is just a checkbox and say we did it, but it needs to be something that we take, we understand, we digest, we turn into actionable items, and then track it and see where we are. So I really like that in terms of kind of following through and not just stopping where maybe compliance ends.

20:44Daniel RamsbrockYeah, that's an excellent point. You know, in the more mature organizations that I work with, they actually have this feedback loop where you know, each pen test finding is assigned to whatever issue tracking system, bug tracking system is being used by the development teams, and each finding generates a ticket, and that ticket has to be actioned in some way, and it's trackable to see how long it took to fix it and so forth. But, you know, unfortunately, especially when external vendors are involved and when the process is not as mature, that's rather rare to see it tracked through completion like that and have some organizational out behind it to force the development teams to do the right thing and to make sure that these issues are seen through.

21:33Robert HurlbutThat's a great— between the both of you, that's a great summary there of the compliance culture and the challenges that compliance actually faces for true security, the type of change that we're trying to implement and make happen in many organizations. I want to kind of transfer now. So we went through the process, and Daniel, you did a great job of just laying out what you're doing from an actual set of steps. What I want to know now is, if I want to get into this web app pen testing game, what are, what are some recommendations that you would have for me as far as, are there, you know, particular resources that you'd recommend?

22:19Chris RomeoYeah.

22:20Robert Hurlbutor courses, or, you know, what do I do to actually improve my knowledge and actually become a web pentester?

22:28Daniel RamsbrockSure, sure. So, um, there's, um, there's a whole slew of, um, sort of purposely vulnerable, um, applications out there, um, and, and they come, you know, they come in different flavors. There's, there's mobile ones, there's obviously a lot of web app ones. And so, my favorite for people to get started is from OWASP, which is the Open Web Application Security Project. They have an application called WebGoat, and it's, you know, it's an application that you can, you know, basically download and install in a VM, and you can just start, you know, and so you're not hitting anybody else's environment, you're just, you know, attacking it in your own your own localhost. And you— there's all kinds of purpose, you know, purposely there are all kinds of vulnerabilities in there. There's sort of instructions and labs to go along with it to kind of guide you. This is all, you know, available for free, open source from the OWASP website. And so then once you've kind of gotten, gotten the basics down, there's a couple of other things you can do. And so you, you mentioned courses. There's, in terms of courses or certifications, GIAC, which is the certification branch of SANS, I think they have some really good courses. There's, there's one that's, you know, focused on web app pen testing in particular. It's the GWAPT exam. And so, you know, the course plus studying for the exam, I think, provides a really, you know, well-rounded view of this if this is something that you want to specialize in.

24:10Robert HurlbutThat's going to be a hands-on, from what I understand. That's not going to be a theory in the classroom. That's going to be a hands-on, you know, getting your hands dirty with, with tools and attack methodologies and things that you try out in the lab.

24:22Daniel RamsbrockExactly. Yeah, all of the SANS courses, you know, are— follow that model. And this, this one definitely does as well, um, where it's, it's a hands-on experience. Um, and, um, and so then once you— or, you know, if— and that may be, um, That may— so SANS courses are often a rather expensive proposition, and depending on if your employer is sponsoring you in this or not, or if it's— if you're just doing this, you know, sort of in your spare time.

24:47Robert HurlbutYeah, let's pretend, let's pretend people are on a budget here.

24:50Daniel RamsbrockSo, so another really good option, um, are, are these bug bounty programs. So almost, so almost every major, you know, IT company out there has some sort of a bug bounty program, which means that they will pay independent researchers for any, you know, a bounty basically for any vulnerabilities or bugs that they report responsibly. And so Google, Facebook, Microsoft, they all have their own bug bounty programs. Now it can be sometimes a little intimidating to try to navigate all these different programs' rules. There's a lot of differences there. So there's some umbrella organizations that have sprung up to kind of help companies run bug bounty programs more consistently. And one of my favorite ones for people that are just getting into this is Bugcrowd. And so this is a kind of crowdsourced bug hunting platform, and they have a lot of, you know, projects that are just open to the public. So you don't need to be like vetted or a member of anything special. You just sign up for an account and you can start, you know, testing all these different companies that have put their applications on Bugcrowd. And so for example—

25:59Robert HurlbutSo the bug bounty process then is, this is an opportunity for people to actually attempt penetrations in a controlled environment of things that aren't necessarily the, the web goats of the world that are already vulnerable.

26:15Daniel RamsbrockExactly. So this, so this is a, this is a great way to play with real code, you know, in a live environment, but to do it in a safe way so that you're not, you know, you're not stepping on anybody's toes. And Bugcrowd in particular, you know, they have some pretty high-profile clients. For example, Tesla, Pinterest, and Western Union are some of their banner clients, but they have a lot of smaller ones. And so you can basically just, you know, start touching these applications without— now it's going to be a lot harder to find things, of course, than it would be in a WebGoat environment, but—

26:57Robert HurlbutYou hope so. You hope it's hard. Yes.

27:00Daniel RamsbrockAnd sometimes you'd be surprised. And you might even make some money doing it if you find something that gets accepted by the client that's not a duplicate and something that's valuable, you would get a payout for that.

27:11Robert HurlbutYeah, I had a colleague that I worked with back at Cisco and he's actually won the bug bounty from United, the airline, 2 different occasions. So what United does is they pay in miles. So for each of the— and he found some pretty gnarly SQL injections in united.com. So it was pretty high profile, but they gave him a million miles each for each of the, of the 2 that he found.

27:38Daniel RamsbrockThat's nice.

27:38Robert HurlbutAnd it was— there's a news story. It was a real public kind of thing. But that just gives you a little bit more perspective on if you play in these bug bounties, you know, some companies pay you money for bugs that you find. Others like United will reward you with what they have, which is access to miles, which gets you free airline tickets.

27:56Daniel RamsbrockExactly. And yeah, I should probably take another look at united.com. I actually submitted a couple of things to their bug bounty as well, but I did not get that kind of a payout from it. Oh, okay.

28:06Robert HurlbutWell, yeah, definitely take a closer look because you need a free vacation.

28:09Daniel RamsbrockAbsolutely.

28:10Robert HurlbutWhat about books? Is there— I'm a big fan of— I'm kind of old school. I'm a big fan of books. I mean, they're usually electronic these days, but I still like the concept of books. Is there anything that, you know, let me pose it to you this way. If somebody that was new to, relatively new to security, but coming from a development or test background said, Daniel, what book should I get to just start this process for me? Is there a certain book or maybe one or two that you would recommend that they quickly purchase?

28:42Daniel RamsbrockYes, there are. So the two that are my favorite that are on my bookshelf are Hacking Exposed web applications. So, you know, the Hacking Exposed series covers all kinds of topics, and there's a— I don't even know what they're up to now. I have Web Applications 3, but I'm sure there's 4 in PyDoc by now. But, you know, they're kind of always updating that content. So the Hacking Exposed series does a great job going through the tools that you would need that you'd be using day to day, and it covers, you know, both the defense and offense side of the house fairly well. And the other one that I like is the Web Application Hacker's Handbook, which again is a very hands-on, you know, kind of tool-driven approach to it. It kind of gives you a lot of how-to information that would be great for somebody who's just getting started.

29:40Robert HurlbutThat's great advice. And those 2, I think, are both going to be a lot of hands-on and probably some theory and a little bit of what are the things that we're actually talking about, but going to be more focused on actually getting your hands dirty and getting to work. Have you ever heard of another book called The Tangled Web? Have you ever seen that book?

30:02Daniel RamsbrockYeah, yeah.

30:03Robert HurlbutOkay. So I like that book because it has some pretty good descriptions of some of the problems that you find. It wouldn't be the first or second book I would tell somebody to go with, but if you've been around in the world of development for a while, it gives you a good perspective on the ideas behind what are the security challenges that actually exist, and it gives you some more theory that kind of helps to drive future conversations.

30:31Daniel RamsbrockThat's right. I guess I should add the mandatory Gary McGraw plug, you know, he's the CTO of Sigittal. His book Software Security actually does a really good job of covering some of those fundamentals as well, just the concepts of application security.

30:48Chris RomeoGreat.

30:52Robert HurlbutWell, Daniel, thank you for taking us on this journey through the world of web application penetration testing. It certainly has been very enlightening for me to understand the process that you go through and And I always love to understand people's— what's the thought process? You know, how are you picking this thing apart from, you know, your brain perspective? So that was challenging to me to, you know, go and learn some more about web app pentesting.

31:19Daniel RamsbrockThanks for listening to the Application Security Podcast. Our intro music is 8-Bit Kung Fu by Boring and TJ, and the outro is Southern Delight by Stefan You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.