---
title: "Daniel Ramsbrock -- Web Application Pen Testing – Part 1"
url: https://appsecpodcast.com/daniel-ramsbrock-web-application-pen-testing-part-1/
date: 2016-10-18
duration_seconds: 1880
guests: ["Daniel Ramsbrock"]
topics: ["Security Testing"]
audio: https://www.buzzsprout.com/1730684/episodes/8122734-daniel-ramsbrock-web-application-pen-testing-part-1.mp3
transcript: true
---

# Daniel Ramsbrock -- Web Application Pen Testing – Part 1

*October 18, 2016 · 31 min*

with [Daniel Ramsbrock](https://appsecpodcast.com/guests/daniel-ramsbrock/)

on [Security Testing](https://appsecpodcast.com/topics/security-testing/)

[Audio](https://www.buzzsprout.com/1730684/episodes/8122734-daniel-ramsbrock-web-application-pen-testing-part-1.mp3)

## Show notes

What should developers and security teams understand before commissioning a web application penetration test? In part one, Daniel Ramsbrock joins Chris and Robert to establish the purpose, timing, and business value of testing an application from an attacker’s perspective. They compare penetration testing with earlier secure-development activities, explain why developers and testers benefit from working together, and consider how waterfall, agile, and DevOps delivery models change the engagement. Daniel discusses risk-based scoping, testing frequency, business goals, internal versus external testers, and the ethical “hat” terminology used in security. The episode ends by moving from program decisions toward the hands-on testing process continued in part two.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
Security Journey provides application security education for developers and everyone in the software development lifecycle.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Daniel Ramsbrock:  
→ [Daniel Ramsbrock on LinkedIn](https://www.linkedin.com/in/ramsbrock)

Mentioned in this episode:  
→ [RVAsec](https://rvasec.com/)  
→ [DEF CON](https://defcon.org/)  
→ [Black Hat](https://www.blackhat.com/)  
→ [Web Application Penetration Testing — Part 2](https://appsecpodcast.com/daniel-ramsbrock-web-application-pen-testing-part-2/)

Chapters:  
00:00 Web application penetration testing, part one  
01:41 Daniel Ramsbrock’s security origin story  
03:23 What penetration testing is  
04:23 Why attackers target applications  
05:29 The developer perspective on testing  
06:34 Developers and penetration testers working together  
07:45 Where penetration testing fits the lifecycle  
09:19 Why late testing creates problems  
13:18 Waterfall, agile, and DevOps considerations  
16:39 How often to run a full penetration test  
18:41 Using risk to set the testing schedule  
20:13 Scoping services and applications  
21:14 Connecting business goals to testing  
22:12 Building a testing capability  
24:51 Internal and external testing tradeoffs  
26:24 White, gray, and black hat terminology  
30:24 Preparing for the hands-on process

## Transcript

*4,824 words · assemblyai*

**0:05 Chris Romeo:** The Application Security Podcast. Here we go. Welcome folks, Chris here. This is a 2-part episode of the Application Security Podcast where Robert and I speak with Daniel Ramsbrock about web application penetration testing. In part 1 of the interview, we focus in on the difference between pen testing and web app pen testing. We discuss where pen testing fits into the development methodology— waterfall, agile, DevOps— and finally, Why should someone care about penetration testing? I connected with Daniel through the RVASEC security conference in Richmond, Virginia. Daniel's been in the field of security for over 10 years, with most of his time focused on application security. He spent 2 years as a full-time consultant at Sigil and is now doing independent application security consulting through his own company, Enigma Technologies. Our topic today that we're gonna, that we're gonna cover is penetration testing, and more specifically, web application penetration testing. But before we can dive into that, we really have to hear Daniel's superhero origin story, because that's a big part of what we do here. We want to know how do people get into security and what was their unique path. So Daniel, what, how did you get into security?

**1:40 Daniel Ramsbrock:** Sure, so you know, as many people, I started out as a computer science major in undergrad and kind of trying to figure out what was interesting to me. I took a— I had a great professor, I think it was my sophomore year, who taught a security class, and that really made me realize that this is, you know, what I'm passionate about, what I want to go into. From there, I decided to pursue my master's degree in specifically in IT security. They called it— at George Mason, they called it the Information Security and Assurance.

**2:12 Robert Hurlbut:** Okay.

**2:14 Daniel Ramsbrock:** And another great professor there kind of led me to application security. He was the head of Booz Allen's software security practice at the time. He was an adjunct faculty, so he had some really interesting and solid real-world perspective on application security, and it got me interested in that. I was also doing some development part-time at the, you know, while I was an undergrad in grad school doing some, you know, Java and PHP development. And so I had that coding background and I had the security interest, and I felt that, you know, application security was the perfect way to, uh, to combine those 2. Um, and so then after I graduated, I went into consulting. Um, first I started at a more general, you know, uh, general service consulting firm that had some security projects Then after about 3 years there, I moved over to Sigittal, which is an application security-focused consulting company where I really got to do a deep dive on application security and all of the various activities that are involved in that.

**3:22 Chris Romeo:** Great. We're glad to be able to pick your brain a little bit today about this idea of penetration testing and more specifically web app penetration testing. For our listeners, let's lay a foundation here about what penetration testing actually is. Let's start with the high-level penetration testing. Robert, I'm going to come to you. I'll let Daniel answer first, but I'm going to come to you from a developer's perspective. Daniel can perhaps give us the tester perspective. Robert, you can give us the developer perspective. Daniel, what do you think?

**3:53 Daniel Ramsbrock:** Penetration testing in general is really just, we've got this thing and we want to see how it holds up when you have a skilled attacker poking at it. And that can be an application, a physical device, a part of a module, a part of an application, really anything. It's really just simulating, you know, a real, as real as possible of an environment where a skilled attacker might be going after that.

**4:23 Chris Romeo:** So why would somebody want to hack into something in general?

**4:27 Daniel Ramsbrock:** Sure, so I mean the basic idea is to To find the issues before the bad guys do, right? If a pen tester comes in and finds an issue, reports it to the customer, then they have an opportunity to fix that before the real bad guys come in and exploit it and make off with money or worse.

**4:48 Chris Romeo:** Okay, so you're saying it's going to be primarily a financial risk then as to what these bad guys are going to do if they find a vulnerability or problem that exists on a website?

**5:00 Daniel Ramsbrock:** Sure. I mean, ultimately, financial, that tends to be what gets the attention of organizations, but there are other intangible downsides, reputational loss, which usually translates into some financial downside as well, not having the trust of your customers anymore, those types of things. But yeah, most organized criminal organizations cybercrime rings today are after money ultimately.

**5:29 Chris Romeo:** Okay, that's interesting to understand. So Robert, from a developer's perspective then, what does the developer think or understand about penetration testing?

**5:41 Robert Hurlbut:** Well, typically it's happening after the development has already been done. So from a developer perspective, typically a penetration tester may not know everything about the application. They may not have developed the application, for example. So as Daniel was mentioning, they may be approaching it from an attacker point of view where this is the first time they're looking at it. They're trying to discover what the application does, how it works. And different on the developer side where they developed it, they know all these things, but they may not know all the security issues. They may not know all the potential problems. And so, you know, a different perspective. They're looking at it, well, I already know what it does. I built it. You're not going to find anything wrong. But a penetration tester is going to look at it a very different way.

**6:33 Chris Romeo:** So is there value in the penetration tester and the developer working together on something like this?

**6:39 Robert Hurlbut:** I think so.

**6:42 Daniel Ramsbrock:** Oh yeah, I think absolutely.

**6:46 Chris Romeo:** Yeah, so what would the developer and the penetration tester, if they were coordinating, what would they be better at if they worked together?

**6:53 Daniel Ramsbrock:** So I think that the biggest advantage of working together, and I always prefer this when I have the chance to do it, as Robert was saying, there's really, as a pen tester, as an outsider, I really don't have a good understanding of the thing that I'm testing, the application or the component. And so just getting an overview of what is this thing supposed do? How is it supposed to work? What is normal behavior? And one of the biggest things that I think the tester can often kind of open the developer's eyes to is unintended consequences, right? How you can take certain pieces of functionality that may not have even been designed to be used in conjunction, to combine those and to execute attacks from that that the developer may not have anticipated.

**7:44 Chris Romeo:** So, developers and testers, sounds like it's a good combination for them to be working together. So, let's step back and answer the kind of the macro question here that a lot of our listeners that are new to security might be thinking about, and that is, why do I even care about penetration testing at all? Why do I even have to do it? Why can't I just skip this and move on to writing more code?

**8:10 Daniel Ramsbrock:** Absolutely. So the, um, so there's an idealistic answer and there's the practical answer to it. So in the perfect world, penetration testing is just a sanity check. It's just, you know, at the very end you've done everything correctly, you have a secure, you know, you have security that's built into your development lifecycle, and at the end you're just kind of making sure you didn't miss anything. That's the idealistic view. In the real world, not, you know, a lot of software makes it to a deliverable finalized stage without necessarily having had security baked into it. And so penetration testing often becomes, unfortunately, a method of discovering deeper issues within the application very, very late in the game, at a point where the developers basically are close to being finished or thought they were already finished with the application, or even worse, the application has been in production for years or in some cases decades, and you're finding these issues after the fact and kind of hoping that the bad guys haven't also found those issues in the meantime.

**9:19 Chris Romeo:** That seems kind of backwards to me, right? So you're kind of working your way backwards from— you're saying that A lot of times organizations don't do the testing until too late, and they should have invested some of those resources earlier in the lifecycle to perhaps look at secure design or even maybe back as far as requirements. Is that consistent with your view?

**9:45 Daniel Ramsbrock:** That's right. I mean, the nature of pen testing requires you to have something resembling a final product. You can't really do a penetration test against individual classes or components that are not— that you can't access in the runtime environment. But to your point, there are other application security activities that they should be considering much earlier in the lifecycle, such as what you mentioned, secure design, secure architecture reviews, code reviews, static code analysis scanning tools that can detect things much, much earlier in the lifecycle before you get to integration testing.

**10:24 Robert Hurlbut:** Yeah.

**10:27 Daniel Ramsbrock:** type environment where you can then perform pen testing as well.

**10:30 Chris Romeo:** Okay, let's now cover the idea of web application pen testing versus— we've been using a generic definition to say penetration testing. Give us a little bit of a perspective about what's the difference between penetration testing and web application penetration testing.

**10:48 Daniel Ramsbrock:** Sure, so the, the biggest difference is that When most people talk about pen testing in general, they're typically thinking about things that are focused at the network level. They're talking about, you know, scanning for, you know, open ports, scanning for certain vulnerabilities that may be present in protocols. And so that's more of a kind of a point-and-click approach where you have, you know, tools that can attack these different types of endpoints. and, and see, you know, see what vulnerabilities may be present. And then of course there's very skilled, you know, penetration testers who can take that many levels deeper once they, once they've kind of discovered where those vulnerabilities are. By contrast, web application penetration testing is very, very context-specific, and I kind of alluded to this earlier. The tester really has to have an understanding of how the application is supposed to function in a perfect environment, in a normal usage scenario. And only then— because for me, the whole goal of an application penetration test is to get the application to do something interesting, whatever that means. And a lot of times, that's driven by conversations with the developers, the architects, the application team. to try to figure out, you know, if X happened, that would be really bad. And so then I always try to make X happen to see if there's, you know, a way to bypass the controls that are in place to make the application behave in unexpected ways. And of course, that requires a lot of context and knowledge about what the application is, and more importantly, is not supposed to do under normal circumstances.

**12:38 Chris Romeo:** Yeah, I think about when I you know, lead people in exercises and threat modeling, I use kind of the same basic approach, and I just, I refer to it as the worst-case scenario. So let's—

**12:50 Daniel Ramsbrock:** Exactly.

**12:51 Chris Romeo:** Pinpoint what is the worst case, what's the worst thing that could possibly happen here, and then let's work our way backwards from that. Let's figure out what we have to do to change the design to eliminate or mitigate the worst-case scenario. And it sounds like you're doing the same type of approach, you're just kind of calling it something different.

**13:07 Daniel Ramsbrock:** Exactly. I typically ask the stakeholders, what keeps you up at night? What would happen when you would get a 2 AM phone call related to this app?

**13:18 Chris Romeo:** Yeah, that's another good way to put it. So from a methodology perspective now, I want to think about this because we have a lot of people that are using different development methodologies. And so some people are using, still using the traditional waterfall. Others are using an agile process that's going to be more focused on user stories and sprints and iterations and things. And then other folks are in this continuous integration, continuous deployment world of, you know, DevSecOps and everything is, you know, being, is running at the speed of light. So does, is pen testing, is it done differently?

**13:59 Daniel Ramsbrock:** Yeah.

**14:00 Chris Romeo:** Is it dependent on your development methodology at all? Is there any connection there?

**14:05 Daniel Ramsbrock:** Absolutely. I mean, you know, to me, the waterfall methodology lends itself most to the kind of traditional approach to pen testing, which, you know, most organizations will test their key apps, you know, once a year or so. Sometimes there's regulatory requirements that mandate certain intervals. Sometimes they have to, you know, run a scanning tool at certain intervals, and then they might perform a more in-depth manual test, you know, once a year. But that doesn't really scale when you're doing it, when you have an agile development methodology or continuous integration. So, to me, you know, if you're using an external vendor, they would have to— I mean, ideally, there would be an in-house pentesting team that can help out with some of this. When you're talking about agile and continuous integration, there has to be very close cooperation between the testers and the developers because, you know, in order to— so let's say you have 2-week sprints. Well, you're not going to do a 1-week or a 2-week in-depth, you know, pen test every 2 weeks. It doesn't make sense. And so the focus is on the changes, on the deltas, right? So the tester has to communicate with the development team and understand, okay, what are the changes that you've made to the codebase since the last release, since the last sprint? And not just at, you know, I mean, obviously, you can look at the actual diff, but to have some narrative around it, some context, and to understand where that testing should be focused so that the testing becomes incremental. And even more so in a continuous integration environment. And, you know, the other thing that typically occurs with sort of normal annual pen tests is that, you know, you'll do a test and then the development team will go ahead and fix the issues and then they'll request a retest. And that you're just basically going back through the findings from the previous report and, you know, either saying, yes, it's been fixed, or no, there's still an issue. And so that happens on a more continuous basis when you're doing agile continuous integration where you're basically always getting retests. Every sprint you're retesting the stuff from the previous time if the development team tells you that, yes, we have fixed this particular issue. Otherwise, there's no need to burn testing cycles on it.

**16:38 Chris Romeo:** Okay. How often then should somebody do a full penetration test regardless of waterfall, agile, or DevOps type of a deployment? What's your recommendation guidance for folks?

**16:54 Daniel Ramsbrock:** It depends a little bit on the criticality of the application. In most cases, I would say once per year is adequate. If there have been major, major changes to the application, especially relevant to security, for example, if, you know, a new authentication mechanism has been added or changed, then in those cases it should be more of an event-driven thing where you would want to, you know, perform that testing before you release that that version of the application. But in most cases, once a year is a good kind of middle-of-the-road approach. And if there's very, very critical applications, and I'm talking about things that maybe in the financial sector that are a year, but for most people, once a year is a good interval.

**17:47 Chris Romeo:** Okay, once a year. That makes sense to me. And I could see how some organizations like a DevOps organization might be on a quicker schedule just because of the nature of what they're, you know, how fast they're making changes. Now, I do want to come back and ask you a question because you mentioned criticality of the application is something that drove it. And you gave us an example of a financial application. And I can understand how a financial application where somebody can access or transfer or, you know, cause a lot of money to move from, you know, from your account to my account is something that I would see as being very critical. Now, how do you determine the criticality of an application? For our listeners out there, if they have 10 applications, what's the thought process that you go through there to say this is high, this is medium, this is low?

**18:41 Daniel Ramsbrock:** Sure. So, you know, that's something that's usually driven by by 2 things: the risk of something bad happening, in other words, the likelihood of it happening, kind of multiplied by the impact that that event would have, right? And so you look at, well, this particular application, you know, and it's— if possible, it's nice to assign dollar values to it, but You know, it's not always— the quantities are not always knowable in these cases. But so you basically, you have— you look at your portfolio, like you said, 10 applications, and you determine, you know, which ones of these is kind of the biggest target for somebody who's after either things that we had mentioned before, either, you know, financial gain or reputational damage to the company, you know, things of that nature.

**19:42 Robert Hurlbut:** Yeah.

**19:42 Daniel Ramsbrock:** And then you look at what controls are in place. So for example, if this is an internal application that's not accessible from the outside, that dramatically lowers the likelihood of somebody compromising that application. If it's a public-facing application where lots of users have logins and there's administrative access from the outside, that raises the likelihood of somebody attacking that application.

**20:11 Robert Hurlbut:** Okay.

**20:12 Daniel Ramsbrock:** Okay.

**20:13 Chris Romeo:** Robert, are you doing this same type of an analysis from a developer perspective when you're looking at the different services or different, say, web services or applications or whatever that exist that you're responsible for? Do you do that same type of a criticality equation?

**20:28 Robert Hurlbut:** Generally, you know, trying to understand, at least from a security perspective, which ones are Let's say, for example, if I'm doing a design review or trying to think about the security, I would think about, you know, where's my attack surface, where are the areas that may be most important from an attacker point of view. But at the same time, I'm also trying to think about what are the business goals, what are the, you know, project manager, where are they focused on, and then that can also help drive may be different than what the security concerns may be. So, you know, it varies, it varies, but a lot of those align pretty well, I think.

**21:14 Chris Romeo:** Daniel, as a follow-on question to Robert's explanation, how do business goals play into the world of penetration testing?

**21:23 Daniel Ramsbrock:** So, I mean, they really drive, you know, they really drive everything because at the end of the day, you know, penetration test is about evaluating the application in its sort of natural habitat, right? And so, um, the— if, if the business doesn't view this application as, as valuable or as risk— as having— carrying a certain amount of risk, um, then, then there's less of a need to, to scrutinize it and perform pen testing. And so Yeah, I mean, everything that we do is driven by what does this application do for the business, how important is it, and how much financial damage can be done to the business as a result of a potential breach.

**22:11 Chris Romeo:** So then how— if we go back to how our listeners might be struggling with how they're going to take this idea and actually make it and do something with it inside of their organization. From your perspective and your experience, is it better to go the consulting model and just have outside people do all of your penetration testing for you, or is it better to try to build some of that capability yourself inside the company and then maybe have like a hybrid model? What's your take on that?

**22:47 Daniel Ramsbrock:** Yeah, I definitely think that there's pros and cons to both approaches, and I think that ultimately most organizations end up with some kind of hybrid model because it makes the most sense. Certainly when you're first getting started, and let's say that you've never done any pen testing on your applications or maybe just very sporadic, you're going to need some outside help just in terms of having the expertise and the staff that's already familiar with the tools and the process. but also just to handle the workload. There's going to be a lot of testing up front before you even have an idea of what you're up against. So, you know, when you're first getting started, you'd be relying much more heavily on consultants, even if you already have maybe a few, 1 or 2 or 3, you know, in-house pen test resources. Now, over time, as the organization matures and especially as security gets built, you know, earlier into the lifecycle, And there'll be less and less, you know, findings that are going to come out of each penetration test because the code is going to just naturally improve as the existing findings are fixed and the process itself, you know, is improved. And so over time you would end up with a combination because, you know, as we were mentioning before, it is very important to have in-house resources who really understand not just this one particular application that they're testing, but also the broader business context of your company and how this app is interacting with the other components, the other applications. And so having that sort of knowledge transfer and organizational knowledge about the technology stack is very important, and that's something that you can't really achieve if you're relying only on external vendors to do your testing. You're going to get a much more in-depth perspective on the testing when you have internal resources that truly understand the applications and the stack.

**24:50 Chris Romeo:** Yeah, that's my take on it as well, is that we have— there are advantages and disadvantages to both, but it's good to have some institutional knowledge in how to do pen testing so that it can be applied more often than a yearly cycle. But you also need the external thought leaders in the world of penetration testing to come in and, you know, maybe try some things that the internal folks just haven't caught up on yet. You know, new things in industry, something that was released at DEF CON or Black Hat or, you know, one of the other security conferences. There's a need for that external knowledge to be brought in-house. And so I think it's a connection between both. is the overall best answer from my perspective.

**25:37 Daniel Ramsbrock:** Exactly, and I agree with that. And I think the other thing that's important, and I've seen this with some of the larger clients that I work with, is having an internal staff that can sort of supervise the external vendors, having an internal staff that has the hands-on technical experience and can perform QA, not necessarily from a perspective of, you know, that the vendor is not doing their job, but that they're doing it in a consistent manner, especially if the organization is using multiple different vendors, is to basically, you know, enforce a consistent process and consistent deliverables, you know, across the vendor base that are performing these tests on behalf of the client.

**26:23 Chris Romeo:** Okay, let's dive now into Get a little closer to the actual testing process and kind of where the magic happens, I guess. One of the things, Daniel, that I see all the time that I think it'd be good to explain to our listeners right from the beginning is it seems that everyone in the world of penetration testing or security attacks and threats and things, they're all wearing a hat. Okay? And one of these hats is a white hat, one's a gray hat, and one's a black hat.

**26:54 Robert Hurlbut:** Right.

**26:55 Chris Romeo:** And so I'm just curious, can you give us a little bit of a perspective on what are these hats everybody's wearing and what does this actually mean?

**27:04 Daniel Ramsbrock:** Absolutely. So white hats are— it really refers to the intention and the integrity with which a tester conducts themselves. So a white hat would be somebody who does everything above board. They don't attack any systems, whether don't have explicit permission, and they always report their findings, you know, to the vendor with plenty of lead time to have to perform a fix before they would publish their results to the wider community. Black hats are at the opposite end of the spectrum. They sort of, they have sort of no allegiance to anybody else other than usually themselves and maybe a team that they're working with. They are, you know, in most cases are cybercriminals. They're in it, you know, to make money for themselves or for a criminal organization that they're a part of. They'll use anything that they can get their hands on for financial gain, and they certainly won't disclose vulnerabilities to the vendor and give them an opportunity to fix it. Gray hats are somewhere in between. So it's usually white hats that may not always do everything by the book. So they may attack systems without permission, but then still, you know, do responsible disclosure with the vendor to make sure that they have time to address it. Or they may attack the system and then, you know, with permission, but then disclose without giving the vendor a chance to fix it because they maybe they feel that the vendor is not acting quickly enough. Okay.

**28:48 Chris Romeo:** So Robert, does a— do the developers of the world have white hat, gray hat, and black hat associations as well?

**28:56 Robert Hurlbut:** Not really, not exactly. I mean, unless you're thinking about on the security side, you know, again, trying to think about how would I develop some tools as a black hat to attack? How would I develop tools to defend? How would I do a little bit of both? perhaps in that area, but if you're just talking about developing applications, I don't really see that.

**29:22 Chris Romeo:** So there's a— it sounds like you're talking about there's a need for developers to have an awareness of what white hats, gray hats, black hats are doing and some perspective on what those folks might be doing to the product that you work on when it gets shipped and deployed out onto the website.

**29:41 Robert Hurlbut:** Oh, yeah, definitely. I think that one of the things that I see a lack of on the developer side is that idea that, hey, why would anybody be interested in this application or circumventing controls or, you know, why am I doing all this stuff? Why am I going through all these hoops to try to make this secure? Who cares, you know, really? Not everybody. I mean, there's a lot of developers who would like to know more, but I've also seen developers who say, Why am I going through all of this to try to make this more difficult to develop, to test, to all these things? Because who really cares? The reality is we need to care and developers need to care.

**30:23 Chris Romeo:** That's a great segue for Daniel now to introduce us to the process of what he does because right now we've got developers that have it top of mind thinking, I don't even understand why I care. why this could be important for my application. Daniel, walk us through the process that you go through to actually run— go through a test. And that, folks, is an Application Security Cliffhanger. Tune in to the next episode to hear about the web application penetration testing process and how Daniel brings it all together. Thanks for listening to the Application Security Security Podcast.

**31:02 Daniel Ramsbrock:** Our intro music is 8-Bit Kung Fu by Boring and TJ, and the outro is Southern Delight by Stefan Kartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.

---

Source: https://appsecpodcast.com/daniel-ramsbrock-web-application-pen-testing-part-1/