---
title: "Tiago Mendo -- How to scan at scale with OWASP ZAP"
url: https://appsecpodcast.com/tiago-mendo-how-to-scan-at-scale-with-owasp-zap/
date: 2022-12-06
duration_seconds: 1934
guests: ["Tiago Mendo"]
topics: ["OWASP Projects", "Security Testing", "Vulnerabilities and Exploits", "DevSecOps and CI/CD"]
audio: https://www.buzzsprout.com/1730684/episodes/11790862-tiago-mendo-how-to-scan-at-scale-with-owasp-zap.mp3
video: https://www.youtube.com/watch?v=FYjhzExhbj8
transcript: true
---

# Tiago Mendo -- How to scan at scale with OWASP ZAP

*December 6, 2022 · 32 min*

with [Tiago Mendo](https://appsecpodcast.com/guests/tiago-mendo/)

on [OWASP Projects](https://appsecpodcast.com/topics/owasp-projects/), [Security Testing](https://appsecpodcast.com/topics/security-testing/), [Vulnerabilities and Exploits](https://appsecpodcast.com/topics/vulnerabilities/), [DevSecOps and CI/CD](https://appsecpodcast.com/topics/devsecops/)

[Audio](https://www.buzzsprout.com/1730684/episodes/11790862-tiago-mendo-how-to-scan-at-scale-with-owasp-zap.mp3) · [Video](https://www.youtube.com/watch?v=FYjhzExhbj8)

## Show notes

Tiago Mendo is a co-founder and CTO of Probely. He has extensive experience in pentesting applications, training, and providing all-around security consultancy. Tiago Mendo is a co-founder and CTO of Probly. He has extensive experience in pen testing applications, training, and providing all-around security consultancy. He started working with security in the early 2000s with a tenure of 12 years at Portugal Telecom, where he built the web security team and worked with 150+ developers. He holds a master's in information technology, information security from Carnegie Mellon University, and a CISSP certification. He's also a qualified member of AP2SI, a nonprofit organization that promotes information security in Portugal, and co-leader of the Lisbon OWASP chapter.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
We help enterprises reduce vulnerabilities through application security education for developers and everyone in the SDLC.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Tiago Mendo:  
→ [OWASP ZAP](https://www.zaproxy.org/)  
→ [Probely](https://probely.com/)

Mentioned in this episode:  
→ [OWASP ZAP](https://www.zaproxy.org/)  
→ [Probely](https://probely.com/)  
→ [sqlmap](https://sqlmap.org/)  
→ [Jenkins](https://www.jenkins.io/)  
→ [PCI DSS](https://www.pcisecuritystandards.org/standards/pci-dss/)  
→ [Cobalt.io](https://www.cobalt.io/)  
→ [LASCON](https://lascon.org/)

Chapters:  
00:00 Meet Tiago Mendo: How to scan at scale with OWASP ZAP  
02:23 One thing we like to do in our podcast is start  
07:14 Very cool. And so from understanding and hearing about the talk  
10:23 About in a DevOps context  
12:56 Then, what are some of the challenges, Diego, from trying to  
22:32 That's either hitting test environments or even production environments as well  
28:55 As we think about, you know, for our listeners, what they

## Transcript

*4,575 words · assemblyai*

**0:00 Chris Romeo:** Tiago Mendo is a co-founder and CTO of Probly. He has extensive experience in pen testing applications, training, and providing all-around security consultancy. He started working with security in the early 2000s with a tenure of 12 years at Portugal Telecom, where he built the web security team and worked with 150+ developers. He holds a master's in information technology, information security from Carnegie Mellon University, and a CISSP certification. He's also a qualified member of AP2SI, a nonprofit organization that promotes information security in Portugal, and co-leader of the Lisbon OWASP chapter. He's also a frequent speaker at security events such as Confraria de Segurança de Informação, BSides Lisbon, BSides Krakow, and LASKON. Tiago joins us to discuss DAST scanning at scale. We hope you enjoy this conversation with Tiago Mendo.

**0:57 Robert Hurlbut:** The Application Security Podcast is brought to you by Security Journey. We help enterprises reduce vulnerabilities through application security education for developers and everyone in the SDLC. Learn more at securityjourney.com.

**1:10 Chris Romeo:** Hey folks, welcome to another episode of the Application Security Podcast. This is Robert Hurlbut. and I am a Principal Application Security Architect at Acquia. And I'm joined by my good friend, Chris Romeo. Hey, Chris. Hey, Robert. It is great as always to be on the line talking with you about all things application security, threat modeling, secure development lifecycles, all those different things that we love to talk about, and OWASP as well. OWASP has been a big theme for us recently, and it's gonna be a big theme for the next few episodes as well as we go forward. Definitely. Always love OWASP talk. Speaking of which, our guest today is Tiago Mendo, who spoke recently at the OWASP LastCon conference. And so welcome, Tiago.

**2:04 Tiago Mendo:** Thanks. Thanks to Robert and Chris. It's a pleasure to be here.

**2:08 Chris Romeo:** Yeah, very cool. And our good friend Izar Terandas, was the one who made a mention of your talk. And I was like, okay, well, hey, let's get Diego on the show here to hear about this talk from LASKOM.

**2:20 Tiago Mendo:** Great.

**2:21 Chris Romeo:** Yeah.

**2:22 Tiago Mendo:** I'm honored by this invitation.

**2:23 Chris Romeo:** Well, one thing we like to do in our podcast is start off with is ask, what is your security origin story? So, could you tell us that?

**2:33 Tiago Mendo:** Sure. So, I started working with security immediately after my degree. So I started first at the more at the network level in the largest telco in Portugal, Portugal Telecom at that time. And there I spent a few years, 3 or 4 years, basically looking at the network, reversing protocols. I had Wireshark at that time. I think it was Ethereal open every day. So basically I would spend my whole day reversing protocols. Sometime later, I joined, I was invited by my friend of mine to join the Portuguese OnionNet project. And we set up a few honeypots both outside the company and inside the company. That was cool. It was a very nice project. So, and, but some years later, I got the opportunity to do a master's in information security at Carnegie Mellon in Pittsburgh. So I couldn't miss out on that opportunity. It was a once-in-a-lifetime opportunity. I would keep my job, I would still getting paid and not having to work and be able to do the master's. So it was amazing. So, and there it was when I met Nuno, the current CEO of the company I work with. And basically he challenged me to join his— to join him at Portugal Telecom. Both of us worked at the same company but in different areas. He worked in Sappu, which was something like a Yahoo kind of thing, like a web portal with 100 web applications. And he challenged me to build an application security team from scratch there. So that was my turning point from the more classical network host security guy to application security. And we built the team from scratch. Just for you to have an idea, we started as 2. At the peak, if I recall correctly, we were 15. And in front of us, we had around 300 people, around 150 developers, marketing, sales, so on and so forth. So, and during a few years there, I did pen testing, application consultancy, consultancy, training, handled the PCI DSS audit for a few years. And during that period, it was when we noticed a small gap in the market. Like, we wanted basically to empower our developers so they wouldn't ask us so many times to do pen tests. We couldn't handle it, basically. We could not scale in that sense. So we tried to find out the tool that we could give them, hey, look, use this tool, scan your applications with it before coming to us, and make sure you don't have, for instance, high-severity vulnerabilities. But there weren't. Basically, the only tools that existed required them to be security experts for many reasons, like being able to triage false positives, so on and so forth. So, there was nothing. And then, the stars aligned, and we got challenged to come up with a security product. So basically, a lot of things happened at the same time. And it was one day we quit it and created our own company, Brobee, where I'm CTO right now, which is basically, basically a web application scanning service that scans both web applications and APIs, but with a focus on making it easy for developers either to use but also to integrate. And this is where I've been in the last 5, 6 years. In between, I've worked with Cobalt doing pen tests. So they do pen tests as a service. Recently, I helped with a friend, with a few friends of mine, restarted Lisbon LASP chapter. So I'm also involved in the community in Portugal. The community is kind of close, small, but it's very, really tight together. So there's a lot of events. I try to help people organize the events. I try to be a speaker whenever I can. So I try to be present at the community.

**7:14 Chris Romeo:** Very cool. And so from understanding and hearing about the talk that you did at LASPCon, You're— when you talk about scanning at scale, which is really what is going to be kind of the overarching theme of what we're talking about here, you're talking about OWASP ZAP. And so let's, let's introduce for folks that may not be as familiar with OWASP ZAP as Robert and I are, let's introduce first of all, what is OWASP ZAP? And then we'll get into how does it fit into the secure development lifecycle? How do you use it as a tool? But let's start by talking about just what is OWASP ZAP?

**7:50 Tiago Mendo:** Sure. So OWASP ZAP is one of the flagship projects of OWASP. It's basically a web application scanner. So it's essentially a tool that you run on your desktop and point it to a web application or API, and it finds vulnerabilities, security problems on your web application. It's not only that, it also, it's also a set of tools that can help you test the application manually. But essentially, at its core, it's a web application scanner.

**8:23 Chris Romeo:** And it's open source.

**8:24 Tiago Mendo:** It's developed by Simon Bennett, a longtime member of OWASP. It's supported by OWASP, heavily maintained. There's a lot of community, a huge community around OWASP ZAP. So it's a great project.

**8:40 Chris Romeo:** Yeah. Simon's a previous guest of the show, so we've had him come on to talk about ZAP as well in the past. But it's a tool that has a lot of power. Like you said, it's flagship from OWASP, so it's really heavily resourced. Granted, from a volunteer perspective, but heavily resourced with a lot of people that are thinking about it and working on it. So let's talk now, Diego, about the use cases in the secure development lifecycle. So let's just talk in general. Where am I plugging OWASP ZAP into my process? As an application security person?

**9:14 Tiago Mendo:** Yeah. The classical use I see from ZAP is a security person within the company using ZAP as a pen testing tool. So at a given point, they need to test the application before reaching production, I hope. So the pen tester comes, fires up ZAP, and uses it basically to analyze, to proxy the requests to the application, to log them so that you can look at them later, fire scans at those endpoints. So most of the times I see ZAP being used as a final stage in the development of an application. So when the application is already built, it's ready to be used, or it's already in use, And the security team uses it to test the security of the application. Developers also use ZAP, but I would say the main use off the shelf is a security professional using it to test, manually test the application.

**10:22 Chris Romeo:** What about in a DevOps context? Where does ZAP fit in, in the pipeline?

**10:30 Tiago Mendo:** So I would love to see ZAP and also other tools sooner in the pipeline. So ideally, the development pipeline and the DevOps cycle says that we need to test as soon as possible. So as soon as there's like something to test, we need to test still within the realm of the development team, ideally. So there's less back and forth between the development team and security team. But, um, I don't see ZAP as it is right now as a tool that can be trivially used at that stage by the developers. So right now it's easier to put ZAP at the end of the pipeline when the application is more or less ready, for instance, in the last stage of a Jenkins pipeline and have it run at that moment when the application is, for instance, being deployed to a QA environment.

**11:29 Chris Romeo:** So, okay. So one of the topics we mentioned was scanning at scale and main focus today. So what do you mean by that, scanning at scale?

**11:42 Tiago Mendo:** Yeah. So, The definition we use for scale, this can be huge for some people or tiny for others. For us, in our context and the context we used to find the results and for the challenges we used the intel kits, around hundreds of scans started per day. So on a given day, there's a few hundred scans that start. All of them to different web applications. And that's our context of scale, a few hundred, which is way more than what a human can handle. And it's already enough to cause a lot of trouble because of the scale. Even with a few tens of scans per day, it would already, we would already experience experienced the problem, a lot of problems because of the volume. Because a single scan to any web application can easily result in, I don't know, 100 findings. Not that all of them are valid, but it is what it is.

**12:56 Chris Romeo:** So then, what are some of the challenges, Diego, from trying to do scanning at scale?

**13:08 Tiago Mendo:** Yeah. So there's a few challenges. I'm going to maybe start with the most common one, which is the number of findings, the sheer number of findings. Any user of a security scanner knows that most of their results are a mix from informational findings, low risk, and a lot of false positives. So, and that's okay from a manual testing perspective where the tester, the pentester, wants to have those findings raised, those suspicions of something weird so he can manually analyze. But that only works in a manual scenario when you are testing one application at a time. If you are testing even 10 applications in a single day, if you need to go through all those findings every day, you are dead. You cannot do anything. So you need to figure out ways to filter out those. And there's a few approaches you can use. We can talk about them later. Another challenge that I see is the scan time, the duration.

**14:31 Chris Romeo:** Right.

**14:32 Tiago Mendo:** And this is especially important if you consider scanning in CI/CD pipelines, because it's common for you to scan, for a pentester to do a scan, and the scan takes the whole night or even a day or 2 because the application is large or it's slow. This is particularly common on test environments where the The hardware is smaller. But if you want to move this kind of testing to a more frequent one, so at a bigger scale, or, and to put it in a CI/CD pipeline, you cannot have scans taking ages because you cannot have your developers looking at the walls waiting for the pipeline to end. So that's also a challenge, making sure the scans don't take ages. Or if they take, you need to figure out, you need to think about ways not to interfere with your developers' working process. So that's at least 2 challenges. But there's a few more. You need to make sure how do you manage, specifically with ZAP, how do you manage the scan queue itself? Because I would say ZAP is great for running a scan at a time and manually taking care of it. But if you simply fire it to a lot of web applications, you start to get some— you start to hit some limitations about how the URLs, the scans, the scan queues are managed. So you also need to be smart and figure out, okay, I cannot just put these on the queue and scan. You need maybe to figure out which ones to scan first or which ones not to scan. So there's a lot of challenges around it, but I would say these are 3 of the most important ones.

**16:28 Robert Hurlbut:** Coding more securely from the start saves your organization time and money while protecting yours and your customers' data. Security Journey provides hands-on secure coding training in an application sandbox that allows developers to identify, break, and fix common security vulnerabilities. Give your developers the opportunity to recognize and prevent common and emerging security issues before they become a problem. Visit securityjourney.com to try our training today.

**16:56 Chris Romeo:** You mentioned one of the challenges about noise and what would be a good solution to overcome that challenge. And maybe in particular with ZAP as well in helping you with scale.

**17:13 Tiago Mendo:** Yeah, so I know as a security professional and pen tester that we normally don't want to disable tests because we're always afraid, what if we miss a vulnerability, then it comes back to bite me. So, but we need to. So one of the things we need to do is to start maybe with a high-risk finding. So, uh, assume the developers will not fix the low-risk findings, neither the medium, probably. So let's just start by enabling high-risk vulnerabilities, taking care of SQL injections, cross-site scripting, XML injections. So that immediately solves, improves the scan time a lot and already reduces the number of findings significantly. So you get rid of all those lows and mediums, uninformational. So I don't care if my server has an Apache or a TLS certificate. It's fine as long as it's not expired. I don't want that finding bothering me. So one idea is just to Start small. Let's focus on the high-risk vulnerabilities. And even inside the high risk, you can do baby steps. Okay, let's just start with the SQL injection. Those are very bad. So let's just first make sure the whole process works properly with SQL injection. So that's one way. Another thing you need to do, and this is something that affects ZAP. Off-the-shelf ZAP will report some false positives because they'll report— it will report anything that smells like a SQL injection. So if it smells like a SQL injection, if the application took a little longer to reply, maybe that payload executed and maybe it's a vulnerability there. So, Of course, this is easier said than done, but one idea is to improve the ZAP scan rules or do some post-processing that do further testing. So a common problem is for delay-based test vulnerabilities to generate false positives because the application is becoming slower because of our own tests. So, and ZAP, for instance, does, if I recall correctly, There's only one, one or 2 requests with a payload like sleep 10 seconds. So the application should take more than 10 seconds, 10 seconds to reply. So one idea is to do more of those tests, but mix them with tests without payload. So do requests with a delay. The application should take 10 seconds to reply. Then do a request without a delay. It should be much faster. Then do it again with a device, then do again, and do this 3 or 4 times. It's basically like a statistics approach. So measure more so you can filter out the outliers. So, and we found out that despite we are adding more tests to the scanner, this is actually not increasing the scan time significantly. Because it— these tests are only going to run when there's already a hint there's a SQL injection. So it's not— they are not going to run every time, and they will save you so much time. Definitely. So the time you save later looking at false positives, um, completely outweighs a few seconds that you take extra. And there's other techniques that I also like which are like, for instance, there's really good tools for some specific vulnerabilities. COMIX for command injection, sqlmap for SQL injection. So why not send the vulnerabilities ZAP thinks they are SQL injection to a tool that is really good at exploiting those SQL injections? Let's see if it's able to exploit them. If it is, We are 100% sure. If it is not, maybe we can discard it, because if sqlmap is unable to find it, the likelihood of an attacker also exploiting it decreases a lot. So, yeah.

**21:47 Chris Romeo:** So from a DevOps pipeline perspective, I've, I guess, over the last couple of years of using DAST tools and trying out different DAST tools, I came to the conclusion myself that DAST is not fast enough to run in the pipeline because I want less than a 10-minute cycle time from start to finish of the pipeline. I don't want— I can't extend past that. And I know ZAP has this little short, you know, you can run a little short scan with ZAP that keeps itself to a minute, but you're probably not getting the depth of testing. So what I've kind of mentally kind of decided myself is I'm going to pull DAST out of my pipelines and I'm going to keep it as an external service.

**22:29 Robert Hurlbut:** Okay.

**22:31 Chris Romeo:** that's either hitting test environments or even production environments as well, you know, at an interval. What's been your experience from that perspective? Like, can DAST exist in the pipeline? Even if we think bigger than ZAP, can DAST exist in the pipeline?

**22:46 Tiago Mendo:** The approach you mentioned is actually a great idea and one we see often. So DAST tools, as right now, you cannot just simply put them in the pipeline and make the developers wait for the scan to run. So the approaches we've seen working and that we recommend are basically to detach, like you said, the scanner from the pipeline. So one idea is to— you have the pipeline deploying, I don't know, a few times a day or nightly. Just build a separate pipeline or task to have your scanner run nightly, for instance. It's already 1,000 times better to have a scan every night than just having once a month or in the annual pen test. So that's one idea that works very well, having a separate task to handle the scans at night. Other approaches we are seeing especially in our customer base, is to make sure— is to have the scans in the pipeline but not block the pipeline. So basically, the pipeline will start the scan, the pipeline will pass, or at least it won't fail because of the scanner. So the developers will still have their codes, I don't know, deployed to whatever environment they need, but the pipeline triggers a scan. And what happens is that the scan runs asynchronously, and eventually it will feed the results to them. So they won't get the results immediately, but most of the times you'll get the scan results well within the development timeframe, so well before the application reaches production. So I don't know, let's suppose you are developing an application for a if you have scans triggering, running on your pipeline since the first week, in the next day you probably get results. So that's— and you don't get locked. So you can also configure the tools, and that's something I strongly recommend, to only report either high-risk findings or findings that you really care about. If you are under PCI DSS, for instance, you probably need to be a little bit more picky and make sure you report more things. Other approaches we are seeing, and we try to build support for those at Prowler, is the ability for people to do scans only to a subset of the application. So let's say I'm scanning changing the admin part of the application. I don't want— I don't need to scan the whole application again. I can just scan the admin parts, the admin subset, and that's also an improvement. But there's a few ideas people can play with, like slashing the scan time, for instance, to 10 minutes. Yeah, 1 minute, it's definitely not enough. But maybe 10 minutes is enough. And 10 minutes, it's— people can withstand that in a pipeline with— in a blocking mode. But for instance, only scan some certain parameters like limits, parameters named limits or order or page, those that are— or search, those that are well known for having vulnerabilities. and maybe ignore some other parameters or put them at the end of the queue. And if we have time to scan them, great. If we don't, eventually they will get scanned.

**26:38 Chris Romeo:** So some of this is a configuration of the tool itself as it runs in the pipeline. So a couple of things you mentioned is you could have a time limit that says, like, hey, we're going to let it run for 10 minutes. You could also let the scanner kick off and then run as long as it takes. But it sounds like what you're also describing is setting up different configurations for the scanning tool so that I'm not going to scan— like, if I'm going to scan a part of the application, I think one of the challenges there now is I can't automate scanning of a particular piece of the application that was changed. Like, think about how SAST tools are doing things now. SAST can do a check of, like, the files that changed.

**27:20 Tiago Mendo:** Yeah.

**27:21 Chris Romeo:** And so you don't have to run a SAST scanner against the whole codebase in every pipeline. You just run it against the diff or against the change. Unfortunately, I'm not aware of anything in DAST that does that yet today. Maybe there's a market opportunity there, you know, for our entrepreneurs that are listening here on the line.

**27:38 Tiago Mendo:** Maybe. I'm also not aware of any DAST tool that can do what you described. In theory, in theory, we should be able to do it, not for all the applications, but for some, because if we know the application routes, and those are typically listed in a file, we can match files that changed. We can, we know with some good odds which routes they match. So we could only scan for those particular routes, and hopefully they will...

**28:12 Robert Hurlbut:** Yeah.

**28:13 Tiago Mendo:** trigger the execution of the files that change. But definitely, that's an issue. And even with the possibility of setting up per-developer configurations, like, even if it's simple, the requirement of having that manual step might be enough for the developers not to use it. So, I agree with you that...

**28:38 Robert Hurlbut:** Yeah.

**28:39 Tiago Mendo:** That must— we must look at an automatic way to do it. Like, for instance, the SASTs, well, they will have access to the source, so it's much easier. But I think that's the way to go. Okay.

**28:55 Chris Romeo:** Well, as we think about, you know, for our listeners, what they can do, do you have any good key takeaways or a call to action? for those who are new to ZAP, haven't heard of it, want to take a look at it, or are using it now and how they can improve. But could you give us some key takeaways and call to action, please?

**29:15 Tiago Mendo:** Sure. Well, based on the subject of this podcast, my takeaway is to basically scan as much as possible and as frequently as possible. So I would say most companies only do pen tests once or twice a year, but there's a huge gap between those pen tests, either because new vulnerabilities appear. I don't know, it's almost one year, but Log4Shell was dramatic last year. The developers likely continue to add features to the application. Libraries get outdated every day, and new techniques to exploit Web applications are researched and found every day. So I think one takeaway is to basically figure out a way to automate your scans, even if you do it just once a month. If you fire away a scan and let it run, even if it takes 2 or 3 days, it's great because you reduce your gap of 1 year to a month, and you don't interfere with your, developers, you don't have to think a lot about putting that in the pipeline. Just set it up in a cron job and you're good to go. For ZAP, for people to be familiar with ZAP, I think ZAP, you need to try it. You need to download it and maybe use it to research a bounty. Go to a bounty program and attack a site that has a bounty. And try to use ZAP, and eventually maybe try to improve its rules, its add-ons. So basically, with the knowledge you have that you gain by analyzing the vulnerabilities and filtering out false positives, try to code that knowledge that you are using, and that eventually will make the rules much better. You can even submit them upstream to ZAP, and it will save you time in the long run.

**31:20 Chris Romeo:** Okay, very good. Well, thank you, Tiago. We really appreciate you being with us today and, and to talk about scanning at scale and, and talking about OWASP ZAP again. And always appreciate just learning more about the tools that are out there and and how developers and AppSec folks can, can use these tools. So thank you again.

**31:47 Tiago Mendo:** You're welcome. It was a pleasure.

**31:50 Robert Hurlbut:** The Application Security Podcast is brought to you by Security Journey. Arm your developers with engaging, secure coding lessons developed with input from leading application security experts. Security Journey offers responsive developer training plans that integrate with your existing AppSec testing tools to identify and address address vulnerabilities in your own code. Visit securityjourney.com to try our training today.

---

Source: https://appsecpodcast.com/tiago-mendo-how-to-scan-at-scale-with-owasp-zap/