--- title: "Tin Zaw -- ModSecurity and #AppSec" url: https://appsecpodcast.com/tin-zaw-modsecurity-and-appsec/ date: 2017-10-17 duration_seconds: 1379 guests: ["Tin Zaw"] topics: ["OWASP Top 10", "OWASP Projects"] audio: https://www.buzzsprout.com/1730684/episodes/8122703-tin-zaw-modsecurity-and-appsec.mp3 transcript: true --- # Tin Zaw -- ModSecurity and #AppSec *October 17, 2017 · 23 min* with [Tin Zaw](https://appsecpodcast.com/guests/tin-zaw/) on [OWASP Top 10](https://appsecpodcast.com/topics/owasp-top-10/), [OWASP Projects](https://appsecpodcast.com/topics/owasp-projects/) [Audio](https://www.buzzsprout.com/1730684/episodes/8122703-tin-zaw-modsecurity-and-appsec.mp3) ## Show notes A web application firewall can buy time during a vulnerability crisis, but only if somebody understands and maintains its rules. Tin Zaw explains ModSecurity and the Core Rule Set, starting with where a WAF sits between users and an application. He distinguishes the rule engine from the rules themselves, describes embedded and proxy deployments, and explains how detection, blocking, and logging serve different purposes. The conversation uses the Apache Struts vulnerabilities as an example of virtual patching while a team prepares a software update. Chris and Robert also ask about writing signatures, sharing rules, tuning false positives, and the risks of adding another component to the stack. Tin closes with practical starting points for learning and contributing to open-source application protection. The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/). About Security Journey Security Journey provides application security education for developers and everyone in the software development lifecycle. → [Learn more about Security Journey](https://www.securityjourney.com/) Connect with Tin Zaw: → [Tin Zaw on LinkedIn](https://www.linkedin.com/in/tinzaw/) Mentioned in this episode: → [ModSecurity](https://owasp.org/www-project-modsecurity/) → [OWASP Core Rule Set](https://coreruleset.org/) → [Apache Struts](https://struts.apache.org/) Chapters: 00:00 ModSecurity and application security with Tin Zaw 01:11 Tin’s security origin story 02:33 Contributing to OWASP projects 03:59 What a web application firewall does 04:42 Embedded and proxy WAF deployments 06:32 The ModSecurity engine and Core Rule Set 08:27 Using and extending security rules 09:31 Detection, blocking, logging, and virtual patching 11:19 Responding to an Apache Struts vulnerability 13:18 Writing signatures from vulnerability information 14:40 Sharing rules with the community 15:36 Tuning false positives and maintaining the WAF 17:37 Why choose an open-source WAF? 18:30 Managing the WAF’s own attack surface 19:59 Getting started and finding resources ## Transcript *3,832 words · assemblyai* **0:05 Robert Hurlbut:** The Application Security Podcast. Here we go. **0:10 Chris Romeo:** Hey all, we're back with another episode of the Application Security Podcast. On this week's episode, Robert and Chris are attending AppSec USA where they are joined by Tin Zaw, an advocate for mod_security. He dives into its background, use of rules, and advantages. As always, thanks for listening and enjoy. **0:48 Tin Zaw:** Hello friends, in this episode of the AppSec Podcast, Chris and I are here in Orlando, Florida. We're at the AppSec USA 2017 conference. And today we have Tin Zhao with us, and he's going to be talking about some of the things that he's talking about here at the conference. Welcome, Tin. **1:11 Chris Romeo:** Thank you. **1:11 Tin Zaw:** Yeah, so one of the things that we do when we do our podcast is that we ask for someone to tell us their superhero story or security superhero story. Everybody has an origin. And so, how did you get into security? Tell us about that. **1:29 Chris Romeo:** Yeah, so I think mine will probably be more like the security practitioner story. So I started getting into IT or computer industry by being a programmer. So I started writing code for the networks and infrastructure, and then I started writing code making security products. And from making security products, I switched into making secure products. which I help companies to make secure web application products, you know, and then that's when I started to get involved with OWASP. And now I'm one of— I lead one of the projects here, and also I'm a heavy user of this mod_security, both the core rule set and the engine. **2:13 Tin Zaw:** Okay, and where are you coming in from? What company again? **2:17 Chris Romeo:** Yes, so my day job, I work at Verizon. I lead the security professional services for their customers who are on our CDN, content delivery network, and we help protect them from external threats like web application exploits and so on. **2:33 Robert Hurlbut:** So before we jump into the details of what we're actually here to talk about, you said you actually run a different OWASP project? **2:40 Chris Romeo:** Yes. **2:41 Robert Hurlbut:** Which project do you run? **2:42 Chris Romeo:** Automated Threats to Web Applications. **2:45 Robert Hurlbut:** Oh, very cool. **2:46 Tin Zaw:** Okay, great. I'm familiar with that. Excellent. How long have you been doing that? **2:50 Chris Romeo:** About a year and a half. You know, I met— I have known Colin Watson, you know, who started the project for a while, and we started talking, and he was presenting about this project and was looking for a co-leader, and that didn't sink in for me for a while. About 6 months later, I thought maybe I can help him with that because I have seen a lot of these automated threats and automated attacks through my day job. So we started talking and we started updating the handbook to version 1.1, working on the 1.2, and so on. **3:23 Tin Zaw:** Okay, yes, I've seen the book. That's how I knew about it. Excellent. **3:26 Robert Hurlbut:** We'll have to have you come back and talk about that a different time. **3:28 Chris Romeo:** I hope so, yeah. **3:29 Robert Hurlbut:** Yeah, let's talk about mod_security first, but that's— I'll put that in the back of my mind for a future talk. **3:34 Chris Romeo:** Thank you. **3:36 Tin Zaw:** So yes, tell us about what you're talking about here at the conference. **3:39 Chris Romeo:** Yes, so my talk is about how to use mod_security and how to have the right expectations out of either ModSecurity or any web application firewall that you would be using. So it's about understanding it, setting the right expectations, and how to continuously manage a WAF device. **3:59 Tin Zaw:** Okay, so tell us what is a— for our listeners, what is a WAF? We've heard that term before, it's been out there, probably developers have heard of it, but what is it? Tell us about that. **4:09 Chris Romeo:** Yeah, so the main characteristics of WAF is that it operates at the application level, in this case HTTP level. It can see through the HTTP transactions because it operates on the web server and that's after the SSL decryption has happened. So, it can look at your web traffic, it can see what kind of payload is going toward and what kind of response is going back and things like that. It can also work at the network level information, things like the IP address and geolocation and so on. **4:42 Tin Zaw:** Okay, and so where does that fit in, let's say you have a firewall, you have your website and other things are going on behind the website in the server, where does that fit in that scheme of things? **4:57 Chris Romeo:** Right, so generally, you know, there are 2 ways to deploy WAF. So one is embedded into your web server, right? So whether you're using NGINX or IIS or Apache, right, there's a mod_security module that gets built into that, compiled into that. The other way to do that is at a proxy level. So that proxy can be integrated into your load balancer or it can be part of the CDN that you're using, and that's where you can also deploy your web application firewall. **5:28 Tin Zaw:** Okay, okay, very good. And so tell us a little bit more about what is mod_security. And so it is a kind of WAF. it sounds like, or help support that sort of thing. So tell us a little bit more about that. **5:42 Chris Romeo:** Yeah, so ModSecurity is really one of the success stories of OWASP, and it started as an open source project by a gentleman named Ivan Ristic back in 2002, and it has been growing in user base and still going strong with the new releases and new people actively working on it. There are some principles behind, you know, mod_security, that is to be open, to be flexible, and also to be passive in monitoring things so it doesn't interfere with the traffic or the pattern, you know, unless you tell it to do so. And predictable behavior in that, you know, what you see is what you get. These rules are open and visible by everyone, so you know what these rules do and you get the predictable behavior out of it. **6:32 Tin Zaw:** Okay, so when you say it's open, is it the source code? Is it— how is it implemented? **6:37 Chris Romeo:** Yes, so Maw Security is an open source project, and actually it is open source projects in that there are 2 components related to Maw Security. One is what we call the engine, right? The engine is just an empty set of rules, but it has, you know, ability to execute these rules. And the other Maw Security project is what we call the call rule set. And that is the standard and the basic rule set for going together with the mod_security engine. So they are 2 different projects, but they work very closely and they are deployed as a package together. **7:12 Robert Hurlbut:** So maybe we can back up a little bit and see when we're talking about mod_security, where does mod_security actually fit in the architecture here? So I feel like I'm I really don't know the answer to that, so I'm curious as far as where do I put this mod_security thing? **7:30 Chris Romeo:** Right. So, this mod_security, the combination of NGINX and the rule set, you can put it inside your web server, right? So, whether you're running Apache or IIS or NGINX, you can use it as a mod. It's a module inside the web server. That's one way to deploy that. The other way to deploy that is as a proxy somewhere in between the user browser and your application or web server, right? So that would be like your load balancer or SSL terminator in your data center, or it could be at the CDN or content delivery network. **8:07 Robert Hurlbut:** So it's like a web— so you could actually take a web server and install mod_security on it, but it doesn't actually host the applications itself. It's acting as that proxy thing. And then the other option would be to install mod_security directly on the web server. **8:22 Chris Romeo:** That is correct. **8:23 Robert Hurlbut:** Okay, okay, so I have a better idea now about kind of where these things are. **8:27 Chris Romeo:** Great. **8:27 Tin Zaw:** And you mentioned those 2 different things too, the engine and the core rule set. In terms of the rule set, how are those managed? Are they something that a team would have to build themselves, or are there a bunch of rules that are already in place? I mean, how does that work? **8:46 Chris Romeo:** Yeah, so the goal behind core rule set is to give you a bunch of rules in place that you can get started. And these rules are geared toward addressing the issues defined in OWASP Top 10, right? So things like cross-site scripting, SQL injection, remote file inclusion, and so on that are defined in the OWASP Top 10. So you can get started with that core rule set. **9:07 Tin Zaw:** Okay, okay. And then what about adding new rules? How does that work typically? **9:12 Chris Romeo:** Yeah, so ModSecurity provides you with the flexibility to write your own rules. So if you have like a custom vulnerability or some custom policy that you want to enforce. So it provides a language called the SecRule, and you can write basically any rule that your heart desires and deploy that with your deployment. **9:31 Tin Zaw:** Okay. And so tell me some of the other things that it provides to you. If you're talking to a team and they're wondering, why should I do this? What are some things I need to know in addition to just setting it up? I mean, what is it giving you benefits, I guess, is another way of saying it. But yeah, tell me about that. **9:52 Chris Romeo:** Yeah, so some of the benefits that a WAF can give you, and especially that, you know, ModSecurity is designed to give you, is to provide you with the attack detection, right? So it knows, like, you know, when a suspicious pattern or attack pattern is coming toward you, right? And also the option for attack mitigation in that you can either choose to block that suspicious traffic or the attack traffic or you can choose to just log it and flag it and you take a look at it later. It also gives you another option for what we call the virtual patching. That is like when your framework or your application breaks, but you need to buy some time to have it fixed. So, you know, while you're updating your code or recompiling your application, it'll be much easier and faster to write a mod_security rule to block, you know, certain traffic from coming in and taking advantage of that vulnerability. You can also use it for policy enforcement. For example, like, you know, you don't want to deal with certain IP blocks or certain countries, or you don't want to support certain HTTP verbs like, you know, OPTIONS or PROFFILE and so on. So you can use mod_security to enforce your own policies against, you know, the incoming HTTP. **11:07 Robert Hurlbut:** So from the use case that you just described about using it to buy yourself more time. Can you walk us through kind of like an example of how that— because like I'm thinking about the big— I mean, there's a big vulnerability. **11:19 Tin Zaw:** Yes. **11:19 Robert Hurlbut:** If anybody's been living under a rock for the last month, then they haven't heard about this Equifax breach and the Struts vulnerability that's been kind of tossed around as potentially being the challenge there. So, if we were using the Struts vulnerability as an example, like what would you— what would be kind of the steps you would go through using mod_security to buy yourself more time? **11:39 Chris Romeo:** Right, that's a great question and, you know, very relevant given the time today. There wasn't just one Struck vulnerability, right? There was a few, at least a couple of Struck vulnerabilities. And if you walk through the sequence of events, right, so the vulnerability was discovered and reported to you know, Apache Foundation in this case, by a researcher and they fixed it. They released— Apache released a new version of the Struts framework and they tell everybody that, hey, you know, you have this issue, please use this latest updated version. You as a developer will have to download that patch or download that new framework. You recompile your code and you do all your QA testing, make sure that, you know, it doesn't have any negative side effects related to that, and that takes time, right? That may take weeks or weeks, right? And that is when you're a fully functioning, you know, worldwide machine, secure development lifecycle and all that, right? But you're still vulnerable during that time. What more security gives you is the ability to write that rule just to block that vulnerability from being exploited. and you deploy that, and that will take days instead of weeks. And that buys you like, you know, 1 or 2 extra weeks to take your time to fix it right and deploy it. And you can take that rule away if you want, or you can keep that rule just to see like, you know, who else might be attacking you with that vulnerability. That gives you some visibility into, you know, your traffic. **13:18 Tin Zaw:** Okay, so you're watching for certain patterns, looking for certain things. How would you code that? How would you write, let's say for that example, how would you look for those kinds of things? How easy or difficult? **13:33 Chris Romeo:** Well, it is moderately complex. It's not rocket science. A good programmer can do it. At the heart of these most security rules are regex, regular expressions. So you have to be comfortable with them. And on top of that, you have to know what parameter that you want to inspect, what HTTP method that you want to block or allow, and things like that. So you write the regex, you provide the context for the regex to run or to inspect, and you deploy the rule. **14:04 Robert Hurlbut:** Where does the regex come from? Can you figure that out based on the disclosure that came, like when they published the CVE entry? Is there enough? **14:14 Chris Romeo:** In most cases, yes, because you know that you're vulnerable. path or the parameter or the, you know, the cookie or, you know, how your framework is vulnerable. So usually that information is either included in the proof of concept code or, you know, if not, you can ask the researcher directly like, hey, how do you actually exploit that and how did you come up with that? And you can write a signature based on that information. **14:38 Tin Zaw:** Do those rules get— **14:40 Robert Hurlbut:** is there like, is there going to be a rule Is there a place to share the rules? Because like you're saying, OWASP Top 10 is kind of the core rules focus, which that would sound like a modern vulnerability that just hit wouldn't kind of fit into that OWASP Top 10. Is there a place where people can share them so that not everybody has to build this from scratch? **14:58 Chris Romeo:** Yeah, so a couple of ways to do that. So one is commercial support for creating and publishing these rules is provided by a company called Trustwave. They are also a supporter of this. more security projects. The other way that what the more security core team is— core rule set team is talking about is to have kind of like a temporal database of, you know, contemporary rules that people can contribute to and they can pull down. So that wouldn't be really part of the core rule set, but that would be, you know, urgent and relevant stuff that people may care about that anybody can contribute and share. **15:36 Tin Zaw:** Okay. All right, so you got some rules, you're putting it in place, you're running it. And so tell me about just running it for a while. What are you seeing? What are the results? What are some logging— what are the kinds of things that may come out of mod_security, for example, that somebody needs to look at? **15:56 Chris Romeo:** Yeah, so one thing about using mod_security or any web application firewall in general is They are not fix-it-all. They are not set-it-and-forget-it. **16:06 Tin Zaw:** Okay. **16:06 Chris Romeo:** Right. What that means is that, you know, you need care and feeding of the machine or these devices, right? So that means that, you know, regularly looking at these logs and outputs and continuously fine-tuning, you know, some of the— dealing with the false positives, you know, dealing with things that you might have missed because, you know, your configuration was a little too cautious on the false positive. positive side, or it could also be like the framework breakage that we just talked about, or it could be the newly discovered vulnerability in your application that you need to update your firewall to protect that again. So, yeah, continuous management is the key in successfully using any WAF and especially more security. **16:48 Tin Zaw:** Okay. Do you see that people, because of some of the— maybe there's extra overhead, I'm not sure, that they say, well, we tried a WAF, but we're not using it anymore? I mean, is that why that might happen? **17:00 Chris Romeo:** Yeah, I have seen a number of both our customers and other projects or companies that try to use either WAF or bot security, and they ended up like, oh, there are too many alarms, too many alerts, I don't know how to manage it. You can get professional help in that case. You may not have the staff on your you know, on your team to have the expertise, but, you know, there are a number of, you know, commercial entities that can provide you with, you know, these helpfulness managing your WAF. Okay, okay. **17:37 Robert Hurlbut:** So there's different products. So there's commercial WAFs, right? And then there's mod_security and maybe using rule sets from another vendor who's helping you to provide this. What's the reason why would I use mod_security over some commercial WAF, whatever we want to call it. **17:55 Chris Romeo:** Right, right. One thing to note is like a number of commercial WAFs are based on ModSecurity. You know, some admit it and some don't. **18:02 Tin Zaw:** Good to know. **18:04 Chris Romeo:** Yeah, but you know, the real benefit that you get from using ModSecurity is the openness, right, that you get with any open-source software. And you can inspect it, you can look at it, you can look at the regex itself. There's no black magic here. you don't need to send your data to the cloud, right? Your data stays inside your web server and the regex that is executing is predictable. It doesn't execute 2 ways, right? **18:30 Tin Zaw:** Okay. How about the WAF itself? Are there different tests against that itself? I mean, in terms of, is there potential compromise of, you know, just another thing I've added? I've changed my attack surface. It's no longer my web server necessarily, it's something in front of it. But do you know anything about that in terms of just making sure that's up to date or any patches or anything like that that sometimes might have to be applied? **18:55 Chris Romeo:** Right. At this moment, there's no known vulnerability in the mod_security engine. The rules are as effective as it is written. It does very well of what it's written for, but your use case might fall outside of what has been defined. In that case, you have the ability to extend it by writing your own rule or modifying the existing behavior of the rule. One thing to note also is, like, if you're deploying that as a reverse proxy mode in another machine, you need to protect your origin web servers from some attacker bypassing that proxy. So, you need to blacklist the rest of the traffic, but only whitelist the proxy, you know, where that proxy is. So yeah, there are cases that you need to be doing, but— **19:49 Tin Zaw:** Okay, so there's still some things if you're, like you said, the reverse proxy, to be aware of that just setting it up is not enough. **19:55 Chris Romeo:** You need to think about a couple other things here too. **19:57 Tin Zaw:** That's all right, good to know, good to know. **19:59 Robert Hurlbut:** So if somebody wanted to get started, so let's say one of our listeners hears this and they say, oh, mod_security, that sounds really cool. What would you recommend to them? What's a good place to kind of get your feet wet or jump into using mod_security? **20:13 Chris Romeo:** Yes, so luckily there's a book that just came out like about a couple of months ago written by, you know, the guys who started this, Ivan, and also the guy who is actively maintaining this core rule set, his name is Christian. There's a second edition of this book called the mod_security handbook, and that is a really good, you know, brotherly guide to getting you through that journey of, you know, installing and deploying and care and maintenance on ModSecurity. **20:43 Tin Zaw:** Okay, good, good. Are there any other resources, blogs or anything like that? **20:50 Chris Romeo:** GitHub, right? So ModSecurity, Coruscant, both are active projects. They're taking feedback from users. Okay, so please, you know, that only makes the project stronger and open source go farther. **21:04 Tin Zaw:** Okay, so you can go there, you can look and see what issues may be there or follow what's going on. **21:10 Chris Romeo:** Exactly, and especially whether you like particular rules or you don't like particular rules. Some rules can be too noisy and some rules can work, so that kind of feedback, even when things are working, is helpful to the developers to know what to keep and what to modify. **21:25 Tin Zaw:** Okay, great. Okay, Tim, are there any other things that you'd like to tell us about mod_security or about your talk? At the conference here? **21:36 Chris Romeo:** Yes, so my talk is at 11:30 AM, just before lunchtime tomorrow on Friday. The closing thought is that, you know, WAFs have become essential components in secure web application deployment, and that thinking has evolved over time from like, oh, you know, we don't need WAF, to like, oh, WAF buys you time. And today the thinking, the general thinking among the GoAsp community is that, oh yeah, WAF should be a standard part of the secure deployment. So, I just want to end with that thought. Okay, great. **22:08 Tin Zaw:** Well, Tim, thank you and we appreciate you joining us today here for the AppSec Podcast at AppSecUSA 2017. Everybody have a great day. Thank you. **22:18 Chris Romeo:** Thank you for having me here. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ. And the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org. --- Source: https://appsecpodcast.com/tin-zaw-modsecurity-and-appsec/