Tin Zaw -- ModSecurity and #AppSec

0:05Robert HurlbutThe Application Security Podcast. Here we go.

0:10Chris RomeoHey all, we're back with another episode of the Application Security Podcast. On this week's episode, Robert and Chris are attending AppSec USA where they are joined by Tin Zaw, an advocate for mod_security. He dives into its background, use of rules, and advantages. As always, thanks for listening and enjoy.

0:48Tin ZawHello friends, in this episode of the AppSec Podcast, Chris and I are here in Orlando, Florida. We're at the AppSec USA 2017 conference. And today we have Tin Zhao with us, and he's going to be talking about some of the things that he's talking about here at the conference. Welcome, Tin.

1:11Chris RomeoThank you.

1:11Tin ZawYeah, so one of the things that we do when we do our podcast is that we ask for someone to tell us their superhero story or security superhero story. Everybody has an origin. And so, how did you get into security? Tell us about that.

1:29Chris RomeoYeah, so I think mine will probably be more like the security practitioner story. So I started getting into IT or computer industry by being a programmer. So I started writing code for the networks and infrastructure, and then I started writing code making security products. And from making security products, I switched into making secure products. which I help companies to make secure web application products, you know, and then that's when I started to get involved with OWASP. And now I'm one of— I lead one of the projects here, and also I'm a heavy user of this mod_security, both the core rule set and the engine.

2:13Tin ZawOkay, and where are you coming in from? What company again?

2:17Chris RomeoYes, so my day job, I work at Verizon. I lead the security professional services for their customers who are on our CDN, content delivery network, and we help protect them from external threats like web application exploits and so on.

2:33Robert HurlbutSo before we jump into the details of what we're actually here to talk about, you said you actually run a different OWASP project?

2:40Chris RomeoYes.

2:41Robert HurlbutWhich project do you run?

2:42Chris RomeoAutomated Threats to Web Applications.

2:45Robert HurlbutOh, very cool.

2:46Tin ZawOkay, great. I'm familiar with that. Excellent. How long have you been doing that?

2:50Chris RomeoAbout a year and a half. You know, I met— I have known Colin Watson, you know, who started the project for a while, and we started talking, and he was presenting about this project and was looking for a co-leader, and that didn't sink in for me for a while. About 6 months later, I thought maybe I can help him with that because I have seen a lot of these automated threats and automated attacks through my day job. So we started talking and we started updating the handbook to version 1.1, working on the 1.2, and so on.

3:23Tin ZawOkay, yes, I've seen the book. That's how I knew about it. Excellent.

3:26Robert HurlbutWe'll have to have you come back and talk about that a different time.

3:28Chris RomeoI hope so, yeah.

3:29Robert HurlbutYeah, let's talk about mod_security first, but that's— I'll put that in the back of my mind for a future talk.

3:34Chris RomeoThank you.

3:36Tin ZawSo yes, tell us about what you're talking about here at the conference.

3:39Chris RomeoYes, so my talk is about how to use mod_security and how to have the right expectations out of either ModSecurity or any web application firewall that you would be using. So it's about understanding it, setting the right expectations, and how to continuously manage a WAF device.

3:59Tin ZawOkay, so tell us what is a— for our listeners, what is a WAF? We've heard that term before, it's been out there, probably developers have heard of it, but what is it? Tell us about that.

4:09Chris RomeoYeah, so the main characteristics of WAF is that it operates at the application level, in this case HTTP level. It can see through the HTTP transactions because it operates on the web server and that's after the SSL decryption has happened. So, it can look at your web traffic, it can see what kind of payload is going toward and what kind of response is going back and things like that. It can also work at the network level information, things like the IP address and geolocation and so on.

4:42Tin ZawOkay, and so where does that fit in, let's say you have a firewall, you have your website and other things are going on behind the website in the server, where does that fit in that scheme of things?

4:57Chris RomeoRight, so generally, you know, there are 2 ways to deploy WAF. So one is embedded into your web server, right? So whether you're using NGINX or IIS or Apache, right, there's a mod_security module that gets built into that, compiled into that. The other way to do that is at a proxy level. So that proxy can be integrated into your load balancer or it can be part of the CDN that you're using, and that's where you can also deploy your web application firewall.

5:28Tin ZawOkay, okay, very good. And so tell us a little bit more about what is mod_security. And so it is a kind of WAF. it sounds like, or help support that sort of thing. So tell us a little bit more about that.

5:42Chris RomeoYeah, so ModSecurity is really one of the success stories of OWASP, and it started as an open source project by a gentleman named Ivan Ristic back in 2002, and it has been growing in user base and still going strong with the new releases and new people actively working on it. There are some principles behind, you know, mod_security, that is to be open, to be flexible, and also to be passive in monitoring things so it doesn't interfere with the traffic or the pattern, you know, unless you tell it to do so. And predictable behavior in that, you know, what you see is what you get. These rules are open and visible by everyone, so you know what these rules do and you get the predictable behavior out of it.

6:32Tin ZawOkay, so when you say it's open, is it the source code? Is it— how is it implemented?

6:37Chris RomeoYes, so Maw Security is an open source project, and actually it is open source projects in that there are 2 components related to Maw Security. One is what we call the engine, right? The engine is just an empty set of rules, but it has, you know, ability to execute these rules. And the other Maw Security project is what we call the call rule set. And that is the standard and the basic rule set for going together with the mod_security engine. So they are 2 different projects, but they work very closely and they are deployed as a package together.

7:12Robert HurlbutSo maybe we can back up a little bit and see when we're talking about mod_security, where does mod_security actually fit in the architecture here? So I feel like I'm I really don't know the answer to that, so I'm curious as far as where do I put this mod_security thing?

7:30Chris RomeoRight. So, this mod_security, the combination of NGINX and the rule set, you can put it inside your web server, right? So, whether you're running Apache or IIS or NGINX, you can use it as a mod. It's a module inside the web server. That's one way to deploy that. The other way to deploy that is as a proxy somewhere in between the user browser and your application or web server, right? So that would be like your load balancer or SSL terminator in your data center, or it could be at the CDN or content delivery network.

8:07Robert HurlbutSo it's like a web— so you could actually take a web server and install mod_security on it, but it doesn't actually host the applications itself. It's acting as that proxy thing. And then the other option would be to install mod_security directly on the web server.

8:22Chris RomeoThat is correct.

8:23Robert HurlbutOkay, okay, so I have a better idea now about kind of where these things are.

8:27Chris RomeoGreat.

8:27Tin ZawAnd you mentioned those 2 different things too, the engine and the core rule set. In terms of the rule set, how are those managed? Are they something that a team would have to build themselves, or are there a bunch of rules that are already in place? I mean, how does that work?

8:46Chris RomeoYeah, so the goal behind core rule set is to give you a bunch of rules in place that you can get started. And these rules are geared toward addressing the issues defined in OWASP Top 10, right? So things like cross-site scripting, SQL injection, remote file inclusion, and so on that are defined in the OWASP Top 10. So you can get started with that core rule set.

9:07Tin ZawOkay, okay. And then what about adding new rules? How does that work typically?

9:12Chris RomeoYeah, so ModSecurity provides you with the flexibility to write your own rules. So if you have like a custom vulnerability or some custom policy that you want to enforce. So it provides a language called the SecRule, and you can write basically any rule that your heart desires and deploy that with your deployment.

9:31Tin ZawOkay. And so tell me some of the other things that it provides to you. If you're talking to a team and they're wondering, why should I do this? What are some things I need to know in addition to just setting it up? I mean, what is it giving you benefits, I guess, is another way of saying it. But yeah, tell me about that.

9:52Chris RomeoYeah, so some of the benefits that a WAF can give you, and especially that, you know, ModSecurity is designed to give you, is to provide you with the attack detection, right? So it knows, like, you know, when a suspicious pattern or attack pattern is coming toward you, right? And also the option for attack mitigation in that you can either choose to block that suspicious traffic or the attack traffic or you can choose to just log it and flag it and you take a look at it later. It also gives you another option for what we call the virtual patching. That is like when your framework or your application breaks, but you need to buy some time to have it fixed. So, you know, while you're updating your code or recompiling your application, it'll be much easier and faster to write a mod_security rule to block, you know, certain traffic from coming in and taking advantage of that vulnerability. You can also use it for policy enforcement. For example, like, you know, you don't want to deal with certain IP blocks or certain countries, or you don't want to support certain HTTP verbs like, you know, OPTIONS or PROFFILE and so on. So you can use mod_security to enforce your own policies against, you know, the incoming HTTP.

11:07Robert HurlbutSo from the use case that you just described about using it to buy yourself more time. Can you walk us through kind of like an example of how that— because like I'm thinking about the big— I mean, there's a big vulnerability.

11:19Tin ZawYes.

11:19Robert HurlbutIf anybody's been living under a rock for the last month, then they haven't heard about this Equifax breach and the Struts vulnerability that's been kind of tossed around as potentially being the challenge there. So, if we were using the Struts vulnerability as an example, like what would you— what would be kind of the steps you would go through using mod_security to buy yourself more time?

11:39Chris RomeoRight, that's a great question and, you know, very relevant given the time today. There wasn't just one Struck vulnerability, right? There was a few, at least a couple of Struck vulnerabilities. And if you walk through the sequence of events, right, so the vulnerability was discovered and reported to you know, Apache Foundation in this case, by a researcher and they fixed it. They released— Apache released a new version of the Struts framework and they tell everybody that, hey, you know, you have this issue, please use this latest updated version. You as a developer will have to download that patch or download that new framework. You recompile your code and you do all your QA testing, make sure that, you know, it doesn't have any negative side effects related to that, and that takes time, right? That may take weeks or weeks, right? And that is when you're a fully functioning, you know, worldwide machine, secure development lifecycle and all that, right? But you're still vulnerable during that time. What more security gives you is the ability to write that rule just to block that vulnerability from being exploited. and you deploy that, and that will take days instead of weeks. And that buys you like, you know, 1 or 2 extra weeks to take your time to fix it right and deploy it. And you can take that rule away if you want, or you can keep that rule just to see like, you know, who else might be attacking you with that vulnerability. That gives you some visibility into, you know, your traffic.

13:18Tin ZawOkay, so you're watching for certain patterns, looking for certain things. How would you code that? How would you write, let's say for that example, how would you look for those kinds of things? How easy or difficult?

13:33Chris RomeoWell, it is moderately complex. It's not rocket science. A good programmer can do it. At the heart of these most security rules are regex, regular expressions. So you have to be comfortable with them. And on top of that, you have to know what parameter that you want to inspect, what HTTP method that you want to block or allow, and things like that. So you write the regex, you provide the context for the regex to run or to inspect, and you deploy the rule.

14:04Robert HurlbutWhere does the regex come from? Can you figure that out based on the disclosure that came, like when they published the CVE entry? Is there enough?

14:14Chris RomeoIn most cases, yes, because you know that you're vulnerable. path or the parameter or the, you know, the cookie or, you know, how your framework is vulnerable. So usually that information is either included in the proof of concept code or, you know, if not, you can ask the researcher directly like, hey, how do you actually exploit that and how did you come up with that? And you can write a signature based on that information.

14:38Tin ZawDo those rules get—

14:40Robert Hurlbutis there like, is there going to be a rule Is there a place to share the rules? Because like you're saying, OWASP Top 10 is kind of the core rules focus, which that would sound like a modern vulnerability that just hit wouldn't kind of fit into that OWASP Top 10. Is there a place where people can share them so that not everybody has to build this from scratch?

14:58Chris RomeoYeah, so a couple of ways to do that. So one is commercial support for creating and publishing these rules is provided by a company called Trustwave. They are also a supporter of this. more security projects. The other way that what the more security core team is— core rule set team is talking about is to have kind of like a temporal database of, you know, contemporary rules that people can contribute to and they can pull down. So that wouldn't be really part of the core rule set, but that would be, you know, urgent and relevant stuff that people may care about that anybody can contribute and share.

15:36Tin ZawOkay. All right, so you got some rules, you're putting it in place, you're running it. And so tell me about just running it for a while. What are you seeing? What are the results? What are some logging— what are the kinds of things that may come out of mod_security, for example, that somebody needs to look at?

15:56Chris RomeoYeah, so one thing about using mod_security or any web application firewall in general is They are not fix-it-all. They are not set-it-and-forget-it.

16:06Tin ZawOkay.

16:06Chris RomeoRight. What that means is that, you know, you need care and feeding of the machine or these devices, right? So that means that, you know, regularly looking at these logs and outputs and continuously fine-tuning, you know, some of the— dealing with the false positives, you know, dealing with things that you might have missed because, you know, your configuration was a little too cautious on the false positive. positive side, or it could also be like the framework breakage that we just talked about, or it could be the newly discovered vulnerability in your application that you need to update your firewall to protect that again. So, yeah, continuous management is the key in successfully using any WAF and especially more security.

16:48Tin ZawOkay. Do you see that people, because of some of the— maybe there's extra overhead, I'm not sure, that they say, well, we tried a WAF, but we're not using it anymore? I mean, is that why that might happen?

17:00Chris RomeoYeah, I have seen a number of both our customers and other projects or companies that try to use either WAF or bot security, and they ended up like, oh, there are too many alarms, too many alerts, I don't know how to manage it. You can get professional help in that case. You may not have the staff on your you know, on your team to have the expertise, but, you know, there are a number of, you know, commercial entities that can provide you with, you know, these helpfulness managing your WAF. Okay, okay.

17:37Robert HurlbutSo there's different products. So there's commercial WAFs, right? And then there's mod_security and maybe using rule sets from another vendor who's helping you to provide this. What's the reason why would I use mod_security over some commercial WAF, whatever we want to call it.

17:55Chris RomeoRight, right. One thing to note is like a number of commercial WAFs are based on ModSecurity. You know, some admit it and some don't.

18:02Tin ZawGood to know.

18:04Chris RomeoYeah, but you know, the real benefit that you get from using ModSecurity is the openness, right, that you get with any open-source software. And you can inspect it, you can look at it, you can look at the regex itself. There's no black magic here. you don't need to send your data to the cloud, right? Your data stays inside your web server and the regex that is executing is predictable. It doesn't execute 2 ways, right?

18:30Tin ZawOkay. How about the WAF itself? Are there different tests against that itself? I mean, in terms of, is there potential compromise of, you know, just another thing I've added? I've changed my attack surface. It's no longer my web server necessarily, it's something in front of it. But do you know anything about that in terms of just making sure that's up to date or any patches or anything like that that sometimes might have to be applied?

18:55Chris RomeoRight. At this moment, there's no known vulnerability in the mod_security engine. The rules are as effective as it is written. It does very well of what it's written for, but your use case might fall outside of what has been defined. In that case, you have the ability to extend it by writing your own rule or modifying the existing behavior of the rule. One thing to note also is, like, if you're deploying that as a reverse proxy mode in another machine, you need to protect your origin web servers from some attacker bypassing that proxy. So, you need to blacklist the rest of the traffic, but only whitelist the proxy, you know, where that proxy is. So yeah, there are cases that you need to be doing, but—

19:49Tin ZawOkay, so there's still some things if you're, like you said, the reverse proxy, to be aware of that just setting it up is not enough.

19:55Chris RomeoYou need to think about a couple other things here too.

19:57Tin ZawThat's all right, good to know, good to know.

19:59Robert HurlbutSo if somebody wanted to get started, so let's say one of our listeners hears this and they say, oh, mod_security, that sounds really cool. What would you recommend to them? What's a good place to kind of get your feet wet or jump into using mod_security?

20:13Chris RomeoYes, so luckily there's a book that just came out like about a couple of months ago written by, you know, the guys who started this, Ivan, and also the guy who is actively maintaining this core rule set, his name is Christian. There's a second edition of this book called the mod_security handbook, and that is a really good, you know, brotherly guide to getting you through that journey of, you know, installing and deploying and care and maintenance on ModSecurity.

20:43Tin ZawOkay, good, good. Are there any other resources, blogs or anything like that?

20:50Chris RomeoGitHub, right? So ModSecurity, Coruscant, both are active projects. They're taking feedback from users. Okay, so please, you know, that only makes the project stronger and open source go farther.

21:04Tin ZawOkay, so you can go there, you can look and see what issues may be there or follow what's going on.

21:10Chris RomeoExactly, and especially whether you like particular rules or you don't like particular rules. Some rules can be too noisy and some rules can work, so that kind of feedback, even when things are working, is helpful to the developers to know what to keep and what to modify.

21:25Tin ZawOkay, great. Okay, Tim, are there any other things that you'd like to tell us about mod_security or about your talk? At the conference here?

21:36Chris RomeoYes, so my talk is at 11:30 AM, just before lunchtime tomorrow on Friday. The closing thought is that, you know, WAFs have become essential components in secure web application deployment, and that thinking has evolved over time from like, oh, you know, we don't need WAF, to like, oh, WAF buys you time. And today the thinking, the general thinking among the GoAsp community is that, oh yeah, WAF should be a standard part of the secure deployment. So, I just want to end with that thought. Okay, great.

22:08Tin ZawWell, Tim, thank you and we appreciate you joining us today here for the AppSec Podcast at AppSecUSA 2017. Everybody have a great day. Thank you.

22:18Chris RomeoThank you for having me here. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ. And the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.