Neil Smithline -- OWASP Top 10 #10: Logging

0:00Chris RomeoHey folks, welcome to season 3, episode 10 of the Application Security Podcast. On this episode, we discuss the OWASP Top 10 A10, Insufficient Logging and Monitoring. We're joined by Neil Smithline, who is one of the project co-leads for the OWASP Top 10 now, and he breaks down what is this issue, how do you find it, and how can we fix it industry-wide. So we hope you enjoy. The Application Security Podcast. Here we go. Hey folks, welcome to season 3, episode 10 of the Application Security Podcast. On this episode, we are going to explore another one of the new items on the OWASP Top 10, 2017 edition. This is insufficient logging and monitoring. And joining us, joining Robert and I today to describe this issue for us is Neil Smithline, who's one of the co-project leads on the new OWASP Top 10 kind of new project setup. You might remember Neil was part of the group that we spoke with at AppSecUSA that took us behind the scenes into the OWASP Top 10 2017. And so Neil, we're real happy to have you with us today to dive deeply into A10.

1:44Neil SmithlineHappy to be here.

1:45Chris RomeoSo let's, let's go, let's jump right in and go for it, and let's just paint a picture for our listeners. What is A10, insufficient logging and monitoring?

1:56Neil SmithlineWell, so first, this was one of them that was one of the— we had 2 top 10 issues that were decided by community voting as compared to from metrics we were given from various vendors. So this was one of the 2 that was voted in. So, you know, we don't have great statistics as to, you know, as to why it's here as compared to most of the others. You know, we don't have actual breaches, but it was very highly rated. It was in the top 2 or 3 of the issues that came in. So what it refers to is any time, basically any sort of failure you have in your logging system that doesn't either alert you to a security breach appropriately or help you figure out what happened in a security breach after it happened. So, it's not something a tool can actually test for because it's not clear that there is perfect guidance as to, you know, what's right. I don't think you could write a grammar that says, look, if the log looks like this, then you know it's perfect.

3:05Robert HurlbutMm-hmm.

3:07Neil SmithlineSo, you know, tools can help. They can make sure you're logging, make sure, you know, you're logging in a secure fashion, you're sending it you know, off the server is typically a good thing to do, or somehow managing it such that the log itself is secure. But there's no real specification for what it means. I asked on Twitter some people, you know, what they thought were good examples of it. And my favorite response was someone came back and said just about every breach could have been made smaller, right? They used the phrase smaller blast radius. If it had had better logging or monitoring. And I thought that was probably accurate. Maybe not every breach, but certainly any ongoing breach, you hope that logging is going to record what's happening. At least probably that for every breach, but for the longer breaches, you hope there are going to be alerts that take place.

4:01Robert HurlbutYep.

4:03Chris RomeoSo from— so I guess the issue that we're describing here with A10, So we've been talking about logging and monitoring, but the issue with A10 is really insufficient logging and monitoring. So does that mean that an application— that people are deploying applications that are not properly— so they're just not doing the right level of logging that they need to do? Is that the real kind of crux of the issue?

4:26Neil SmithlineSo I think people are at times not logging appropriately. Um, I find that particularly common with teams that have less cloud experience that, you know, they seem to feel there's always a debugger, they never even need to think about logging. But I think the bigger problem is that there's not appropriate monitoring of the logs. Logs either contain not enough or too much information, and even when they contain the right stuff, nobody notices that they were. You know, the Target breach took place over months, a month. Um, there were millions of records, tens of millions of records downloaded, you know. And, you know, it's not clear, but there's people who've worked on the security team mentioned that there's, um, that there were alerts that took place, but they were all sort of isolated, happening in individual— happening in, in little logging monitors. And there was no— In little teams and not centralized into one location. So nobody— it never occurred to anyone, oh my gosh, we have a big breach going on. And I think that sort of identifying, you know, that there's a breach going on, having the appropriate alerts produce pagers, you know, and that you need some sort of infrastructure to do that. If you have any sort of a reasonable application, you probably need to be using some logging application. some monitoring application that's looking at it.

5:58Chris RomeoAnd that's, you know, I guess some of the products in this space are what we refer to as SIEM products that are S-I-E-M that are doing, you know, security event management and helping with that. I think we'll get into that a little bit later when we kind of talk about solutions and things here. So, So incident response and logging, I'm thinking, are going to be things that are very closely tied together. So when you have a— when you have something bad that happens with your application, and let's face it, every application that's ever been written is going to have something bad happen with it. So this is really the monitoring side that you were describing there is really more from an incident response perspective. So when something bad happens, the people who are responsible for cleaning up the mess actually have something to look at and something to work with, right?

6:51Neil SmithlineYeah, it's, it's the ops part of DevOps or DevSecOps, right? It's happening while the system's running.

7:00Chris RomeoYeah, and I, I worked incident response for a number of years, and I can tell you how frustrating it is when you come in and a customer says, okay, we, we think we've been hacked, come in here and, and help us figure out what happened. And you go, you start to look at the infrastructure, you say, okay, you know, where's the logging server? And they're like, what? What's a logging server? And you're like, this is going to be a long day in this investigation because I got very little to work with as far as what actually happened here. So, so I think we talked about what can go wrong if our application is vulnerable to this. If we don't have insufficient logging and monitoring, there's going to be breaches and things. How do we find this problem as it exists? Yeah.

7:42Neil SmithlineSo I'm not sure logging actually stops the breaches.

7:46Chris RomeoOkay.

7:47Neil SmithlineRight. Logging, logging might detect, you know, if someone's trying lots of logins, you know, trying common passwords or something. But I think that it's really important in helping you identify that a breach has taken place or is taking place and let you, you know, cut it off at that point in time.

8:08Chris RomeoOkay, so like, like the quote was saying, it limits the blast radius then. Yes, it doesn't take out the whole city, it just takes out a, you know, a small, small portion, and then you're able to detect it along the way. Okay, that makes sense.

8:21Robert HurlbutSo, so in terms of taking a little bit different approach on this, there are a lot of tools out there, scanning tools and so forth, that align themselves with the OWASP Top 10. They bill themselves as being able to find the OWASP Top 10 risks. But in terms of logging, that's something that's going to be interesting.

8:42Neil SmithlineHow would you do that?

8:43Robert HurlbutWould you think about, or maybe suggest, some other ways to find if a company is doing the proper logging? I mean, like, for example, a checklist or something like that? What are some thoughts on that?

8:56Neil SmithlineYeah, so I, I don't think there are automated tools for detecting this. I know there are tools that advertise they're compatible with the OWASP Top 10, but— and that they find all the vulnerabilities, all the issues, but I imagine they're being liberal there. So, I think a checklist is sort of the right way to go. There needs to be a basic pattern of what you log. You log all security events. You log— data accesses, whatever seems appropriate. You go through depending on what your app is and what you're trying to protect. You go through, you create the logging of that information, and then you have to take those logs. Besides what you log, you then also have to take care with the logs. They should certainly be in some write-once append-only style. I think they really need to go off the machine and go to some sort of secure logging server. Linux, Unix has had syslog for, well, forever. And most of the cloud platforms have something in place for that as well. So, you have to get the log off. And then sort of the third thing is you need something looking at the logs that's trying to identify the problematic behavior. You know, this is where a SIEM tool could potentially be very helpful. You know, we don't talk about SIEM tools in the top 10 because we didn't really advertise for-profit tools. We tried to stay away from that, but it certainly seems appropriate here. So, you know, and some kind of rules engine that's looking through your logs, looking for things that are problematic, that are atypical.

10:50Chris RomeoSo, to start, we have to ensure we're properly generating the right logs so that we have something to even perform monitoring on. And then we likely need some type of a strategy or solution that's going to take— because, you know, you can get 20 million log entries from a firewall in a given day, in a production firewall, no problem. And so, I think that's where, you know, that's where the challenge of just the depth and breadth of the number number of log entries comes into play here. It's not an excuse for why somebody doesn't do the proper monitoring. It's just realizing that there is a lot more that kind of comes into managing the volume of logging.

11:30Neil SmithlineYeah, you definitely need some automated solution. There's no way somebody could do this manually. Even, you know, even if you were to look at the log only once a day, you know, it's just too hard to pick out things that are problematic from You know, or potentially problematic from the huge array of data that's gathered.

11:49Chris RomeoYeah.

11:50Neil SmithlineSo, and you need— I'm sorry, you need to gather that data. So if you need to perform a postmortem on a breach, you have data there to help you.

11:57Chris RomeoYeah. So I love that. I love this idea of a checklist and creating some type of a checklist. I'm seeing some references that were included in A10. Neil, what are some of the places that you would recommend people go to in other kind of OWASP projects as far as where they can find things to add to their own checklist if they're going to make just a specific checklist for logging?

12:21Neil SmithlineWell, OWASP has a cheat sheet on logging, and there's also the ASVS, which has an entry given to logging. It's not a terribly long entry, but it sort of has the high-level things of what you should be of what you should be worrying about. And proactive controls also talk about intrusion detection inside of them. I think those are probably the best locations to look, you know, other OWASP sources.

12:57Chris RomeoYeah, I think that'll, that'll help our listeners if somebody's thinking, hey, I want to make a checklist now because I like this idea. I wanted to give them some places to, to be able to go and look. So have you seen anything in the industry? Has there been any kind of big events that have occurred where we can kind of point the finger at logging and say, here's a good story for how logging really had an impact in a particular kind of real live event?

13:30Neil SmithlineSo, I'm more familiar with cases where logging wasn't obviously sufficient or the monitoring wasn't sufficient.

13:39Chris RomeoOkay.

13:42Neil SmithlineYeah, I can't think of one off the situation where, you know, someone, you know, there was something that became public. I think those tend not to be public so much. Yeah, because incident response.

13:56Chris RomeoYeah, incident response and incident response in general doesn't— that teams don't normally do a lot of broadcasting about what, uh, the final results of their reports and, uh, and, and whatnot.

14:08Neil SmithlineSo, um, so, you know, the Equifax attack took place over months and months, you know, exfiltrated huge amounts of data, huge number of requests, um, you know, and nothing seemed to have triggered for quite some time. So, I mean, that's certainly, you know, the flip side of when it was sufficient, of when it was insufficient. And, you know, I've seen just a few weeks ago, I think Equifax published that perhaps even more data than they said was stolen got stolen. So obviously they're having trouble going back through their logs and figuring out what data was actually exfiltrated from them.

14:45Chris RomeoYeah.

14:46Neil SmithlineSo, you know, you know, that's certainly a situation where it's bad. One of the things the Top 10 recommends, and I think is a good idea, is to let your penetration testers or use a penetration penetration testing app, some sort of dynamic analysis tool, run and see what happens. See what your logs say. See if you could figure out in retrospect what was done. See if any of your alarms go off. And if not, you know, try to figure out what you need to do to fix the problem.

15:18Chris RomeoYeah, I think that's good advice just to— yeah, just to test your overall logging infrastructure and see if you're getting the right alerts and if you're getting any alerts, because I'm sure there's situations where you think things are all set to log at a super high level, and then you start running tools and you're not getting anything coming out of the, out of the systems. And it has to make you think and go back and fix, hey, something's missing here in our overall logging approach.

15:44Neil SmithlineYes.

15:47Chris RomeoSo I want to pick on another issue that I see that was kind of drawn out here in A10. And that is kind of the crossover between information leakage and logging. So what do we need to worry about from an information leakage perspective with our logs?

16:01Neil SmithlineSo I think there's a few situations that we need to worry about. First, logs frequently have confidential information. They might have, right, whether that's specific information regarding a user, is something that will be in violation of PII or GDPR or other privacy requirements. They could also have account numbers and, you know, other things. You know, try to mask those as best as possible in the log, certainly. But, you know, those things can end up in logs. So, just an attacker just getting on your system and stealing your log, maybe get enough information to execute a successful breach or stealing of the log may be the breach itself. They may be able to recreate information from the log. Also, the logs are sort of— an attacker can watch the log and see what's being— the more familiar they are with the logs, the information that's going there on the monitoring system, the better job they can do to try to avoid setting off alarms, setting off things that are going to show up in the logs. So you want to carefully watch to make sure that, you know, the logs are not someplace where an attacker is likely to get them. They should be protected as they're sensitive information. I certainly wouldn't, you know, leave them on the same server. Putting them in some kind of secure log system, you know, Splunk is very popular right now. That seems a fine choice.

17:39Robert HurlbutYeah.

17:41Neil SmithlineBut somehow getting them off the machine into someplace that's secure and probably encrypting those logs once you got them, right? You still don't want— you're going to need to keep the log for some period of time. And I'm not really sure, but certainly we're probably talking months, you know, maybe even longer, because you want to be able to look back if you discover a breach to try to figure out what happened. So you want to be encrypting the logs when you put them into long-term storage. Make sure they're encrypted when they're backed up. And generally just try to, you know, remember that the logs are important.

18:15Chris RomeoYeah, and I've seen, you know, one of the biggest issues that I know different applications have struggled with in the past, and there's a definitive requirement that exists for all applications, and that is never log passwords because, you know, we still have applications that that exist in the world today that are going to log passwords in the event of some type of a problem or something. They're going to think, hey, we'll help the developer by logging the password. It's like, no, that's part of that sensitive data that should never be logged. And if you do have to log a password, you want to make sure you block it out and fill it with asterisks or Xs or whatever so that it's not— You don't have live passwords that ever exist inside of your log files for the reasons you spoke of there, Neil, about the fact that an attacker could get a copy of that log at some point. And why would we make their job so easy by placing system passwords in the log file? Seems like it's too easy.

19:16Neil SmithlineAnother technique I've seen is some kind of hashing done on account numbers. So that allows you Being the same hashing algorithms used over and over, an account number, you can, you see a hash number, you see a hash number and you can use that to identify what took place in a specific session or involving a specific account. But, you know, it can sort of obscure the account number so that it can't be directly stolen. You know, like if you're logging a credit card number, you probably don't want to do that. You know, you could either log the 4, the last 4, or use some kind of algorithm to perform a mapping. Though you need to be careful there, right? Because you don't want it to be reversible.

20:04Chris RomeoSo yeah, I know the PCI DSS folks are— that's one of the things they're always on the lookout for is reversible schemes for things like credit cards. You know, if you're gonna try to obfuscate the credit card numbers that you store, Is your scheme properly not allowing somebody to trace through backwards and kind of figure out what they are? So, definitely a concern.

20:30Neil SmithlineYeah.

20:34Chris RomeoSo, and I guess my final question, I know Robert probably has some more questions here as well. But my final question here is, if we were to, if we were to think kind of into the future, so if we thought 5 years into the future from now, And at that point, we were able to say, hey, you know what, we kind of solved this logging thing as an industry. Now, what do you think would have to happen between now and 5 years into the future for us to be able to make that statement and say, hey, logging is solved, we can take it out of the top 10, it's— we'll replace it with something else? What do you think are some of the things that have to happen there?

21:10Neil SmithlineI assume you mean technically, as compared to legally, because putting companies out of business for leaking huge amounts of data seems like a real motivating effect in my mind.

21:24Chris RomeoYeah, and that's GDPR, right? I mean, GDPR, we don't know, we're still what, at this stage, we're still a month or so away from the enforcement or 2 months away from enforcement. That could potentially do that. I don't think any of us really truly know what the impact— we can guess what the impact of GDPR is going to be. But yeah, so I was thinking more on the technical side though, but great point about the legal side. We could squash this with a particular legal move.

21:48Neil SmithlineYeah, and I suspect that's going to be needed. So on the technical side, I think we need to see more inclusion of automated logging detection. I would like to see some formalized checklist or even some automated tool that could try to help you create the appropriate data in your logging. It's not clear to me what really what that tool would do, how it would work, but that would be a good start. You know, in the end, even if you have a checklist, you know, it's still a manual process, at least, you know, as we understand it now. One of the ways you could avoid, you know, is make it better is by using an existing library and trusting that the library is doing the logging. But, you know, certainly the better ones are doing appropriate logging or close to appropriate logging, but it's not clear to me that all of them— in fact, I'm sure all of them aren't. So I, I think that's a hard thing to reach. Um, I'll tell you, I'd love to see it gone in 5 years. I suspect, you know, in the next T10, this is going to be up in the list as well.

22:59Chris RomeoYeah, so you think it could actually become kind of a bigger— it could, it could grow in its, its, the, uh, the, the depth and breadth of this particular type of problem then?

23:09Neil SmithlineYeah, I, I suspect that's going to be the case. You know, I, I think Equifax is, you know, is just, you know, one of the, the earlier ones of what we're going to see here. Um, well, I'm not even sure it's the one of the early ones. There's been plenty more. And I think we'll continue to see that. So, I feel like we don't really have the right technology in place now.

23:31Chris RomeoYeah. And that actually made me think of something else. And I'm going to actually send this question to Robert first. And Neil, that'll give you like 30 seconds to think of an answer on this one. But I was just thinking about this whole IAST kind of revolution that's going on in the application security tool space. the interactive application security testing tools that are embedding themselves inside of the runtimes of Java, .NET specifically. Robert, I'm going to go to you first on this. What do you think about that as a— I mean, could that potentially solve our logging problems if we had instrumentation inside of all of our applications directly? If it could log from the inside out, could it do better than developers just having to specifically create logging statements throughout all of their code? Would IAST solve this? Maybe?

24:27Robert HurlbutI was thinking it might help. In fact, one of my questions was, I know a couple other things that was mentioned in the top 10 page there was 2 tools, AppSensor and mod_security. We've talked about mod_security before on this podcast. I think AppSensor may be something we'll look at in the future. But even in any of the IAST type of integration, yeah, it sounds like at least in theory that it might be able to provide some of those kinds of resources or at least put that in place to help with developers who haven't thought about where do I need to put the logging here or there and so forth. So in theory, it sounds like it might work.

25:15Chris RomeoYeah, Neil, you got any thoughts on something like IAST being part of the solution here for logging?

25:22Neil SmithlineWell, I think that it provides a centralized place for logging to take place, which is helpful. But it's still not clear to me that it is doing that. IAST is still a fairly new technology. And, you know, we'll have to see. Perhaps that's the way that injecting something into your system like IAST is going to be the long-term solution.

25:54Chris RomeoYeah, I'm just thinking about the scenario, like, it seems like logging, and, you know, you touched on this in kind of your list of how to get us to solving this problem, problem within 5 years of, you know, having some type of a tool that can determine what logging needs to happen. It almost seems like logging, in theory, should be something that we could wrap around an application.

26:18Robert HurlbutYeah.

26:20Chris RomeoAnd we should be able to provide either a library or something that could help the developer. The developer shouldn't have to make all the decisions about, oh, should I log something right here or not? not. That seems like an antiquated approach to how we can do this. And I don't know, maybe somebody's listening to this and they're going to go create the next big security startup that's going to figure out how to wrap logs in an automated fashion around an existing piece of code. And who knows, maybe they'll become a billionaire from our idea. Hopefully, they'll send us a t-shirt or something.

26:50Neil SmithlineRight.

26:52Chris RomeoI think that's really the answer to where we go and how we solve this problem in over the next 5 years.

26:59Neil SmithlineSo I don't know that that's really a complete solution, though, because even if you manage to record the appropriate log actions, you still need to respond to them. Yep. And I think that's probably the more difficult of the 2 tasks.

27:14Chris RomeoDo you think that's something that should be in— so when I think of response, as somebody who used to work incident response, used to be part of a security operations center, I always think of these things as being very disconnected. So meaning that the security operations center, the incident response function is completely different than the DevOps function. But are you thinking that there's, in your kind of way of thinking about logging, are you thinking that logging is more something that advises the developers on how to fix the problems or something that advises the incident response team kind of from the more classic way I'm thinking about it?

27:48Neil SmithlineI think it, I think, you know, the only way it would affect the developers were if they were in a DevOps sort of role. I definitely think it's more to the incident response team. You know, I think it's hard for developers to be on call all the time. You know, it depends on lots of things. So, I do think I like it more in the incident response, at least in terms of an actual breach occurring. If there's, you know, notifying someone who's in a dev DevOps role, I think, is very appropriate also. In many situations, you don't want your developers actually having access to the production systems. So exactly what, you know, what a tool is going to provide, what an alert is going to provide there, is not clear to me.

28:40Chris RomeoYeah, yeah. And I see your point about it's not a complete solution just to have— even if we had I hate to say the word artificial intelligence, machine learning. We had some type of higher order technology that could determine where the logs are. There's definitely still a need for somebody to respond to those logs through the incident response function. So even a tool's not going to solve the entire problem here. You still need people that are chasing down these problems and helping to resolve them.

29:15Neil SmithlineAnd I think SIEM tools are looking to sort of be, you know, if I could use the AI word, they're looking to be AI at analyzing the logs. I'm not an expert in SIEM tools, but they still seem sort of primitive and awkward to use as far as I can tell. My experience with them has been, you know, that they're very costly.

29:38Chris RomeoYeah.

29:39Neil SmithlineI don't mean—

29:39Robert HurlbutYeah.

29:40Neil SmithlineI meant in terms of effort.

29:41Chris RomeoI think it's both, actually. I think they're costly in terms of effort and in terms of— because it seems like, I don't know, I mean, I've been— I might be being unfair, and if somebody out there is working for a SIEM vendor and you want to argue this with me, I'd love to, I'd love to, to hear your perspective on it. Um, but so it's been a few years since I've been removed from maybe the integration side of this, but it seemed like nobody could really buy a SIM off the shelf is what I'm remembering. There was no such thing. Like, there's always, there's always some degree of customization that has to happen, which means customization always means costs more money in the end, at least kind of from, from my perspective.

30:19Neil SmithlineYeah, they use words like training to make it sound more appetizing, but it's, it's still a fair amount of work and you still get false positives, um, that need to be addressed.

30:33Chris RomeoAnd there's still SIEMs that sit there and don't get used as well. I've heard horror stories about people that do these giant integrations of SIEMs and then they never actually look at it. They just leave it running in the corner because now they can check the box that says, hey, we have a SIEM, we do, we do that logging stuff. And there's still no— and that's once again back to one of my pet peeves, the compliance over actual value.

30:56Neil SmithlineYeah.

30:57Chris Romeoin the security proposition.

30:58Neil SmithlineYeah, yeah. And, you know, I think you're raising an issue about checkbox security, you know, where the security exists just to fill a checkbox and not actually be used. That's probably a different conversation.

31:13Chris RomeoYeah, but I think logging can fit into that, though. Logging can fit nicely. Logging is one of those areas that can very easily become a compliance type of activity. Did you log that? Yes, we logged it technically. Did anybody ever do anything with it? Well, that wasn't the question.

31:27Neil SmithlineYeah, right, right. And even if— even checklist I've seen says, you know, logs are regularly reviewed for security events, wording of that sort, but it's not clear what that means. You know, you know, you really need something that is, you know, I feel you really need something automated to help manage this, and it has to translate into emails or pager alert or whatever seems appropriate if something is actually detected.

31:58Chris RomeoYeah, I agree. And that's— I think that's the future of logging and monitoring and incident response and how they all fit together. The future is automation because we just can't— we can't pay enough people to sit in a room to triage all the events manually. that are occurring for a large enterprise. It's just— there's just too much stuff happening, and somebody's going to lose something along the way.

32:23Neil SmithlineRight, right. Even if you could pay people to do that, people aren't good at that.

32:28Chris RomeoYeah, it's true. Luckily, computers are. We just have to make sure we get the right solutions to truly find the problems that exist.

32:36Neil SmithlineYes, computers are good at slogging through stuff.

32:40Chris RomeoThat's good, and we're very happy I love that. So Neil, thanks for taking the time today to introduce our listeners and us to the details of A10. Pointers I guess I'll give people, obviously, if you haven't read the OWASP Top 10 2017 edition, what rock are you hiding under at this point? It's been out for a number of months at this stage. And so Neil, thanks for taking the time. Thanks for breaking it down for us. And we look forward to seeing you at a conference here soon.

33:09Neil SmithlineThanks for having me. Thanks for listening to the Application Security Podcast. If you enjoy the podcast, please do us a favor and visit the iTunes Store and give us a 5-star rating. Our intro music is 8-Bit Kung Fu by Born and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @appsecpodcast or on the web at www.appsecpodcast.org.