---
title: "Guy Barhart-Magen -- Log4j and Incident Response"
url: https://appsecpodcast.com/guy-barhart-magen-log4j-and-incident-response/
date: 2022-09-23
duration_seconds: 2625
guests: ["Guy Barhart-Magen"]
topics: ["Software Supply Chain"]
audio: https://www.buzzsprout.com/1730684/episodes/11372259-guy-barhart-magen-log4j-and-incident-response.mp3
video: https://www.youtube.com/watch?v=_lBVIHWhZKk
transcript: true
---

# Guy Barhart-Magen -- Log4j and Incident Response

*September 23, 2022 · 44 min*

with [Guy Barhart-Magen](https://appsecpodcast.com/guests/guy-barhart-magen/)

on [Software Supply Chain](https://appsecpodcast.com/topics/supply-chain/)

[Audio](https://www.buzzsprout.com/1730684/episodes/11372259-guy-barhart-magen-log4j-and-incident-response.mp3) · [Video](https://www.youtube.com/watch?v=_lBVIHWhZKk)

## Show notes

With nearly 25 years of experience in the cyber-security industry, Guy held various positions in both corporates and startups. In his role as the CTO for the cyber crisis management firm Profero, his focus is making incident response fast and scalable, harnessing the latest technologies and a cloud-native approach. Guy is the BSidesTLV chairman and CTF lead, a Public speaker in well-known global security events (SAS, t2, 44CON, BSidesLV, and several DefCon villages, to name a few), and the recipient of the Cisco “black belt” security ninja honor – Cisco’s highest cybersecurity advocate rank. Guy joins us to explore his front-row seat for the incident response with Log4j.

You are now listening to the Application Security Podcast, brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
Hey folks, welcome to another episode of the Application Security Podcast.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Guy Barhart-Magen:  
→ [Profero](https://profero.io/)  
→ [Log4j](https://logging.apache.org/log4j/2.x/)

Mentioned in this episode:  
→ [Profero](https://profero.io/)  
→ [Log4j](https://logging.apache.org/log4j/2.x/)  
→ [Aleph One's "Smashing The Stack for Fun and Profit"](http://phrack.org/issues/49/14.html)  
→ [Heartbleed](https://heartbleed.com/)  
→ [OpenSSL](https://www.openssl.org/)

Chapters:  
00:00 Meet Guy Barhart-Magen: Log4j and Incident Response  
02:18 I promise I'll be back for year 60 at least. for  
07:40 That's really cool. I miss those days. There was a, there  
10:10 From the kind of going back into the security realm, what  
28:31 I mean, that's historically— when you think about some of the  
31:06 Yeah. So here's a question for you as you kind of  
36:02 Yeah, I mean, that's a common challenge though, right  
39:01 Yeah. That's, that is definitely a challenging scenario. And as an

## Transcript

*7,657 words · assemblyai*

**0:00 Chris Romeo:** With nearly 25 years of experience in the cybersecurity industry, Guy Barhart McGinn held various positions in both corporate and startups. In his role as the CTO for the cyber crisis management firm Profaro, his focus is making incident response fast and scalable, harnessing the latest technologies in a cloud-native approach. Guy is the BSides TLV chairman and CTF lead. He's a very well-known public speaker across lots of different global security events and is the recipient of the Cisco Black Belt Security Ninja honor, Cisco's highest highest cybersecurity advocate rank. Guy joins us to explore his front-row seat for the incident response that happened with Log4j. There are many AppSec lessons to learn by understanding the greater depth of Log4j. I know I personally learned lots of different facts that I'd had no perspective on as far as what happened when Log4j was released and the days after the release and notification to the industry. We hope you enjoy this episode with...

**0:58 Guy Barhart-Magen:** Guy Pearce.

**0:59 Chris Romeo:** Guy Barhart McGinn.

**1:00** You are now listening to the Application Security Podcast, brought to you by Security Journey. When you finish this episode, check out our other show, High Five, to stay up to date with all the hot AppSec news.

**1:10 Chris Romeo:** Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo, Chief Security Officer at Security Journey, and I am also joined by my good friend Robert Hurlbut. Hey, Robert.

**1:22 Guy Barhart-Magen:** Hey, Chris.

**1:24 Chris Romeo:** Yeah, it's Robert and Principal Application Security Architect at Acquia, as well as Threat Modeling Lead.

**1:29 Guy Barhart-Magen:** Great to be here.

**1:30 Chris Romeo:** Yeah, it's been a few weeks since we bumped into each other in the illustrious Las Vegas. We were both out there for, I guess, a little bit of Black Hat, a little bit of DEF CON, a little bit of a lot of different things in Las Vegas. A lot of catching up. That was the best thing for me is just catching up with friends I hadn't seen in 2 or 3 years in person. And that was just fantastic. Just a great opportunity. Yeah, that's the part that we missed during the pandemic is not being at conferences, connecting with people and stuff. And so, and that was also my first visit to DEF CON. So that was an interesting experience for me. Took me, you know, it was only, you know, year 30. What a good time to go and show up at the first one.

**2:16** Absolutely.

**2:17 Chris Romeo:** And I promise I'll be back for year 60 at least. for DEF CON 60, but maybe, maybe in between. But let's get to our guest here, Guy Barnhart, again, who I happened to bump into in the hallways at Black Hat, which seems like, you know, with the statistics say that you shouldn't really bump into anyone you know in Las Vegas. But we had the opportunity to bump into each other and reconnect. Guy and I worked together back at Cisco a decade or so ago and excited to have him on the show. And Guy, we always throw our guests right into the deep end of the pool with the question of, what is your security origin story? So our audience loves to know where are people coming from. And so as far as your origin story, what say you?

**3:01 Guy Barhart-Magen:** Good question. I started out with computers pretty early. I think I was maybe 5 or 6 with my first computer. I had a Commodore 128D, which was a gaming computer of the time. It had, like, a built-in pixel editors, you can program games. So I was really drawn into that aspect of programming and writing programs in BASIC at the time. And a few years later, with my first 286 XT, where we had actual games, I was frustrated by the games constraining my progress by having silly things like health counts and score and stuff like that. And I was intrigued of how would you actually go around and implement those things and can I change them while I'm running the game? Hence, uh, permanent state resident programs were born and cheat codes and reverse engineering the software in order to understand what actually went on behind the scenes. So a lot of assembly, and I started writing in C, in Pascal, an assortment of other languages less prone to reverse engineering like Lua and Lisp, unfortunately. And that was really my first steps in programming, trying to understand memory structure, disassembly, reverse engineering software. Nothing security-oriented in mind at the time. This was— maybe the security was of the games trying to protect themselves against people like me, but there were no networking games at the time. Anything that you would do would only affect your own computer. There was nothing like you could have cheat codes for Halo 5 or whatever. So it was always very localized. And when the BBS movement started, and I'll be the first to admit I owned the BBS at the time, the BBS movement started, we started sharing information about what you could do with this kind of skills or what kind of programs you can find out and what you can do elsewhere. A very famous paper came out which really blew my mind at the time. It was how to smash the stack for fun and profit. So that was like my very, very early steps in security and it all went downhill from there.

**5:13 Chris Romeo:** Yeah. It's interesting that you, you know, you've basically been breaking things since a very young age is what I took away from that. Like you've, you know, been understanding how things work and then breaking them and putting them back together, which, is a great foundation for security. I mean, it sounds like you've had a security mindset even before you knew what security really was. You were pulling things apart and putting it back together. And so, yeah, that's really kind of that breaker mentality.

**5:46 Guy Barhart-Magen:** I think that a lot of people in our community, in security specifically, but even in larger communities, when you think about what is the security mindset, it's actually much bigger than that. It goes to engineering. It goes to problem solving. It goes to the delight that you get from solving a puzzle. Not all of these skills apply specifically to security, but you'll find a lot of people in security having those skills. You've just been at DEF CON, so the lockpicking village is getting bigger every year. Lockpicking is not actually exactly a skill that you need in application security, but there's a lot of correlation between people who enjoy picking locks, solving those puzzles, and doing the same in application security, trying to understand what's exactly going to be the input that's going to break that API. Very cool.

**6:41 Chris Romeo:** You know, one of the other things you mentioned was BBS. I hadn't heard that term in so long. Bulletin Board System or Service for those who may not even know what that is, but I can remember the dial-ups and all that stuff. So fun memories. You're making me nostalgic here. I know, right? I mean, I had a BBS as well. Okay, here it is. This has never been announced on the Application Security Podcast. I had a BBS called The Doors of Perception.

**7:09** Oh, wow.

**7:10 Chris Romeo:** I mean, how cool is that? That is very cool. I watched The Doors movie like 27 times. You know, and Jim Morrison, when the doors of perception are cleansed, all things become known or something like that. So I don't know, Guy, if you want to admit the name of your BBS.

**7:25 Guy Barhart-Magen:** I'd be happy to. But I will preempt that saying that I was 11 or 12 when I opened it. Its name was Action Packed Sandwich.

**7:33 Chris Romeo:** I love it.

**7:37 Guy Barhart-Magen:** APS. Excellent.

**7:39 Chris Romeo:** That's really cool. I miss those days. There was a, there was a different sense of community When you weren't connected 100% of the time. Like, the internet is great for so many different things, and— but having that time limit that the BBS has brought in, just— it brought like— it was such a— such an elite experience too, because like only people like us were the ones that were running BBSs and logging into them and typing messages and playing games and doing those types of things. But yeah, I miss those simpler days of computing. Yeah, definitely. Well, our topic today, Guy, and thanks for speaking on it and talking about it, is Log4j. And that's been in the news certainly in the last 2, 3 years. But for those of us who are developers and can remember years ago building applications and needing to figure out a good logging solution, Help those who may not have heard of it. What is Log4j?

**8:42 Guy Barhart-Magen:** So dismissing the entire security aspect of it for a moment, Log4j is a good library. It's a very well-known, well-vetted, very widely used logging library. Basically, if you're going to write a program in Java, in most probability, you're going to use Log4j. Log4j as a library is not only handling all of the logging backend needs that your application might have, but also has the capability to add much more on top of that. If you want to enrich your log files with additional metadata, modify formatting, keep strict controls over what goes into the logging framework and how the logs should display depending on what your inputs are. So, it's a big chunk of software. It's not something small.

**9:33** Yeah.

**9:34 Guy Barhart-Magen:** small or minuscule. And obviously big chunks of software will have issues, and they do have issues from time to time, but that haven't stopped Log4j from being so prolific. So in general, it's a small team. There are about 3 to 4 developers more or less working on it. And I don't have numbers of how widely used it is, but from reading the Log4j backlash, you can understand it almost Anything that touches Java touched Log4j. So it's a very widely used library used for handling whatever logging needs you may have.

**10:10 Chris Romeo:** So from the kind of going back into the security realm, what was the problem that resulted in the Log4j vulnerability and all of the compromises and stuff? You know, if we kind of flash back to what was the issue that an attacker originally found that they were able to take advantage of and, you know, cause all this chaos that hit the world for a period of time?

**10:40 Guy Barhart-Magen:** It's actually a funny story. I'll try to intertwine 2 different timelines together. We were first made aware of an issue in Log4j from one of our remote teams in New Zealand. So I'm located in Israel, they're in New Zealand. They're about half a day apart. So late night Thursday for them, they found some chatter on Twitter that a new vulnerability has been found on Minecraft servers. So, you know, we take note of it, we care, but we actually, we don't care. It's Minecraft. So whoever has vulnerability, deal with it. So it's not very important. But the vulnerability was disclosed as something related to the logging of Minecraft. So we took a note of it, handed the key off to the Israel team who started Friday morning. We took a look at it, didn't find anything interesting. We said, okay, if something pops up, we'll take care of it. Now, switching back, what was the actual issue with the vulnerability? So as the vulnerability goes, it's not a vulnerability such as, I don't know, SQL injection or heap overflow, something like that. It's a misuse of a feature. I touched upon the enrichment part of Log4j earlier. What you could do with Log4j was basically think of it as a search and replace. you could log certain components into your data stream, and Log4j would replace certain identifiers or tokens or magic keywords or whatever with a different piece of information that you might want to enrich your data with. For example, let's say that you have user IDs coming in with the log, with whatever log input you're using, and you would like to replace those user IDs with the username or user domain or some enrichment coming from LDAP, over LDAP from your domain controller, Active Directory, et cetera. So, the logging library allows you to query that remote piece, like Active Directory, and ask him, this user ID, do you know it? Can you give me back the username? Or whatever piece of information you want. You put it in your log file, and now your log file is transformed from something having an inscrutinous piece of an integer, like a user identifier, Mm-hmm. It is something that you can actually read the log and use, like, what is the actual username that did this specific request? The vulnerability here was that there was no proper limitations or controls over what the remote could be. So you could add in your syntax, in your logging syntax, a call that says, query this remote LDAP server, which the attacker controls. So why is that important? That means that you would cause some log line to be entered into the system. For example, typing a username or something like that, which is logged. The log would have a string very similar to cross-site scripting or SQL injection. We have some piece of string that says, fetch this piece of information from that remote server. And it'll go to fetch that piece of information from a server which you control. You supply that piece of information. of information, and you will return back to the Log4j instance a piece of information that would cause it to crash or cause it to behave unexpectedly or cause it to behave very specifically in the way that you would like to do, like causing a remote code execution. So the flow would be, you're sending some attacker-controlled input into the logging system. The logging system would try to log it. It would identify that this token needs to be enriched. It will reach out to whatever remote server provides that enrichment facility. The remote server will respond with some payload, supposedly the enriched data. In fact, an attacker-controlled piece of data that will be able to circumvent or execute whatever code is needed to execute on the logging server itself, therefore allowing the attacker to control or run remote code execution on the logging server itself, by issuing some logging input into the system. There are 2 things to note here, and I'm going back to our timeline in a moment. 2 things of note here. The first one is that this is not a vulnerability in the way that this system is working. The system was supposed to reach out to external facilities to get their data. It's designed into the system. It's not a vulnerability. For example, one of the features that you can use the system for is that if you want to log an IP, you would also like to log its FQDN. So you have the possibility to query DNS server, this is my IP, please give me the FQDN, and log the FQDN along with the IP. So it's a reasonable feature to ask. It's been in the code for ages. It's not something new in any way. And it's not a vulnerability per se, because the fact that you reach out to external providers for additional data is basically how APIs work all over the world. The problem was that there were no sufficient controls. And the bigger problem was nobody expected this to happen. People were thinking of a logging library as something that takes data from input and puts it into a file or into stdout or whatever, and didn't consider that this library has much more functionality than what you would expect. Because, you know, it's been around for a while. It's been used extensively. It features Features get piled on over time. It's very difficult to keep a piece of software to its core features over time. And nobody reads the documentation. Sad reality, but that's the way it is. The solution in the end— jumping ahead— the solution in the end was to fix the library in the source to not allow or to disable this kind of remote enrichment requests or those remote lookups. Okay. But that doesn't really solve it. They turned off the feature. So from an architectural point of view, this feature could have been implemented differently to provide proper controls, proper configuration, and manage expectations by users of that library. I mentioned in the beginning that this was an actual vulnerability. It's not actually a vulnerability in the sense that the problem was not in the software, in the sense that this is the bug. However, it was a vulnerability in the sense that attackers were very quick to use it in order to gain access to remote systems and start running them. Going back to our timeline, Thursday night, Minecraft published— there was a publication on the attack on Minecraft. Friday morning, haven't seen anything. Friday noon, we started seeing the first rise in coin miners along some of our customers. We didn't put one and one together at first because coin miners, ExaMrig specifically, have nothing to do with Log4j. They've been around for a while. They basically— coin miners, for those not familiar with it, are software running on your computer just like any virus, but instead of infecting your computer for, I don't know, whatever nefarious purposes, they actually want to make use of your CPU and your resources in order to mine different coins like Bitcoin or Ethereum or whatever. So So again, it's financially motivated. They just want to get as many bots on their network as possible in order to increase their chances of actually mining the coin. So we saw, we saw some XM-Rig instances being deployed and we started investigating as an IR firm. That's what we do, we investigate. And we started to investigate what was the attack vector, how was the coin miner introduced into those systems? So these were cloud instances, so they were not part of a network or something like that. So we knew that they had to come in externally, and we looked at the logs. And looking at the logs, something immediately popped up, which was the string that the attackers used in order to create this remote code execution, to create that enrichment. So once that popped up, we started looking at all of the logs and looking at that, and we saw that this was a specific way to invoke the Log4j vulnerability. Just like in an example that was posted on GitHub, I don't know, 2 or 3 years prior, something like that. So it was not that the attackers were trying very hard. They were just spraying this new attack coming from the Minecraft server, going to the POC published on GitHub, taking the first example. It works, worked, got access, put a coin miner on the server. So this was Friday noon before all hell broke loose. Hell broke loose around Saturday afternoon for us. So that was like Saturday early morning for the US. And we already saw that. We started putting some mitigations in place, starting to try to answer customer questions like, how do I know if I'm vulnerable to Log4j? Do I have Log4j in my system? And we didn't have a good answer for that because usually when we are hunting for threats, let's say there's a threat actor in your system, there'll be breadcrumbs, there'll be evidence, there'll be someone trying to move from system to system. Right.

**19:39 Chris Romeo:** system.

**19:40 Guy Barhart-Magen:** In a piece of software doing its work as intended, there's not a lot of issues to find. And more than that, most of the appliances or systems that you're running in your business, in your enterprise, that use Log4j don't have a software bill of materials, SBOM, to tell you, this is the things that we are using, and Log4j is one of them. So our customers and ourselves were pretty much in the dark. We had no idea how much Log4j was out there, who was using Log4j, are they using a vulnerable version, are all versions vulnerable, is there a mitigation that we could use or not? So these were, like, really, really early steps of that journey. The next step for us was trying to take measure of how prolific the problem was. Are we looking at some specific instances of somebody being able to deploy coin miners, or this is something that looks like it's going to be much wider scale? And it looks very— it looked very, very scary at first. We started doing the research. We have collaborations with other research teams around the world looking at the same problem, trading IOCs, information reports, et cetera. And we took whatever we had and we published it and said, look, this is going to be very big. Start looking for those IoCs in your system. This is things that you need to put in your WAF right away, that if you see a request containing these kinds of strings, block them. We don't know if you're going to be vulnerable to that or not, but at least block them at the source and find out how much of a vulnerability you're carrying later. And then, we started to look at tools. Our customers asked us, how vulnerable are we? Do we have systems with Log4j? And this is something that I was not comfortable answering because we actually don't know. It's not something that we can answer. We know that you have, I don't know, 5 different servers running Windows. Do they run Log4j? You have 20 different servers running Linux. Do they run Log4j? I have absolutely no idea. And the customers didn't have any idea at all. And one of our customers reached out to us and said, look, I found some scanner on GitHub that scans for Log4j. What do you think about that? So we took a quick look at the code. That was pretty funny. It actually worked. What it would do, the scanner, it would kind of like take a list of IP addresses. You can supply it in a file. It'll go and probe each of those IP addresses and send a request with a DNS payload that says, go to this DNS and resolve it and give me back the FQDN.

**22:23** Okay.

**22:24 Guy Barhart-Magen:** and it would hook to the DNS in the backend. And if somebody made a request for that specific IP address to that— sorry, for that specific FQDN to that DNS server, that means you're vulnerable because no one except the Log4j would make those requests. However, our answer was an emphatic no, don't use this software because the DNS, which we looked into, was hosted in China. which basically meant that anybody running these tools are going to have a list of all of their servers being sent, all of their Log4j vulnerable servers being sent in real time back to a DNS controlled by someone in China who might not be malicious. I don't know, it might be a pet project by someone, but it would not be a good idea. I would try to do something self-hosted in this scenario.

**23:12 Chris Romeo:** Yeah, the threat seems to be high. Just in that scenario, like, just being very careful.

**23:17 Guy Barhart-Magen:** Try not to publish the list of your vulnerable servers to an external entity you do not control.

**23:22 Chris Romeo:** Yeah, it doesn't matter where they are in the world.

**23:24 Guy Barhart-Magen:** Yeah, right.

**23:24 Chris Romeo:** It doesn't matter if they're in Zimbabwe, we don't want to publish a list of all vulnerable servers to anybody.

**23:31 Guy Barhart-Magen:** Yeah. So we took that as a challenge and we actually built our own piece of software and released it in about 2 days as open source to the community. community. And our piece of software did basically the same thing. It would, kind of like Nmap, you'd give it a scope, a subset of network addresses, like, I don't know, 192.168.0.0/24, and it will scan all IP addresses in that range, try to send a payload, a Log4j vulnerable payload to each and every one of them. The difference for us was that the same binary that was sending those requests was also a neutered LDAP server. So what would happen is it would send the request with an LDAP payload in it, and it would say, go to this remote LDAP server, which is basically the same binary sending the request, and it would locally log any kind of incoming LDAP request and say, look, this server tried to contact me, trying to run an LDAP query. Now, because this is not a real LDAP server, this is our binary running and scanning the network, we have a 100% true positive rate. We might miss out false positive, might miss out negatives, but if somebody contacted back the binary, it was 100% vulnerable to Log4Shell. We released that tool to the community. We got some feedback. Everybody that was using it was really into it. Completely local, not sending information to anyone. And that actually helped us find out a couple of things. The first one was give a good answer, somewhat reliable answer. So how many of your devices on your network are vulnerable to Log4j? So it's not a definitive answer because it would only scan for specific IPs on specific ports for specific protocols. But basically, if you're doing logging in Java, you're probably going to use HTTP, which means you're going to use either 80 or 443. You're going to process HTTP protocol requests.

**25:28 Chris Romeo:** Okay.

**25:28 Guy Barhart-Magen:** And Pareto's laws, if we're going to hit 80%, 90% of instances, let's start with that and worry about the other 10% later. So that was our thinking at the time. So we started running this tool at our customers' locations, and we started getting hits. We saw how prolific the problem was. Tens, hundreds of servers answering back saying, yeah, I'm trying to resolve this LDAP request, writing it down, saying this is a vulnerable server. And we could give our customers a list saying, go ahead, fix those servers. The funny thing was that we had a couple of vendors on the list, meaning that some appliance connected to the network answered those requests. So what I mean by appliance is that these customers went ahead and went to some big vendor and bought some server and hosted that server on their internal farm or connected it into their network one way or another, VPC, whatever. Doesn't really matter here. And those appliances were vulnerable to Log4j. The customers have no control over the internal mechanics of that appliance. They have no control over the source code. It's a complete black box to them. But they do know that it's absolutely 100% prepositive. It called back. There was a callback. It was vulnerable. So they reached out to those vendors, and at least 2 of those vendors said, no, we're not vulnerable for Log4j. And mind you, big names have big booths in Black Hat. And they said, we are not vulnerable. And they pulled us into the conversation saying, look, our IR team, they found out that your appliance is vulnerable. We need the patch and we need it yesterday. And they said, no, we are not vulnerable. We have no indication that we are compromised in any way. So we got into the conversation and we told them, you know what? You might not be compromised at all, but you are vulnerable. Look, here's the evidence. Here's the payload, go run it yourself. And it took this back and forth something on the order of 5 to 7 days to get a proper answer from them. And in the end, I reached out to my own connections in those companies and asked, what's going on? It's like the entire world is burning. Why it's taking you 7 days to acknowledge that you're vulnerable when we already proved it on day 1? And the answer kind of surprised me and kind of didn't. And that is that, While it might be very easy to prove that an appliance is vulnerable, it's not very easy to push a fix through QA and through all of those different hoops and stages that you need in order to have a hotfix out in the field for something that big. So even if you know that you have a vulnerability, expecting a very speedy fix for that vulnerability is something that probably is not going to work for you. So our recommendation at the time was focus more on mitigations, detections, because fixes will only come in at a later time. So maybe that's another thing to take from this. If you see something very big, global, spanning the world, don't expect fixes the day after. It's going to take longer than that.

**28:30 Chris Romeo:** And I mean, that's historically— when you think about some of the other big historical issues we've had as a community, community. Like, I mean, I lived through Heartbleed. I think we all lived through Heartbleed. My first big vulnerability from an incident response perspective was an SNMP issue back in, oh man, '99 or 2000 maybe, that was industry-wide. And there were— it was very hush-hush. There was an industry group kind of meeting behind the scenes to try to deal with it before it was made public and stuff. But I think that same So I've seen that same scenario you just described for 20 years of like mitigations are where it's at in the very beginning because you can look to a vendor and say, oh, please, you need to fix this. And they say, yeah, we do. When are you going to have a patch? Hmm. Yeah, maybe a week, maybe a couple of weeks. You know, we're working on it. It's top priority. And it is. It's just a testimony to the fact that those vendors just can't move at lightning speed. which is unfortunate. But a whole other issue is, you know, you think about DevOps in the world of applications, nobody's doing DevOps from an appliance-based software perspective. Like, they have the technology to push 50 changes a day and then send them out across a fleet of appliances. They just haven't gotten there yet. That suite of products just hasn't gotten there. I might have just gone off on a tangent.

**29:52 Guy Barhart-Magen:** I don't know. No, that's an interesting parallel. I was talking to my a partner a day or two ago, and he was like, he was raging on because getting an over-the-air update for his Tesla takes minutes. He can get like 2 updates a day. Updating his PlayStation is something that requires him to physically connect a wire, go through different hoops, download it, and making sure that the firmware doesn't crash. And it's like, it's the same technology, basically. We can do the same kind of things where we are not in exactly the same place. Yeah.

**30:25** Yeah.

**30:26 Chris Romeo:** And Tesla is certainly pushing the envelope of everything in a lot of different areas. But yeah, the over-the-air software updates, like they are pushing the envelope on what can be done with a 5,000-pound motor vehicle that's going down the road. A Tesla is really just a giant computer. Like that's what I figured out when I took a closer look at them. It's a car with a gaming PC built in basically that that has all the functionality you need.

**30:52 Guy Barhart-Magen:** Basically, yeah. And a not-so-bad security model. They still have issues from time to time, but they have a very strong team working on it. So I have a lot of faith in the way that Tesla is working.

**31:05 Chris Romeo:** Yeah. So here's a question for you as you kind of step back and think about Log4j from maybe the 10,000-foot view. Has— did Log4j cause a change in our industry? Like, have people woken up to the challenges, the importance of logging, and the challenges of logging? Because I have a running joke, like, whenever there's a new top 10 list that's ever released, you know where logging's always gonna appear. Logging's always— it makes the number 10 spot, the number— or maybe the number 9, but that's as far as it can go up because it's like everybody's like, oh, we have to think about it.

**31:46** We have to put it on our list, but—

**31:47 Guy Barhart-Magen:** Yeah.

**31:48 Chris Romeo:** I don't really want to focus on it and draw people's attention. So has Log4j changed the perception and the threat landscape in regards to logging?

**31:57 Guy Barhart-Magen:** That's a good question. I honestly don't know the answer to that, but I can draw some conclusions. One of the things that really made me sad to be part of this industry was when we reached out to the Log4j team It's a different story, but I'll try to summarize that. We saw how hard that team was working. To everyone who does not have that memory, everybody worked on Log4j straight through 2 days or 3 days before Christmas, through Christmas, after Christmas, into the new year. This was like the worst time of the year to have that vulnerability out in the open. So people were really working around the clock. And what we could see, was that the Log4j team was getting a lot of heat, a lot of hate also on Reddit, on Twitter, and elsewhere. Who wrote this stupid library? Why is my logging library even making remote connections? What kind of nitwits coders did this? This is amateur stuff, blah, blah, blah, et cetera. So, we believe that they were being really put in the spotlight for the wrong reason. So we reached out to them and said, look, we know you're working so hard. They're basically 3 guys. You can look at the GitHub page. They're hosted on the Apache org and they have the GitHub page. They're just like 3 to 4 guys doing pull requests and modifying that library, maintaining it for so many years. It's not their fault that this issue was just popping up before Christmas. Somebody found it out, tried to make something of it. They have to pay the price.

**33:37 Chris Romeo:** Yeah.

**33:37 Guy Barhart-Magen:** They're not getting paid for this work. They're not getting paid for maintaining this library being used by basically anything running Java on the planet. They didn't get any kind of compensation. And they worked through Christmas making release after release, fixing things as they came in, as new ways to use different mechanics of Log4j were being discovered. They released another fix and another fix and another fix. By the way, that also got hate from the community. Why can't they fix this already? Why do they keep hotfixing it instead of going to the root causes and fixing it? Blah, blah, blah. When we reached out to them and said, look, we appreciate your efforts. We want to make a donation. We made a $5,000 donation to the Log4j team through the Apache Foundation. And they told us that they don't really have a process to do that because it never happened before. Nobody ever donated money to them.

**34:29 Chris Romeo:** Wow.

**34:29 Guy Barhart-Magen:** They got like the occasional, you know, $50 here, $100 there through GitHub Pages, but nobody was ever interested in supporting that project and the work that they were doing. It was like, you know, a couple of guys wrote a logging library. Does it matter that it's over a couple of billion devices? Nobody cared. And we tried to rally people to the flag and saying, look, this is wrong. This is not the way it should be. You are not going to pay for using that library because it's open source? Fine. At the very least, You fix some bug, push it back upstream so everybody can enjoy that fix. You made some effort, you're using that in your appliance, you're selling their appliance for millions and millions of dollars, throw $1,000 to the guys that worked over Christmas to fix the stuff. Do something, at least support the community. Be supportive of the community if you're not even going to financially support this. And think about it from the other way. Think about all of the different companies, all of the different vendors, the huge lists on GitHub that people maintained of different appliances and versions that were vulnerable to Log4j. Not one of them stepped up, came forward and said, look, it's just 3 guys. I'm going to take 2 engineers on my team for the next 2 weeks and get this done, get this solved. No one did it. Nobody supported this.

**35:46 Chris Romeo:** Yeah.

**35:48 Guy Barhart-Magen:** So I don't know. In the end, my feeling was that we raised the flag, we jumped the barricades, we tried to rally people. We got a resounding no from the community. So people are very happy to use open source, not that happy to support it.

**36:02 Chris Romeo:** Yeah, I mean, that's a common challenge though, right? Like Heartbleed did result in some resources being allocated. At least I remember people giving lip service to it. I don't know if it— I don't know. I didn't follow the story for the next year or two to see if people actually came through with money and engineers. The thought was that people were going— that people had committed to provide OpenSSL as a library with some level of resources from inside of companies. And yet you think about the dependence that we have on open source. Like, we talk about it and people just brush it off like, eh, you know, it's just not that big of a deal. Well, Log4j, you saw it, and Heartbleed, you saw it front and center. You have a small group of people that aren't getting paid that are, you you know, giants that are carrying a giant portion of the industry from a software perspective. And like you said, nobody cares about it. And, you know, it's only when we have a vulnerability of the size of Log4j or Heartbleed that people even have visibility to it. Yet, how many other libraries? We could sit here and name 20 libraries that 27 companies are relying upon.

**37:13 Guy Barhart-Magen:** Mm-hmm.

**37:14 Chris Romeo:** every day in their production products that have the same style of Log4j, same type of issue, could be sitting just waiting to be discovered and could be the next Log4j that gets all of our attention.

**37:28 Guy Barhart-Magen:** I agree. And I think that problem is not going to be solved by chasing the next vulnerability every time that a vulnerability happens and paying some lip service or actual service when stuff like that happens. True dedication to going over that and supporting those projects needs to happen. And I think that in a lot of scenarios, it already happens, but behind the scenes. One of the things that I've personally seen in the corporates I worked for, in the large enterprises I've worked for, was that when engineers actually saw an issue, they would probably fix it. It's open source. They have the source code. They usually fix that problem. they engineered a way around the issue. They made some modification to solve that. But due to legal issues, liabilities, licenses, et cetera, they did not have permissions to push that upstream. So even if they already have done the work, they're already aware that there is an issue, there's no mechanism in place like a safe harbor to disclose it or to share it without legal ramifications. Or think about it this way, you're an engineer, you found a bug, you want to fix the bug, you can't do that because now you're—

**38:36** Yeah.

**38:37 Guy Barhart-Magen:** sending IP that belongs to the company, that bug fix upstream to someone who has no legal relationship with your company. So you're barred from doing that. And that's kind of messed up.

**38:48 Chris Romeo:** But yet your company will take the next version of software.

**38:53 Guy Barhart-Magen:** That's not a problem because the open source community is saying, go ahead and take it. Free as in beer. Yeah.

**38:58 Chris Romeo:** For you to use.

**38:59 Guy Barhart-Magen:** Go ahead. Right.

**39:00 Chris Romeo:** Yeah. That's, that is definitely a challenging scenario. And as an industry, we're gonna have to work this out. Like, there's got to be a better way to support these. And I know you've got some different, you know, you got the Open Source Software Foundation, you've got the Linux— there's a number of groups that are coming together that are at least starting to draw some attention to these particular issues. But yeah, I mean, it's definitely some significant challenges. So, Guy, thank you for taking us through kind of that walkthrough of how you were involved in Log4j kind of right on the ground. I think that was really, really interesting to get that take of what did this look like in reality versus just news stories describing what was happening. So when you think about key takeaways or a call to action for our audience, what are the key takeaways from Log4j? And then you get a chance to give our audience homework here? I mean, come on, free chance to hand out homework.

**40:00 Guy Barhart-Magen:** So I think our biggest takeaway for this and the biggest takeaway for our customers and people that we've worked with was that when you have an incident, consult your IR team and see what they can help with. Usually, incident responders have, let's call it, not only security mindset but also out-of-the-box thinking. They will usually understand how attackers think about problems, and they will have different toolsets and different capabilities to try to solve the same problems that you have. For Log4j, for example, one of the big challenges was, give me a list of all the servers that are running Log4j and which version they have on each of them. For an application security guy taking this task, this is a monumental task. How are you— how the hell are you going to run over every server in your organization listing what kind of Java they're running and which one, of Log4j they're having. It's like a monumental scaled-up problem that you don't know how to solve. But for an IR team, this is basically what we do every day. Give me a list of all of the processes running across all of the fleet, all of the endpoints in your enterprise, and run some memory forensics on them to find out what kind of strings they have in memory to locate exactly the hit points that we're looking for. So this is the kind of questions that we get asked every day, and we have the toolsets and we have the capabilities to answer those questions at very large scales. So going back to my first sentence, consult with your IR team. They usually have more capabilities and more ways to think about problems and solve problems than what you have. Collaborate outside your field and you will get new, uh, new ways to think about problems and probably new ways to solve them as well.

**41:37 Chris Romeo:** Very cool. And yeah, I think that, uh, is really good advice. From someone who's working as part of the AppSec team, the incident response folks have experience doing this every day. Like, this is what incident response people get out of bed in the middle of the night. Often it's in the middle of the night. They get called out of bed in the middle of the night.

**41:59 Guy Barhart-Magen:** Usually it's Friday afternoon or later. Yeah.

**42:02 Chris Romeo:** Yeah. I used to do incident response years and years ago. And so, you know, I got more calls at 3:00 AM on Saturday morning than any other time because that's when people would run out of time. They'd try to solve it themselves during the work week, and then it would get to the weekend and everybody would freak out. But yeah, I mean, I think that collaboration is a big part of the takeaway here. The IR team is a group of people that are highly trained in being able to deal with these. You know, we used to call— people used to call the team I was part of the digital— we were digital firefighters. That's what we did. Somebody rang the bell and we came and we put out the fire and we contained it and we figured out what the cause of the fire was. So, you've got folks in incident response that have that experience. If you're a developer or somebody who's working in AppSec, you can tap into that. Don't try to solve it yourself. Bring those people that are experts and they'll probably let you come along for the ride on the fire truck and learn as you're going, but they're going to get there a lot faster than you are. Guy, thank you for sharing this experience with us, and we look forward to having another conversation with you in the future about something else.

**43:10 Guy Barhart-Magen:** I'd love that, and thank you for having me.

**43:12 Chris Romeo:** Definitely. Thank you.

**43:19** Thank you for listening to Security Journeys AppSec Podcast. You can find us on Twitter @AppSecPodcast, on LinkedIn as the Application Security Podcast, or on the web at www.securityjourneys.com. Securityjourney.com/resources/appsecpodcast. Find Chris on Twitter @edgerow and Robert @robertherwatt. Remember, there are many application security paths, but only one destination.

---

Source: https://appsecpodcast.com/guy-barhart-magen-log4j-and-incident-response/