Neil Matatall -- AppSec at Scale

0:00Chris RomeoNeil Matatall is an engineer with a background in security. He's previously worked at GitHub and Twitter and is a co-founder of the LocoMoco Product Security Conference. Neil joins us for his second visit to the podcast to discuss account security at scale. He describes the underlying principles behind security at scale, how he worked to build a sign-in analysis feature, and how attacks were detected with that feature. We ended the conversation with an authentication lightning round, with Neil responding to various statements, true or untrue, about authentication completely off the cuff. We hope you enjoy this episode with Neil Matatall.

0:39Neil MatatallYou're about to listen to AppSec Podcast.

0:43Chris RomeoWhen you're done with this, be sure to check out our other show, High Five. Hey folks, welcome to another episode of the Application Security Podcast. This is Chris Romeo, CEO of Security Journey and co-host of said podcast. I'm also joined by Robert, who appears to be dressed for the occasion. I know, right? Robert, are you in like a cold area or something? Where are you?

1:05Robert HurlbutWell, we're getting some snow here this weekend, so yeah, it's been very cold. And Robert Hurlbut. And good to be here as well. Threat modeling architect and really looking forward to our conversation again with a new guest, or actually a returning guest.

1:22Chris RomeoReturning guest who is in a place where they don't know what snow actually is, but at least I don't think they do. Maybe one time it's Node. But yeah, Neil had a chance to be with us in a previous conversation. We'll put a link to that in the show notes so you can go back and listen to the first conversation. But today, we're going to talk about account security. And since Neil's been here before, you can listen to the other episode for his origin story, but he's going to tell us a different story as a way to get started in this. And so, Neil, I understand you were at GitHub, which when I always think about like companies that are doing security and doing it at scale with a lot of different, you know, a lot of users, a lot of load and stuff that's going on there. I'd love for you to just give us kind of the background story on account security and what you were dealing with at GitHub.

2:13Neil MatatallSure. Just one quick clarification, and this is not a well-known fact, is Hawaii very much knows about snow. We have a mountain called Mauna Kea, which literally means White Mountain because it's frequently covered with snow just because of, you know, altitude. Fun little fact there.

2:29Chris RomeoVery cool.

2:29Neil MatatallNot a little correction, but definitely not intuitive. But anyways, yeah, I really enjoyed my time at GitHub. I think the time where I was working on this account security story was probably the best part of my entire career. And I was really glad that I got to do it. So, and that kind of is a little hint at the culture of GitHub back then, where ownership of things was very nebulous. And if you wanted to work on anything in the product, you could. I had just joined sort of as they had introduced managers, as they had introduced teams, as opposed to just like everybody does everything. So there was officially a security team. And then eventually, you know, we were getting good at like stopping XSS. We were getting good at stopping SQL injection. We had a pretty good story around kind of like the classical AppSec problems. But we had a pretty big problem with our authentication stack. Not that it had like vulnerabilities in it that could be bypassed, but it wasn't doing a whole lot to protect people. Now, the classic response is use 2-factor authentication, right? Like that, that'll make your account secure. It'll make, you know, so if your password is leaked, it won't be used. But telling everyone to use 2-factor authentication for a lot of products, not all, but probably most, is not a realistic scenario. A lot of people think of GitHub as having a very technical user base. And I do think that that is more true than most places, but we have plenty of people who are not technically savvy using GitHub every day. Maybe they're not writing code, but they're still using GitHub. And again, like, just use 2FA totally dismisses how difficult 2-factor authentication actually is. I think people that would be listening to this podcast might have a different opinion about the difficulty of using 2FA since it's more likely that we are using that on all the services we use. But Telling everyone to use 2FA is just not practical today. And just to give you, like, an exact number, GitHub's technical user base still only enrolled about 15% in 2-factor authentication. So the vast majority of people were not using 2-factor. And we're pretty confident in that number too, because we limited it to, you know, accounts of a certain age that have been active in a certain period. So not just like these bot or spam accounts or accounts that just sign up and never use the thing again. So it's a pretty accurate number, in our opinion. So we had to think like, okay, 2FA is not the answer. Where do we go from here?

4:58Chris RomeoYeah, that's a good reminder. You're— I often forget this, and I'm sure a lot of people that are listening to this forget, to your point, that we just take 2FA, MFA.

5:08Neil MatatallWe just—

5:09Chris Romeoof course we do that. I mean, but we're security people. We're not the normal user profile. And so that's a great reminder right off the start of this conversation that our users are not as security savvy as we are. And if you just assume that they're going to embrace whatever security control or security technology or things that you're going to put in front of them, you're opening your system up to a whole plethora of threats that, you know, there's other things that you can do. And so the system's got to protect the users, especially in those non-security-savvy users. Yeah.

5:45Neil MatatallSo, yeah.

5:45Chris RomeoSo any kind of other thoughts on that from that perspective about the user experience there?

5:50Neil MatatallYeah, I mean, it has to be perfect or else it'll be terrible. You know, we even had a big problem with just the way we did 2FA setup where people would kind of abandon the process halfway through, and then they'd be left in this sort of state where they had like downloaded their new codes, but they weren't actually valid. And since they had overwritten their previous installation, they just got locked out of their account. And there's just so many ways it can go wrong, and human beings do various things. Some people share phones together. Some people just use a burner phone. Some people just seem to reformat their phone on a weekly basis and lose credentials all the time. And we can't tell people to stop doing this. So we have to make their life easier. I think GitHub did a lot of things and they're still doing things today. I think, I don't know if you saw, they recently released push-based authentication if you have the GitHub mobile app installed. You know, like, that's not as strong as WebAuthn, but it's certainly convenient. And if you have trouble retaining your OTP, your one-time password codes in your application, for example, this is a great backup. And I can talk a little bit more about that in detail if you want. But there was a lot of thought put into the backup scenario because, like, recovery is more important than, like, having the ability to use 2FA.

7:11Chris RomeoYeah, I think we'll unpack that as we go, as we get a little deeper here. But I'm still trying— I'm still pondering that quote you said, if it's not perfect, it's terrible? Is that the—

7:20Neil MatatallYeah, it's just, I mean, well, I guess, I guess, did I say that? I don't remember saying that.

7:27Robert HurlbutSomething like that.

7:28Chris RomeoSomething like, well, I was going to borrow it and use it as the name of my third book, right? It's not perfect, it's terrible.

7:32Neil MatatallWell, 2FA is not perfect, and therefore it is terrible in the context of securing all your users. How about that?

7:38Robert HurlbutWell, my take on it was that you were saying that, you know, your customers, they want to feel like They want to be successful, right? They want to feel like, oh, you give me something to do, okay, I want to be successful at doing what you're asking me to do. But if it's terrible, they will feel like they failed, but they won't look at themselves as failures. They will look at the company that's imposing this as failing them, if that makes sense. And so that's how I took it. I don't know if that was your intent or in terms of trying to get it to be perfect.

8:11Neil MatatallOtherwise, It's bad. Well, it's also like so many people have been turned off of 2FA because of some bad experience they had somewhere. You know, like some people will just, you know, never use it again. And, you know, even speaking to the security group, like I've seen people in the security industries where the Twitter account got popped and it's like, well, you didn't have 2FA. It's just a terrible experience. And I do think things like WebAuthn are going to help in this sense. scenario, but I think we're still years and years away from ubiquity.

8:41Chris RomeoBefore we go to the next question, I know Robert's got another question, but we've mentioned WebAuthn 2 times. Let's get a definition in front of everybody as far as what do you mean when you say WebAuthn?

8:51Neil MatatallSo, the primary protection that I'm referring to when I talk about WebAuthn is that you have a credential that is bound to an origin that is enforced by the browser. So, You know, with 2FA, you get a text message or you have an app, you enter a code, but there's nothing protecting you from typing that code into a third-party site, like a phishing site. And 2FA does not protect you from phishing. It just makes an extra step in the phishing process. But if you were to have a WebAuthn credential, which could be a security key, like FIDO key, or YubiKeys are very, very common, but also nowadays, your web browser can act as a credential. So Face ID, for example, can be an origin-bound credential that you can use to log into GitHub, for example. And it's phishing-proof. There's just absolutely no way it's ever going to happen. The system was designed very, very well. It's never going to happen is always a stupid thing to say, but I'm saying this is the strongest protections we have available to us today. And barring a bug in the specs or the implementations of these, honestly, I think the browsers are going to take over the share of the the WebAuthn landscape just because everybody has them in their pockets at this point. And it's just a matter of, like, I think Android support is a little bit off. And, you know, even with Windows Hello, you can do it as well. And these systems are very cryptographically strong and fairly usable.

10:18Robert HurlbutYou've sort of, I think, spoken to this in terms of, you know, making sure that customers feel good about their experience and so forth. But what were some of the, other underlying principles that GitHub was trying to apply or think through as they made decisions about how to improve some of these ways of authenticating users?

10:39Neil MatatallSo we always had to balance friction with, I guess, return. You know, we could sit here and have the most draconian sign-in process with 19 steps, and you have to, like, mail in a verification letter. And then no one would ever want to use GitHub. I've been in situations where things like first page render or first click to action were so important that you could never even consider things like friction. But GitHub, I think, definitely was on the— we erred on the side of a little bit of friction is okay, but let's not do this just because it's friction. Let's think about this and let's try to not just do the base thing and really try to focus on the experience of everyone, which leads me to our first principle, which actually seems a little bit counterintuitive based on what I just said, but we didn't want to make a lot of security choices. We just wanted to make you do the thing or not do the thing. You can opt into 2FA, but that's pretty much the only bit of security that you can opt into on GitHub because we just do it. Also, things like CAPTCHAs have been historically banned at companies I've been at because people think that they're just a horrible user experience. They're not going to slow down someone using some sort of like Turk system to just manually brute force all these things. But they're pretty darn good against attacking, preventing lazy attacks. And lazy attacks are incredibly successful if you're not doing anything about them. And yeah, like I said, GitHub wasn't doing anything about them. So underlying principle number 1 is we're just going to do the right thing for everybody and we're not going to ask them to opt into anything. Because we didn't give anyone the choice, like I said, we really had to focus on the experience. And so principle number 2 was we can't overload our support team with hate mail. So if people are getting very upset with the things we're doing, we should react to that. And we should listen to them and we should respond to them. We should still do what we think is best. And every once in a while, there were some kind of hard decisions that led to special situations around, like, for example, us responding to the support tickets because the support team did get overwhelmed at one point with one of the changes we made. So think about how it's going to affect our support team as much as it's going to affect the people using the product. So, you know, don't give them the choice, but definitely listen to them.

13:13Chris RomeoSo how did you build this sign-in analysis piece? Because when I think about GitHub, it's not like 5 users a second or something are coming into GitHub. Like, this is a monumental, you know, system that's delivering. And sure, you're not authenticating people on every request or anything like that, but it's still— it's a level of scale above what a lot of people have ever even experienced. So how did you build this analysis system to be able to determine if something bad was happening?

13:44Neil MatatallYeah, I think it's a pretty interesting thing because I think it's definitely something that's repeatable in any size organization. We did run into some issues with the scaling of some of these things. And I, you know, I did bring GitHub Sign-In down more than once in the process of doing this, but it's actually very, like, rudimentary. There's no AI involved. There's no, like, tuning of models involved. Very, very basic system. And it's kind of the foundation. It originally came from the idea of like, okay, I work out of the US. I don't really use a VPN, but I have. And my VPN is, you know, spits me out in France or something. If I sign in from South Africa, like that's probably not normal. So the idea was to send an email, sort of a reactive process, just like, hey, like you might want to review the sign-in. It was kind of the first step we took. And just to do that, all it took was every time you sign in, we just create a database record. It tracks the IP address, what we call a device ID cookie, and it's tied to your user session, which is also like a database object in our infrastructure. And that works great unless you work in Europe where you might sign in from Spain, go work in France, and travel through some— well, that's a bad example because there's no country in between them, but a typical person in Europe might travel through multiple countries a day. And borders are a little bit of an imperfect science, but they're an okay science in that regard. So if you would change IP addresses, we would create another record of that too. And again, it would be all tied to the session. And you can actually see this in the GitHub UI if you like drill down into an individual session that has been to multiple countries. We have a very poorly drawn map, but it is the map that we use in the product. So you can geographically see where your session has been. So again, like if you just happen to sign in on a specific spot to where your train was that day, even though you go through that path every single day, we don't want to tell you about that. And I think that product was really well received. Most people are pretty appreciative of it. And because we had put some thought into like the different use cases of people who do cross borders, it didn't seem to really upset anyone. No one was really like, oh, these notifications are low quality, you know, why doesn't GitHub know I come here all the time sort of deal. So, you know, we had to put a little bit of thought into abuse. You know, we put a limit to the number of individual records that can be tied to an individual session because we had bots that are just going over hundreds of thousands of IPs every second. And that was a bit of a problem. You know, we also have people who will just sign in incessantly millions of times a day. And so eventually, like, if you're signing in from a known location, we just don't really need to track that session at the time. And so from there, I might be skipping ahead a little bit here, but I think this is all related here. I mentioned the device ID cookie, as well. This is a unique value that's generated for every browser session. It sticks around forever. You know, if you have multiple users sharing the same computer, they will all share that device ID. Obviously, if you come from an incognito browser, it's a fresh cookie every time.

16:54Chris RomeoOkay.

16:55Neil MatatallBut most people actually don't come in from a fresh browser from what we saw. Like, most people will sign in on the same computer multiple times, and they won't clear their cookies. There are the people who use incognito every single day. So, but they are very much in the drastic minority. I want to say, like, something like at least 60% of sign-ins would come in from a known device. And so we thought, like, okay, 60% of sign-ins coming from a known device, but that still means 40% come from an unknown device. Like, do we really want to send out emails for 40% of sign-ins? Like, that's not going to be a very good user experience. And so we thought, like, well, what if we just, you know, add some friction in that case? And so the first thing we jumped to was email-based sign-in challenges. So if you're going to sign in from an unknown device, we'll send you an email with a code that you then have to enter into the browser to sign in. Very common implementation across the internet. You know, certainly, You see this even at some of the smaller companies too. I do think it's becoming a standard. I've never read the NIST standards. I wouldn't be surprised if email-based challenges based on suspicious logins are just a standard now. But that was the most impactful thing we ever did, but it was also the hardest thing we ever did. I did say that the incognito people are the minority, but there are still millions of them. And if you're getting an email every time you sign in from the same IP address, that's really, really annoying. So we kind of said, okay, if we don't recognize the device, but we recognize the IP, you signed in here, like you signed in from here before, not just you've been in here before, we will allow you to bypass that. And that was most of the— we had to allow a few other things. Like, there were a few other criteria that would allow you to bypass that because that was only in the transition.

18:44Robert HurlbutYeah.

18:45Neil MatatallA year or so after we had implemented this, we— very basic rules, you know, if it's not from a device or an IP that we recognize, you're going to get an email challenge. And looking that stuff up, and that was all inline, that was not something that was out of band. That was like, as soon as you hit login, we're going to run all this analysis and check. So it had to be incredibly fast. And we went through a few iterations of tuning it and the way we were querying the data. But eventually, it worked out pretty well, and the performance impact was just negligible at best.

19:17Chris RomeoHmm.

19:19Neil MatatallSo that really helped prevent the password-spraying attacks, you know, which— so before we put in this device challenge system, we had all this data. We're just watching mass account takeover just happen. Now, is every anomalous sign-in an account takeover situation? Absolutely not. But when you see that number spike, like 10x for hours on end, that's not something normal. That's basically someone taking a password dump from a third-party site and trying it on GitHub and being incredibly successful. And thankfully, our system was able to stand up to that increase in activity. So we could just see these mass account takeovers happening just on the regular. But as soon as we put this device verification in place, among other things we did, as well, virtually eliminated it via the web.

20:11Robert HurlbutYeah.

20:14Chris RomeoThe thing I love that you're bringing out in this conversation is the fact that you guys didn't just come to this problem going, hey, we're security and we're going to do these various things about authentication the way that the requirements say we have to. And like you said, the NIST standard, we're going to grab the NIST standard and we're all going to implement this. You step back and you said, Hey, let's build usable security. Let's focus in on not disrupting, not making the users shake their fist and say, security people! You know? And we still see that, even in this modern day, where there isn't that collaboration between dev and security, where there's like, hey, let's work together to find the best way to serve our users so that they don't hate us because of a security feature we're trying to put in place. And that's all— as you're describing this, I'm like, that is exactly how you guys were doing this here, is you were putting the user first and ensuring that you balanced how much extra effort they were going to have to put in to maintain their security. And so I'm guessing you didn't hear from a lot of people who were screaming very mad about, you know, having to do device verifications and stuff like that. But I'm sure there was always a percentage. But I'm going to guess it wasn't 20 or 30% of people that were saying it. People were—

21:29Robert HurlbutYeah.

21:29Chris Romeoyou know, just absorbing it because most people were only going to hit it once, you know, once or twice in a couple of month period.

21:35Neil MatatallYeah. I think it was on average maybe something like when the dust settled and things were in a steady state, I think it was maybe like 1 in every 3 sign-ins was challenged. And if you had 2FA, you could bypass this entirely. But yeah, it really was, I think, like a, you know, a feel-good example of like, This wasn't the security team telling someone else to do something. This was the security team doing something in partnership with support. So I was talking with support daily. Every time we wanted to make a change, we say, like, you know, based on your hunch, like, do you think this is going to be a rough change? And the allowance for the incognito users that I mentioned, that was not in our original plan. Like, we were going to put our hard foot down and say, like, you know, If you're an incognito user, like you're signing in every day, you're probably okay doing 2-factor or something, you know, like you can bypass this with SMS is way faster than email, by the way. So maybe you should consider that. Not that everyone should give out their phone numbers, but yeah, we had to acquiesce and we saw the support tickets drop back to a reasonable level. And people will say very mean things on the internet and they would say very mean things to support. And there were a lot of people that were very upset about this move. But I think actually a different change that was also part of the story, too, actually upset people even more. And that was when we started banning compromised passwords from, like, Have I Been Pwned, for example. We started off with Have I Been Pwned. And again, because we have all this telemetry, it's like, whoa, look, there's a strange correlation between a high number of anomalous sign-ins And sign-ins that seem to be using passwords in this Have I Been Pwned dataset. I think there's a lot of like, oh, if you ban 500 million passwords, what if it's the 500 millionth and one? Do you go 600 million? Do you keep going? And the answer is yes. If you're using an actually randomly generated password that's strong enough, it's not going to be in this dataset. And we've since integrated other datasets that I believe get their information from the FBI. But anyways, crazy, very successful. While the device verification stuff was more impactful, we still saw a big change, a big amount of, you know, what could have been malicious sign-ins blocked. And this is another story about usability and balancing security, too. Our initial thought was— well, I don't remember what our initial thought was, but the initial rollout is, we're not going to block people from signing in with these passwords. We're just going to let them know about it. And well, how do you let them know about it? Do you send them an email? Well, hell, like, no, I don't think that's the best way to do it. How about a banner at the top of the page? I said, okay, that's reasonable. Now, should we allow people to dismiss this banner? Hmm, I don't know. So we decided not to. We make this banner big, red, and obnoxious so that people will want to change it. I think that's a good idea. So we got the hate mail of, why are you broadcasting to my coworkers and everyone sitting around me that I have a bad password? This is really embarrassing. You know, how dare you? And then we respond with, you know, like, well, this is what happened. Some people would say, like, I use this password everywhere. How dare you make me change it? I've used this for 20 years.

24:56Robert HurlbutWhy do I need to change it now?

24:59Neil MatatallExactly. Exactly. Or, or, you know, there's some people straight up gave us their passwords in email, which was ridiculous. Some people would, you know, refuse to admit that it was used anywhere else. And, and some people were like, you know, the passwords I generate in my head are super strong. And it's like, well then how do we know it? Um, you know, it was a little, it was a little tough. Like I definitely had to kind of like make, turn on my empathy power, like to the max when I was reading these things. 'Cause it was definitely like a little bit.

25:25Chris RomeoYeah.

25:26Neil Matatalla little bit difficult for me. I'm not really used to that kind of exposure.

25:29Robert HurlbutSure.

25:29Neil MatatallBut, you know, like some of it was valid. So some people, like our original message was something like, your password sucks. And that was like, well, that's not really accurate. It's more that your password was found on a third-party site.

25:40Robert HurlbutYeah.

25:41Neil MatatallAnd then we had to tweak the language some more to say like, no, like we aren't talking to any third-party sites. We're not sharing your password with other people. We didn't use the Have I Been Pwned API. We pulled down the data and ingested it and queried it through a database. So there was no privacy concerns at all. But it was really hard to convey that in a universally understood way because GitHub site is only in English and that's not our user base.

26:05Robert HurlbutHmm.

26:08Neil MatatallSo we iterated on it and we got to a place where we seemed to get less hate mail. But we were still seeing people's accounts get popped through these methods. And this was actually before the device verification codes. So this was while account takeovers were still pretty rampant. We decided to make the decision, like, you can no longer use a password we consider compromised on the site. If you try to log in, we'll force you to change it. And again, that sent in another round of hate mail, but you know what? It really helped get our users into a better state. So when we started ingesting the new data feeds from the FBI sources, we had to also think about that. If your password was good yesterday and it's no longer good today, and we're forcing you to change your password today, that's inconvenient. So if your password was in the new dataset, we set basically like a 30-day countdown where you still get the banner, you can still use the site, but after that 30 days, we're going to force you to do the reset. And that was really, I think, a crucial step in making this much less painful because there was much less surprise. People had plenty of time to take action upon it. And unless you're a bot that can't read, you did take action on it.

27:19Robert HurlbutSo you've talked about quite a few ways of detecting the different attacks and so forth, but did you see any trends or shifts, and did that change the strategy in some ways?

27:32Neil MatatallYeah, I definitely kind of hinted at that in saying that it would stop password spraying attacks against the website, right? But GitHub has an API, GitHub has a Git interface, And both of those accepted passwords at the time.

27:48Robert HurlbutHmm.

27:49Neil MatatallGiven the success with things like device verification and the compromised password ingestion, we had to make a decision. What do we do for the API? Do we ban bad passwords first, or do we just go straight to no passwords at all? And that was— that took me, I think, almost 2 years to convince people it was the right thing to do. 2 years ago, I wanted to get rid of passwords on the API before these projects even start. 2 years later, when we're in the middle of all this and we come to this decision, it was really like, it's time to just get rid of passwords in the API and not even have this intermediate step, not even have to think about notifications because humans don't see API responses for the most part. So how do we put a banner in an API response? Huh. So yeah, we sent out communications. We did— we would email people to let them know that they were using a deprecated form of authentication. We had 2 brownouts. So we just temporarily turned off support in advance of actually turning it off to give people sort of a simulation of what would've happened. We did it, you know, so one would happen in, you know, the Europe workday and one would happen in the US workday. Yeah. So the reason we did that is because as soon as we fixed the web, we saw all the traffic just shift to the API. And it's like, oh, look, there's a ton of successful API calls at an anomalous level at a sustained rate for a couple hours. I know what that looks like. It literally, like, the translation was almost exact. And if you think about it, it's way more efficient to test passwords in an API because you're not rendering a whole page, you're not pulling down all that HTML.

29:27Robert HurlbutSo fast.

29:28Neil MatatallLike, that's, that's where I would have been doing it from day one. So that deprecation period was very long because it was disruptive. But in the end, when we turned it off, it was probably our least controversial change ever. We got no hate mail for the most part. The internet mostly celebrated this as a good thing. And it felt very good. So we had all the confidence to just do the same exact thing for the Git interface. The Git interface was definitely less of a threat because it's It's a little bit harder to test credentials via Git, but it was still an avenue that someone could use a password dump to try out. And same thing, we were very scared. We had a long deprecation period, lots of communication, turned it off, and nobody cared. And so at that point, you know, a leaked password was not very useful.

30:17Chris RomeoSo using the API example, let's play this one out a little bit because I think it would be beneficial for our listeners to understand So with the API GitHub, you banned passwords. What did you replace it with and why did you replace it with that? And how is that new thing secure?

30:35Neil MatatallOh, yes. So GitHub offered 2 ways to authenticate to the API, 3 ways for Git, but for the API specifically, you could use a password or you could use what's called a personal access token. And if you're an OAuth app, you can use an access token there too. The personal access tokens are, you know, 160 bits randomly generated, securely randomly generated. It might not be stronger than your password, but it's stronger than practically anyone's password. It looks like a SHA-1, so it's very kind of convenient. I think it actually is. And yeah, the idea is that these 2 credentials are interchangeable for the most part. For, for funny, kind of funny historical reasons, GitHub actually had an API that required passwords so that you could create personal access tokens. And as soon as I learned about that, I definitely said that does not seem like a good idea at all. Using a credential to create more credentials is kind of a little bit scary in that sense. But if we're going to remove password support from the API, that means that API is going away too. So there was a couple cases where we definitely had to, like, just remove support for certain APIs because we couldn't support it. There were some that kind of had to be re-architected. There's some that kind of required introduction of what's called OAuth device verification, which is a little bit more like authorizing a client versus, like, authorizing an application. So all these things had to be done in support of it because we couldn't just, like, turn things off without offering alternative solutions. Now, a personal access token, if that gets leaked, that's a pretty darn powerful thing depending on the scopes that have been granted to it. I guess that's another difference is that a password can't have scopes. It's everything. An OAuth token or personal access token can be limited to what it can do. So that's very powerful in itself. And GitHub also recently released support for adding expiration dates to these personal access tokens. So, you know, if you don't just leave it on some server, 10 years later, it's still valid.

32:39Chris RomeoSo the— when we summarize kind of the value of the PAT, the personal access token, it is the fact that it's generated using a strong cryptographic function. So there is a certain amount of entropy or randomness in the token itself. So the user isn't able to set it themselves. They can't put password1234 or something. The system generates a strong cryptographic artifact. You still have some of the same challenges on the other side of the transaction where that if you, if you don't protect the PAT, for example, in a container, if you expose it in a running Docker container as an environment variable or something, you could potentially have some, you know, some of the same challenges you would have with a password. But it gets you like maybe 60 or 70% of the way better than password. I don't think I would say it's 100% better because I've got the other side, not the GitHub side, but the client side. still has to deal with it as a credential. But it is definitely better than letting the user create a credential and then potentially create something that's weak.

33:42Neil MatatallYeah, I do think if this account security project kind of kept going in the direction it was going, I think we would have, you know, things like third-party OAuth PAT compromises were kind of like the next big thing that we wanted to solve. So if you go into an integration and just make some, you know, you give a PAT to a third party, You know, they might lose it, they might give it to somebody else. These things are kind of very powerful. And a PAT's also better than a password in the way that if you accidentally commit it to a GitHub repo and it's public, we'll revoke it immediately. So we would scan all the incoming commits for things that looked like personal access tokens. And because we can actually verify what they are, we can decide to revoke them or not. which was very convenient. And that's a service they offer to other providers as well. But obviously, like, I can't tell if a Slack token is valid. I can just tell you if it looks like a Slack token.

34:32Chris RomeoNo, that's neat that GitHub detects that as they come in in a commit and then immediately shuts them off. Like, that's security behind the scenes, making the world a bit more secure place. And there's nothing— I didn't have to take an action. I didn't have to like, oh, let me just push the button. Like, it just happened behind the scenes. Well, we got to kind of come to a conclusion in our conversation here. So we're going to do a little lightning round. It's like maybe the second time we've ever done a lightning round. Last time we tried it, the lightning round went for a few minutes per answer. So this is a bit of an adventure, but we've got a couple of statements about authentication. And so I'm going to throw these out. I'm going to read these out, and we're going to give you— you're going to have to give us like 15 or 20 seconds, like right to the point. Like, how are you going to refute this statement? Or maybe you agree with it.

35:17Neil MatatallI don't know.

35:18Chris RomeoWell, let's give it a shot and have some fun here. So the first one says, SMS 2FA is not secure.

35:22Neil Matatall2FA over SMS is secure enough for most people, but not every application. Your threat model matters. Normal people can use SMS. Okay.

35:33Chris RomeoWow. That's the best lightning-round answer I've ever heard in the history of the Application Security Podcast. The next one says, WebAuthn is for everyone.

35:41Neil MatatallWebAuthn used to be very inaccessible, but nowadays with things like your iOS browser and your Windows Hello and other systems making this free and easy and accessible, that is going to change in 2 years. Okay.

35:56Chris RomeoRequiring a phone number backup is bad for privacy.

35:58Neil MatatallIt's so good for recovery though. And recovery is such a big problem. I personally, it's okay for me.

36:07Robert HurlbutOkay.

36:09Chris RomeoAnd then we have storing OTP or one-time password codes in 1Password isn't 2FA.

36:18Neil MatatallOh, that's kind of a tougher one. It is 2FA. Even though the credentials are in one place, they're still doing 2 separate things. 1Password had a really good write-up on this, and I definitely agreed with their opinion, but I'm having hard trouble stating it right now, though.

36:33Chris RomeoEspecially in a lightning round where you only have like 5 seconds of, you know, the pressure is— you're in the pressure cooker here. So final one, not requiring 2FA to disable 2FA is always wrong.

36:46Neil MatatallI got an opinion on this, but I'm gonna let you go. So one thing— this is— sorry, this has to be a longer answer. But one thing that we talked about at GitHub was, if you have a session, you're logged in, you're able to disable 2FA without having to do a 2FA challenge. We had discussed adding a 2FA challenge in the past. And based on the implementation, it was actually more difficult than we originally thought. So we kind of scrapped the idea for a little while. And then we actually thought about, well, this is actually really good for recovery. Like, If I lost my phone and I have a session somewhere and support can tell me about that, and I can realize that I'm logged in on my iPad and disable 2FA, that's really convenient for me. We actually never got any data on how often that was used, but we even considered productizing to the point where it's like, hey, like, we noticed you're having trouble signing in. You're logged in here. Would you like to approve that sign-in on that other machine? But that felt a little bit weird. It was like, if you need to disable 2FA and you have a session, you can do it there. Now, if someone has access to your machine and they're able to disable 2FA, I think you have bigger problems.

37:44Chris RomeoYeah, I was going to argue it the other way, but I'll go with your answer here because you focused on— it's user-focused security has kind of been the angle that we've been discussing here. And my answer of just saying, of course they have to do 2FA to disable 2FA, is maybe more of a classic security answer that makes it harder for the user to be successful using the security features. So, Neil, we thank you again for your second I'm definitely brainstorming a new show called The Lightning Round with Neil. That's just a working title. I'm working through it a little bit, kind of brainstorming, kind of creatively working on it. But yeah, I appreciate your history at GitHub with account security and the insights that you were willing to share with us. But then also just knocking out some of those things that people throw out like they're just all truths about SMS and WebAuthn and stuff.

38:35Robert HurlbutYeah.

38:35Chris RomeoIt was great. So thanks for taking the time and we'll look forward to a future conversation. We'll find something else cool to talk about. We didn't even get to security at scale, which is something else we want to talk about. So maybe in the future. But Neil, thanks for your time today.

38:47Neil MatatallThanks for having me. Really enjoyed it.

38:49Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast and on the web at www.securityjourney.com/resources/podcast. You can also find Chris on Twitter @edgeroute and Robert @roberthurlbut. Remember, with application security, there are many paths, but only one destination.