Matt Clapham -- The Technical Debt Ceiling

0:05Chris RomeoThe Application Security Podcast. Here we go. Hey folks, this is Chris, and I am at the Converge conference in Detroit, Michigan right now. And this continues our series of the hallway conversations, and I'm joined today by Matt Clappell. And Matt, this is his second time on the podcast, so we are very glad to have him back again. And so we're gonna talk today about technical debt. When you say technical debt, what is this thing that you're even talking about?

0:57Matt ClaphamTech debt is kind of a broad term, but I like to think of it as stuff you need to address later, right? If you look at some of the security problems we have today in IoT and various other technology stacks, a lot of it is stuff that was old, stuff that had been in there for a while and just kind of left, set once and forget it. And, you know, set it and forget it might work if you're Ron Popeil working on some little thing that cooks a chicken, but it doesn't necessarily work in technology when it rapidly advances so much. And the things that worked yesterday are going to be insecure tomorrow, right? So I've seen this problem kind of pervade as we get more and more into this IoT world. And so I wanted to kind of, you know, highlight that. And I think it comes from areas like dependencies and other things.

1:41Chris RomeoOkay. So let's— I guess the side of IoT and application security, both of those things fascinate and scare me to death. I think about these new companies that are putting IP stacks and adding web interfaces to your refrigerators and things like that. So what are the ramifications of tech debt then? So like, let's look, kind of work through an example. Like, what could happen from an IoT device perspective? What could they forget that you would classify as technical debt that would come back to burn them in the future?

2:15Matt ClaphamIt starts with things like just the third-party components. components, right? It might be the base operating system that's in the device. It might be the well-known libraries. Those things have flaws regularly. People are poking at them, they're finding vulnerabilities, and those things just kind of get left behind. But it also goes further than that, right? There is tech debt and problems that are going to exist in the code that any team writes, not just the third-party components. So I think we, you know, we need to kind of bring it not just in the IoT base space, but also kind of move it forward and start thinking about what does tech debt look like for the apps and the stuff that we're writing that's within the developer's span.

2:57Chris RomeoOkay, so this is a people problem, this is a technology problem, or this is a process problem? Where does tech debt— or D, all of the above?

3:09Matt ClaphamI don't want to blame the people making it because there's There's healthy tech debt, right? There's that trade-off of, well, do I get it done now and get it working and be able to monetize it or do whatever I need to do with it, or do I try to solve all these tech debt problems? So, I don't want to blame development teams because they're trying to get stuff done. That's really important. Shipping is also a feature in this day and age.

3:32Chris RomeoYep.

3:32Matt ClaphamBut also, we've got, I think, somewhat of a process challenge in that we forget about the tech debt problem, both in the bugs that we have that we kind of push off and keep kicking down the road. And I think there's also those bugs then point to a technology problem that also exists. So, I would say mostly all of the above.

3:52Chris RomeoOkay. And I think some of that is dependent on the industry. And, like, so if you're talking about a startup, a startup is going to incur more general technical debt than a large enterprise because they're going to be— they're not as— risk isn't that important to them. They're more thinking, how am I going to survive for the next— if we don't make the next couple releases, we may just be done. We run out of money and we're done. Whereas an enterprise kind of has a different risk management perspective. So how do you see technical debt playing out for the startup versus the enterprise?

4:28Matt ClaphamExcellent question. I think in the startup world, from a tech debt of the components that they're building on top they're probably in relatively good straits. Because if they're choosing a modern cloud provider, modern platform as a service provider, they've got a lot of that stuff kind of— they don't have to worry about it right now.

4:46Chris RomeoGot it.

4:47Matt ClaphamThey might have to worry about it later, but right now it's covered. So, that part of tech debt rather is something they don't need to worry about. But where they really start to incur the tech debt, I think, in maybe the startup world is in the design that they build, their side of the demo. So, the cloud provider does a lot of cool stuff. On their side of the DMARC, the code that they write needs to be relatively well-designed, right? And so, the tech debt that I think a startup is going to incur is going to be in the deltas and the challenges and the bugs and the design faults in what they put together.

5:19Chris RomeoOkay. Okay. So, and then from the enterprise perspective, what's their— How does an enterprise incur technical debt? Is it different than the startup, or is it— I mean, because enterprise, I think of, is going to be more process-heavy, they're going to be more policy-driven, but we know they still have technical debt at the end of the day. So, is it different for a startup versus an enterprise, or are they really the same?

5:48Matt ClaphamFrom my experience with an enterprise, it's going to be a different kind of challenge there in some respects, in that They might have a lot of technology on the other side of it, right? Because they're often, at least traditional enterprise in the past, was responsible for managing all that lower layer stuff.

6:03Chris RomeoOkay.

6:03Matt ClaphamAnd so, where the startup might have to worry about the stuff on top and the faults in their design, the enterprise has got all that and the bag of chips, right? They've got the other half of that and they have to worry about taking care of that platform. Unless they've gone to some sort of methodology where they can push that off to somebody else. But they've got to deal with all that old and crusty base stuff. in addition to the faults and problems and the tech debt on top of what they've built. And pretty much every enterprise is a custom software house to some degree.

6:31Chris RomeoSure.

6:32Matt ClaphamSo, they've got the development things that come along with that in that they might have old designs that haven't been updated for that business app in 10 years, right? And the support for the technology that they put in there, those libraries are way out of support. So, they're incurring the same kind of things that the startup will. In fact, they probably already have it. They just don't realize it or haven't thought about it.

6:56Okay.

6:57Chris RomeoSo, let me ask you this question then. What's the difference between technical debt and security technical debt? Because are we talking in the context of this conversation, when you say technical debt, are you talking about potential third-party libraries that are not updated and have vulnerabilities? So, is technical debt the same thing as security technical debt, or are they 2 different things?

7:21Matt ClaphamI would lump them all into one umbrella because there are things that are not necessarily a security problem but are still technical debt. Maybe you haven't designed for high availability even though you realize later on that it's something you need to do.

7:36Chris RomeoSo, you're crashing under load.

7:38Matt ClaphamYeah.

7:38Chris RomeoToo many users.

7:39Matt ClaphamIt's a design fault. figured out your scale-out, whatever. That's technical debt, right? It's impacting your ability to do what you need to do. Is it necessarily a security problem? It could turn into one if somebody figures out how to leverage it against you, but no, that's different. Security technical debt, I think, is just a flavor of that, right?

7:55Chris RomeoYeah.

7:55Matt ClaphamIt's a subset of not just thinking about the overarching technical debt, but focusing in on, well, why do we care about these particular things? Well, they're the layers, the parts of technical debt that lead to those security vulnerabilities.

8:07Chris RomeoOkay. So, what's the impact then of technical debt?

8:10Matt ClaphamWell, we talked a lot about vulnerabilities. There's also things like it might slow down your release cycle because you've got to go back and fix stuff. You try to do that cool new thing, but the old and busted can't handle it, so you've got to figure out a workaround.

8:26Chris RomeoSo, kind of like a flawed architecture. You're trying to build new cool stuff on top of a flawed architecture.

8:31Matt ClaphamYeah.

8:32Chris RomeoEventually, it's a house of cards. It's going to come from all the other dimensions.

8:35Matt ClaphamAbsolutely. You're going to see maybe a shorter time to life for a product because it's got all that technical debt behind it. Somebody's going to say, oh, you're still using that? Okay, well, yeah, thank you. They're going to give that kind of— it's going to have an unintentional marketing effect, and it's going to affect the lifecycle of the overall product. It might lead to a slower MVP because as you're trying to get that minimum viable product done and get it out the door, you're bumping into some of the tech debt of stuff that you've brought in or security vulnerabilities that you then need to go back and fix because they're found late in the dev cycle or that sort of thing. You're going to see things like greater cost to fix problems, akin to what I mentioned earlier with the slower release cycle, right? When you have a problem that you find post-release, you've got to spend a lot more money trying to work around the technical debt.

9:21Chris RomeoYep.

9:22Matt ClaphamAnd then another thing you might see is something like— I mentioned this, the loss of customers, but Stuff that, you know, you end up stuffed in a niche because of that technical debt.

9:36Chris RomeoYou could—

9:36Matt Claphamyour product could be in that one little corner where the folks who still want to go use that really old thing and don't want to upgrade, they'll be fine. They'll sit on it and you can maybe stay there. But the new stuff that's marching forward, the new integrations, all those cool stuff, it's going to get that attention and that marketing bump.

9:53Chris RomeoYeah.

9:54Matt Claphamis just not going to happen for you, especially when that tech debt is really, really obvious.

9:58Chris RomeoOkay. So, what do we do about it then? So, we've established the problem space here. We have technical debt. I think everybody that's listening is going to, if they work on any type of development project, they're thinking right now in the back of their mind about all the development, all the technical debt that they have. So, what do we do about it though? So, we have to have a solution to this problem.

10:19Matt ClaphamWe need to plan time for removal. Quite frankly, I'm sure people are thinking, wow, yeah, that's pretty straightforward. But it is, right? We need to remind ourselves that we need to spend, I don't know, just pulling a number out of the air, maybe 10% of our time going back and say, hey, what's getting crusty? And it could be stuff that isn't that old, but it's foreseen to cause a problem later on. It's recognized as technical debt, either security technical debt and a specific flavor or just general. And so, we've got You've to do that. got to plan time in the schedule and make resourcing available to handle that.

10:51Chris RomeoSo if you're doing Agile, you may have to set up a particular sprint for rework or some amount of— or dedicate some amount of time to each sprint to say we're going to dedicate 5% of our resources to fixing technical debt or 10%, but you've got to have a strategy.

11:12Matt ClaphamA tech debt that you've identified as part of your project should go on the backlog. In the Agile case, it's yet another work item. And you've got to remember, you have to prioritize it appropriately. So, you know, 10% is my arbitrary number, but it means that you've got to say, okay, great, we've done all these 20 things. We need to make sure 2 of them are tech debt. Even just little tweaks here and there that are going to improve it overall.

11:31Chris RomeoSo, what else do we do?

11:33Matt ClaphamA good mantra a friend of mine said is, get clean, stay clean. Right? So, if you've got a project that's got a lot of technical debt, that could be something like static code analysis. You did some security static code analysis on your codebase and it found a bunch of problems, right? Maybe there's a lot of problems, you can't do it all at once, but, you know, work towards driving to get clean. And then once you have it relatively clean, stay that way.

11:58Chris RomeoYep.

11:59Matt ClaphamYou know, even push it to the next level and say, okay, now that we're clean, we're gonna stay clean by making sure that all new code that gets checked in stays clean. And if you can't write clean code that, you know, passes the static analysis system, then it bumps back, kicks out your change, and you have to go back and fix it.

12:16Chris RomeoYeah. So, I see that as one potential strategy when you're thinking about lines of code that individual developers are writing. It also makes me think from a third-party software perspective, dependency checkers, software that looks at that particular piece of code and looks at a database to see, does it have any vulnerabilities? That's another place where you can follow that get clean, stay clean mantra with your third-party software. Get clean by updating what is vulnerable and then stay clean by having a dependency checker that every time you do a build, it throws an error and says, sorry, you got a vulnerability.

12:50Matt ClaphamYeah. And there are lots of tools available to do that. And that kind of feeds into another mantra: upgrade wisely and often. Now, there's certainly risk in upgrading, especially when things change over time, and the APIs might be different, and not all dependencies are great at making sure they don't break back to pad, right? But at least when you're planning in that upgrade for that dependency or fixing that module that, you know, you wrote that all the other stuff is dependent on, you're putting that effort in, and you're making sure that you're staying at least abreast of the problems. at least somewhere on the current release or current supported release edge of that wave.

13:30Chris RomeoSo I've seen some nightmare cases where products have never had their third-party software updated. So it's like a Linux-based OS, and this product's been shipping for 5, 6, 7 years, and they've never— they're so far, they can't even upgrade anymore. There are no security patches. for the particular version of Linux that they're using. And that's how old it is. So that upgrade wisely and often, I think, is really good advice because if you— it's a smaller batch size. If you're always upgrading and looking and testing, if you make that batch size smaller, it just makes it less likely that you're going to break the whole system by doing an upgrade. Now, if you have one of those products that's been 7 or 8 years and you try to do a full upgrade, Good luck trying to pass your regression test. I mean, everything is probably going to fail because it's all going to break apart. So, anything else that we should be thinking about as far as what we should be doing?

14:23Matt ClaphamI saw a great recommendation online, and I've suggested this for security bugs as well. Have a debt ceiling so that as you're building up this tech debt, right, you're going to have some sort of thing on your backlog. You can use that as a rough rubric or thing to pick from. Your tech debt is going to go up and up and up. You've got all these things. backlog, you know you need to fix, but you keep declining to put them, or there are too many, so you can only do so many. At some point, you are going to hit the ceiling of, hey, if we don't fix these now, it's going to be a big, big, big problem. So set that limit and then say, okay, well, last time we said we are only going to do 10% towards tech debt, but this time we've really got to say, okay, we need to— excuse me. This time we really need to say, okay, we've got to start spending more resourcing, so maybe we kick it up to 20%, 25%, or, you know, pick a number and say, we've got to do it until we can get it back down below that ceiling, right? And then we've got the ability to say, okay, we're back to a comfortable spot.

15:25Chris RomeoYeah.

15:26Matt ClaphamAnd then on the opposite side of that, you could even say, well, we've got a floor, right? We can handle a certain amount of technical debt. We can focus the 80% of our resources on the new the new hotness. And then know that we've got, you know, we can go down to about 20% old and busted before we really have to say, okay, well, you know, we don't have to do it anymore.

15:43Chris RomeoYeah, I like that idea of a floor and a ceiling. I think that's a good way for people to think about this. We can't go any higher than the ceiling, but we have the ability to accept some technical debt that allows us to be innovative and keep things moving quickly. It makes me think of the ceiling, it makes me think of the Gates memo at Microsoft. when they— the trustworthy computing memo where they stopped everybody in 2004. I might be getting my years wrong, but they basically called a halt to any shipping of new features and spent 3 months just clearing up as many security problems as they possibly could. So that— I think that's, you know, they would have never called it a technical debt ceiling, but that's basically the principle that they applied there and were very successful.

16:23Matt ClaphamYeah.

16:25Chris RomeoSo anything else that we need to to do?

16:28Matt ClaphamWell, people say, well, you know, how do I get started in dealing with this, dealing with it? I think a good starting point is some of those third-party stuff. Everybody's got third-party stuff. Dependencies suck, but out-of-date dependencies are suckier. So, take a focus on those. Pick one of the dependencies and update it to the latest cycle, right? Get to that most recent released version of that is still in the support window, that fits with your particular version string. Basically, get up to date.

17:00Chris RomeoYep.

17:00Matt ClaphamAnd then, after that, obviously, retest. But if those updates are relatively small, then the verification should be fairly straightforward. Or, better yet, if everything's automated, then it's just a matter of updating some stuff, rerunning the verification tests, making sure stuff's still working. It passes the smoke test, it passes verification. We're good.

17:21Chris RomeoI think this is actually easier for people that are operating in a DevOps world now because they're already focused in on a small batch size.

17:27Matt ClaphamMm-hmm.

17:28Chris RomeoI think where this is more difficult is in the waterfall, and there is still a lot of waterfall out there. I'm shocked with how much time— many times I see waterfall still as a primary methodology. Waterfall definitely is going to make it harder because the release cycles are longer and you don't have as many touch points like you do in a DevOps world. But I think it's workable from both perspectives.

17:52Matt ClaphamYeah. And then, you know, if it breaks something, I plan to fix it. And so if it's an easy fix, tweak it, get it done. If it's a real big and complicated one, it breaks a lot of stuff, okay, well then now you know that goes on the tech debt backlog. Yeah, that's a big problem. We really got to go back and fix it. It's too much for this release, so maybe we can roll back knowing that, okay, well, this is a top priority, or this is a big thing that we're going to have to So when we hit our tech debt ceiling, we know we've got a big chunk to deal with.

18:19Chris RomeoHave you seen any— just kind of wrapping this up, and I guess, what would you say that in conclusion, what should a team's goal be about tech debt?

18:33Matt ClaphamWell, if you look at where the rest of the industry is going, even in more traditional waterfall models like the operating system, Times are changing, man, and we're going to rapid update cycles.

18:43Chris RomeoYeah.

18:44Matt ClaphamWindows 10 is supposedly a— in fact, is being pitched as, and if you look at the upgrade cycle, it actually is being delivered as this routine, hey, once every 6 months, once every year, it's going to get a new version. So, they're going through updates. They're going through incremental upgrades. They're actually increasing your tech debt underneath of you, right? And if you look at the way updates for that platform are coming out, the platform itself is is getting a monolithic patch. So again, there's a bunch of stuff that's going to happen beneath, just at the operating system level, that is beyond your span of control.

19:16Chris RomeoOkay. Is there any resources that you can think of? I guess, what's it called? Well, first, what's— let's do resources first. If somebody wanted to dive deeper into this topic, is there any place you've seen people talking about this? Like, is it happening? Is there conversations on Twitter? Or is there— Any good blogs or anything that you've seen?

19:37Matt ClaphamCertainly a search on tech debt, the good, the bad, the ugly. You're going to see a couple of great articles. When I was doing research for this topic today, that's where I got the idea for— I'd seen it in one of those blogs. Sorry, I don't remember which one, but that's where they showed the sawtooth wave that talks about the debt and the ceiling and the floor. To get started though, it's really just, you know, taking an introspective look at your own project and saying, well, what areas might we have tech debt in? Maybe running some free tools or some handy available tools to look at the state of maybe the base operating system, if your project actually includes that. There's plenty of tools that will scan the base operating system and tell you what you got, out-of-date modules, vulnerabilities, that kind of stuff.

20:23Chris RomeoMm-hmm.

20:24Matt ClaphamAnd then similarly, there's going to be more and more tools on the market or regularly available that will look at the libraries on top of that. And some of them are even getting smart enough to look at the source code, not just the binaries. So we can say, hey, you took that source code from that open source library and plugged it in as a source code module and compiled it yourself.

20:42Chris RomeoYeah.

20:42Matt ClaphamWell, the tools are getting to be available that will look at that and tell you. So really picking a few targets, just something easy, and start there and get a sense of, okay, it's not as bad as it seems when you look at this big tech debt wall and your eyes glaze over and you don't want to approach it.

21:00Chris RomeoWell, hey, thanks very much for your time here. I think that was a good introduction to this topic. I hope we get some people thinking and approaching— you know, I take a very programmatic approach to application security, and I think this is a great topic to include within that kind of approach to culture because technical security technical debt is just another piece of how you approach security from a cultural perspective.

21:24Matt ClaphamDefinitely. Thank you very much for your time today. Well, thank you.

21:32Thanks for listening to the Application Security Podcast. Our intro music is 8-Bit Kung Fu by Boring and TJ, and the outro is Southern Delight by Stefan Kartenberg. You can find us on Twitter @AppSecPodcast or on the web at www.appsecpodcast.org.