Dr. Anita D’Amico -- Do certain types of developers or teams write more secure code?

0:00Chris RomeoDr. Anita D’Amico is the CEO of CodeDx, which provides application security orchestration and correlation solutions to industry and government. Her roots are in experimental psychology and human factors. Her attention's now focused on enhancing the decisions and work processes of software developers and AppSec analysts to help make code more secure. Anita joins us to discuss research she's done answering the question, Do certain types of developers or teams write more secure code? Being a security culture fanatic, this topic is near and dear for me. We hope you enjoy this conversation with Dr. Anita D’Amico. You cannot hack yourself secure. Everyone wants to focus on the offensive side of the equation. The challenge is that developers get bored with hacking broken pieces of code after a while. Sure, it's a shiny, cool new thing in the beginning, but how about one year later? At Security Journey, we focus on long-term, sustainable security culture with the developers as defenders. Our approach integrates experimentation together with learning. We believe that developers need hands-on experience, but not at the expense of fundamental knowledge. Visit www.securityjourney.com to sign up for a free trial of the Security Dojo or schedule a demo. Hey folks, welcome to this episode of the Application Security Podcast. I'm flying solo for this episode. This is Chris Romeo, CEO of Security Journey, and the topic that we have for today is we're going to ask the question, do certain types of developers or teams write more secure code? I'm joined by Dr. Anita D’Amico, who is someone who has thought a lot about this. I've had a chance to meet and get to know Anita over—

2:00Anita D’AmicoMm-hmm.

2:03Chris Romeoover her time starting CodeDx and bringing that along. But Anita, our audience is always on the edge of their seat because they know the first question that I'm always going to ask everybody is, what's your security origin story, or how did you get into this crazy world that we call application security?

2:20Anita D’AmicoWell, my story is probably different than anybody else's. So I have no— I'm an experimental psychologist by training. My PhD is in experimental psychology, and I never took a computer science course in my life. So the way I got involved was that I was working at Grumman when Northrop bought Grumman. And one of the things that they did was they selected certain people from Grumman to go into a class. I guess you could say that it was a class that in which they would inculcate us with the Northrop culture. And it was a bunch of us that were out in California. And we were learning the entire Northrop Grumman way of doing program management. And they would bring guest speakers. And they were like division presidents. So the person who was the Executive Vice President of Business Strategy, Dr. James Roche, came into the class. And this is the person who made decisions about mergers and acquisitions, the entire vision of the company. And by the way, he later on became the Secretary of the Air Force. Okay, so he stood there, and this is in the mid-'90s, and said, you know what the future of this company is? And we all sat there going, what do you think? That's your job, right? And he said, it's information warfare. He said, Northrop Grumman is going to be a leader in information warfare. And He said, who knows what information warfare is? And we all sat there like, oh, you know, looking, we don't know it, nobody knows what it is. And he said, well, I'm telling you, this is the future of the company. And then he went and talked about something else. So after this 2-week course was over, I went back to my office and this was just in the really early stages of the internet. You didn't like go onto the internet and look up stuff, you know, you had to go to the library. And so I went through all these Aviation Week and Space— Aviation Space Week, whatever it's called, the Aviation Weeklies, and, you know, looking for what is the story about information warfare. And I really couldn't find anything other than these statements, right? So I wound up writing a memo to Dr. Roche, In fact, I was chicken. I had 3 other people co-sign it. But I wrote this memo that said, you know, you said that the future of the company is in information warfare. Well, I'm sitting here in data systems in the research group, and I would think if anybody knows what information warfare is, it would be us. And I don't know what it is. And anybody I ask, they don't know what it is. But I can guess. And so here's some thoughts. I think it might be this, and not knowing exactly what it is, but here's 10 ideas of what we might do. Sent the memo off. Now, this is before everybody was comfortable with email, so it went into a manila pack— a manila envelope that you sealed up with a little thread, right? Went into the pouch that got sent from New York out to California.

5:47Chris RomeoWow.

5:47Anita D’AmicoAnd I forgot about it. So about a week later, I get an irate call from my new boss. And he said, did you write a letter to Dr. Roche? And I said, yeah. He goes, you didn't run it through channels. So I was new to Northrop Grumman. And Grumman was a family organization where you saw the president of your division, you said, hey, hi, Jerry. But in Northrop, it was run like a military command, and there was a chain of command. And I had leapt over like 3— I didn't talk to my boss or my boss's boss or my boss's boss's boss. I went right to the top and said, you know, you don't know what information warfare is, here's 10 ideas. So, um, he— I was in hot water, and they convened a video conference. Now this is in the 1990s, and it looked— it was sort of like the Get Smart cone of silence. We went into some conference room and it was really dim, and there were the people in California, and there were people— and I was in the New York group, and they had like all the division presidents there, and I'm sitting there. And There were my— and a couple of my cosigners are sitting there. And Dr. Roche says, so— he's a huge man. I mean, huge man with booming voice. So I understand that some people think that we don't know what information warfare is. And he said, well, let me tell you what I think it is. And then he goes into this whole thing about radars and electronic warfare. And, and I thought, oh, I am so out of my league.

7:36Chris RomeoRight.

7:37Anita D’AmicoAnd I just slinked into the chair. And then he said, so I think we need somebody to write a position paper on where the company is in information warfare. Now, who might that be? Oh, smarty pants. And so he asked me to write a position paper on what information warfare is. And I did. And then he called me to his office in California, and he wanted to discuss it personally. And then he said, here's what I think we need to do. We need to put together an integrated product team, an IPT. Remember those days, IPTs? And he said, and I want you to pick 10 people, any 10 people in the company, and come up with a strategy for information warfare for Northrop Grumman. And so I learned about navigation warfare. I didn't even know that there was something called nav war. There were people who were working in that, electronic warfare, communications warfare, cybersecurity. People didn't even use that term. They used information assurance. And I put together a team of 10 people And we studied the area and the directions that Northrop Grumman should go in information warfare. And that's how I got involved in cybersecurity. It was once again an example of my getting involved in something I knew nothing about. And I think my career— if I had a motto to my career, it has been that ignorance has never stopped me from success.

9:22Chris RomeoThat'll be the title of the biography someday. Right.

9:30Anita D’AmicoAnd so I wandered into it and have never left.

9:35Chris RomeoYeah, that's definitely a fine way to find your way into it. I bet you were sitting just quaking in that room.

9:43Oh, yeah.

9:44Chris RomeoYou're thinking, is this an execution chamber? What's happening here? There's a lot of witnesses. But that's incredible, though, that, yeah, just speaks to your ability to not Not be afraid to go up and put something on paper. And, you know, look what happened. You know, a lot of people, anybody that's sitting there thinking, I'm afraid to do this because of the consequences, you know, look at this as an example of how that can work out for you.

10:10Anita D’AmicoIt did. Yeah. So I know the rest of my career.

10:15Chris RomeoYeah, that's incredible. And now, you know, as CEO of Code DX, you've, you know, you've come from the world of government, into the world of commercial tools trying to solve problems that a lot of customers have out there, you know, as they're working through correlating different results from different tools. And that's something that CodeDx is— that's your specialty. And so—

10:40Anita D’AmicoYeah, and we started out as a research project and wound up being a commercial success.

10:47Chris RomeoOkay, so you came from— so it was a research project that was being done to further the government's trying to help the government be better at solving this problem directly?

10:59Anita D’AmicoWell, so for the— after I left Northrop Grumman, I started a cybersecurity research division of a software company, and the software company was Applied Visions, and I started Secure Decisions. Most people know me from Secure Decisions, and we did all kinds of cybersecurity R&D, and we worked for DARPA, for IARPA, for the Department of Homeland Security, Air Force Research Lab, Naval Research Lab, and we were working on research and we would develop proofs of concept and prototypes of new capabilities in cybersecurity. And CodeX started as a small business innovation research project that was funded by the Department of Homeland Security. We got $99,000 in 6 months to come up with a proof of concept. Remember my motto? Ignorance has never stopped me from success. So Ken Proll and I worked on it, and here we are years later, and we have a commercial company, and we have big customers, both government and commercial, that are buying our products in application security.

12:09Chris RomeoThat's awesome to hear how that— how your career, the trajectory that you've been on, and how those pieces fit together.

12:17Anita D’AmicoAnd the research part of it, and by the way, I did, you know, research in that led to CodieX, but the topic that we're going to talk about today, human factors in secure code development, also came from a research project that I started with DARPA. So it's always been started with some kind of research.

12:38Chris RomeoThat's a good transition. It takes us into kind of the primary thing that we wanted to talk about today. And I know it was, I think, prior to the pandemic on that last conference year. I know that you talked about this at a couple of different big events. So this is research that got a lot of attention and got accepted at a lot of big conferences.

13:01Anita D’AmicoYeah.

13:01Chris RomeoIt's fascinating to me as someone who is focused much of the last 10 years of my career on security culture and how do we help developers get better at security? How do we incentivize them? How do we understand them better?

13:14Anita D’AmicoRight.

13:15Chris Romeowas fascinating that, you know, the first time I heard, I heard you deliver this talk about this topic. And so let's, let's define when you say human factors. I think our audience could, you could interpret that a lot of different ways, kind of human factors. So I'd like to get some definitions first of when you say human factors, what does that mean from your perspective?

13:36Anita D’AmicoSo human factors are the properties of people and their environment that affect their performance, the way they do their job. So they could be psychological human factors. For example, somebody's ability to focus their attention, it could affect how well they, they perform, or whether they have good short-term or long-term memory. And that's a psychological human factor. How they— their style of making a decision could be a human factor that affects their performance. And then there are group psychological human factors. For example, how people collaborate, how they resolve conflict, how they communicate. Communication styles are human factors that affect job performance. So that's the psychological. There's also physiological. human factors that affect performance. For example, how tired you are, uh, that— how much sleep you've had, how much stamina you have, your health. People who have the common cold just perform their jobs less well than they do when they don't have a cold. And then there's something called circadian rhythm, and circadian rhythm is the changes that your body goes through during the course of a day. You know, sometimes like in the afternoons you feel a little sluggish, whereas in the morning you're a little bit more bright-eyed and bushy-tailed. Well, that actually has to do with physiological changes in your body, and that affects your performance. And then the last type of human factor is environmental. So for example, lighting will affect certain types of job performance, or the temperature, or whether or not you're in a distracting environment. So those are examples of human factors and the human factors that affect performance.

15:34Chris RomeoAnd so how does those— when we think about quality and security of code, how does quality— and I love the fact that, you know, in the research that you've done, you're thinking about quality and security together because it's been one of those issues where it's like, A lot of people don't see the light in that. They don't see that there's such a direct correlation. Like, can you have a secure piece of code that has a low quality? Like, there's that connection there. And so how do you see these things fit together with quality and security?

16:10Anita D’AmicoWell, I was really interested in looking at human factors and how they affect code quality and code security. Because I thought it was another way of finding vulnerabilities. You know, most of the effort on finding vulnerabilities has to do with looking at the code. And, and so if you say, well, okay, I'm— you know, you want a static code analyzer and, you know, you find vulnerabilities, or you do a manual code review and you find vulnerabilities. But first of all, when you, when you're doing a manual code review, where do you even start? I mean, where do you start looking for the problems? And same thing with, with you run source code analyzers, uh, you know, where there's, there's all kinds of issues that come out. Which ones do you want to look at first? And so I thought maybe there is something about the people who write the code that could orient us as to where to look for vulnerabilities in the code. So maybe if we knew something about the developer their experience, the time of day that they committed the code, how many people were on the team, uh, whether they were distracted or not. Could we then take that information and use it to orient us as to where to look for vulnerabilities in code? So it was a little bit different perspective, and that's sort of how I started getting interested in the human factors that affect secure code development.

17:39Chris RomeoSo I'm curious about the dataset now. So tell me, tell me a little bit about the depth of data that you were able to collect and where's that data coming from. And it'll help us as we start to talk about conclusions that you had from it to understand the source of the data.

17:56Anita D’AmicoRight. So, well, first of all, everything that I'll talk about today was based on research, scientific research that was done. And so there's research that's been done in— most of it's been done in software engineering, and some of it's been done in other areas like in team performance. But when you're looking for the factors that affect code development, most of the work is actually done on open source because there's a lot of open source. And so there are universities such as Rochester Institute of Technology that regularly look at open source, and, and they look at the publicly disclosed vulnerabilities. Now here's the way the research is typically done. You have an open source repo like Chromium or Apache Struts, something like that, and we know how many publicly disclosed vulnerabilities there are in in Chromium because people report it and it gets logged. So the way the researchers typically do the work is that they take all the files in a code repo, like an open source repo, and they make this pile over here that has no publicly disclosed vulnerabilities and this pile over here that has publicly disclosed vulnerabilities. Now I use the word publicly disclosed because the pile that doesn't Doesn't mean they don't have vulnerabilities, just means that they haven't been disclosed, right? But they— so you have these 2 piles, and what they do is they look for what are the things that maximize the difference between those 2 things? What are the real things that discriminate between those piles? So that's— and, and the things that I'll talk about today, a lot of it was based on research like that using Chromium and Apache Struts and, and, and other Linux kernel those types of repos. The other way to do the research is to take proprietary code and to run static analyzers on it that find quality and security issues. And then to— and then based— and then you look at where all the problems are, and then you say, okay, now what do we know about the development team? How many people were in the team? Were they doing multiple things at once? What was their environment? like and do the research that way.

20:21Chris RomeoWas there surveys that went to these teams that they could choose the different kind of factors that impacted them, or you were able to collect information about their whole environment, or how'd you get to some of those human factors that we talked about at the beginning?

20:38Anita D’AmicoSo first of all, most of what I'll talk about today was based on the open source, but we did do proprietary work as well. And there are other companies like like Microsoft, who also did proprietary work, and I'll report on some of that. But the— when we started the research project, it was very opportunistic. It was more or less, what can we collect?

21:02Got it.

21:03Anita D’AmicoBecause— and so when I— this was a project I was working on that was funded by DARPA, and I was at Secure Decisions. And in fact, Secure Decisions is still continuing to do this work. after I left. And so we actually instrumented the environment of some of the developers. And we asked them things like, when they came in in the morning, how many hours of sleep did you get last night?

21:30Hmm.

21:31Anita D’AmicoAnd what are you listening to? Because, you know, those types of things. It's a very difficult to collect this kind of information. And it was really a hard thing to do. So, most of the work is based on open-source projects.

21:50Chris RomeoOkay. And so, when we think about kind of what you were able to conclude, you know, based on certain types of developers or teams, like, is there— do we have the opportunity to create super developers and super development teams? Like, as a result of what you were able to analyze there? Like, are there some hidden things we can untap to create super developers that are passionate about security? And like, what's the— is there some secret sauce here that we can share?

22:23Anita D’AmicoWell, I would say rather than focus on one of the super developers, I think it's more what are the things that we shouldn't be doing, right?

22:31Chris RomeoOkay.

22:31Anita D’AmicoSo what are the things to avoid? Or what are the— there's the things that we think you should avoid, and there are things that people mythically think are a factor, and maybe they're not. So I think, you know, at the end I could give you some ideas on the types of things that managers should keep in mind if they want their team to write good quality and secure code. there are certain things that they should do and certain things that they should have their team avoid doing.

23:06Chris RomeoYeah, and I definitely want to come back to that. Let's, let's look at a couple of case studies. This, this last 12 months has been strange for everybody. Like, yep, you know, there's a lot of things you could get people to argue about on the internet, and I think if we put that out there, we would just get a bunch of likes. Like, nobody's gonna say, no, it wasn't, this was a normal year. And so we think about how software development teams have been working. A lot of teams historically have been co-located. They've been in the same office. They've been able to share the whiteboards and do these types of things. A lot of people believe this is the way development should be done, but things have been done differently over the past year. What's, what's your perspective on this? Is remote a good thing? And is that something that we should continue even when the world returns back to normal? Or what's, what's your kind of opinion of this?

23:50Anita D’AmicoWell, I'm not going to give you my opinion on things. I'm going to tell you what the data says.

23:55That's the best.

23:56Chris RomeoThat's a scientist answer right there.

23:59Anita D’AmicoSo there's one study on this that Microsoft did, and they, they did a study of the post-release failures in Windows Vista and the Office 2010 binary. So it was done a while ago, and what they did was they compared the work product of different groups, and they were specifically interested in colocation. And so they compared the failure rate of code that was developed by teams that were co-located, like in the same office, that were in the same building, that were in the same campus, that were in the same region, or as far away as different continents.

24:46Right.

24:46Anita D’AmicoSo they looked at— they were really interested in how remote is remote. And they found no difference in the failure rates between the teams. None.

24:58Hmm.

24:59Anita D’AmicoPeople who differed by a continent did not necessarily have any higher failure rate than those who were in the same office area. Now, and I've presented this research a couple of times, and one of the comments that comes in from time to time is, that, well, Microsoft has really optimized remote. So, you know, and that's a fair comment. You know, it may be that if you haven't optimized the way you do remote work and collaboration, that it may not— it may have an effect if it's non-optimal. But that research showed that there really wasn't any difference in whether you were co-located or not. There were factors that actually did predict whether or not they were going to have post-release failures, but it wasn't colocation. There was other human factor.

25:58Chris RomeoThat's interesting because when I think about, like, how would I answer that question purely based on my opinion, which is one of the only things I'm an expert in, is my own opinion, I would have answered it exactly the opposite way just based on my gut instinct of saying, But of course, when people are together, it's going to be better because I can just talk over the cube wall or whatever. But I mean, the data speaks, right? And that's how we're able to draw conclusions there is by saying, hey, the data is telling us a story based on analysis that was done.

26:29Anita D’AmicoYeah. Well, I mean, but you just said you could talk across the cube wall. So one of the things that's worthwhile studying is whether or not there's good collaboration.

26:42Mm-hmm.

26:43Anita D’AmicoSo, uh, the— even though you're remote, if you're on Slack, if you— I mean, I know our software development team, especially since we have been working all remotely during the pandemic, they're on Slack all the time, and they, they work as a cohesive unit. Uh, and so there are ways of collaborating without just, you know, crawling over the cubicle wall.

27:08Chris RomeoYeah. And I think it's easy to see that coming out of the pandemic here, work has been changed forever. Like, we're not— nobody's going to go back to the way things were exactly, you know, January 1st, 2020, or December, November 2020. Like, that's— there's a new setup for work, and that's just going to be the way it is. Now, let me give you another example. Because I know during the pandemic, a lot of people have been working strange hours because they're juggling taking care of schooling their children online. They're, they're juggling just a lot of things, maybe caring for their parents. And they're like, I'm going to squeeze in some work here from 10 PM until 2 AM because I got to get my job done, but I got a lot of other responsibilities.

27:56Mm-hmm.

27:57Chris RomeoDoes the data tell us that there's an impact on the quality or security of the software they produce if they're working in the middle of the night?

28:04Anita D’AmicoYes, it does. And, uh, and the data shows that late-night commits have more bugs in them than morning commits. There's been a study that looked at, uh, the commits that were made between midnight and 4 AM, uh, and found that the bugs in them were statistically higher than the commits that were made between 7:00 AM and noon. And I remember when we looked at this research and I just looked at it and said, oh, of course. And people said, what do you mean, of course? I said, well, it's circadian rhythm. And so I did my doctoral dissertation on the effect of sleep deprivation and circadian rhythm on watch-standing officers on a ship.

28:54Chris RomeoHmm.

28:55Anita D’AmicoAnd I know from all the research that your body changes over the course of a day, that it has a diurnal cycle. And the cycle is that at about 6 o'clock in the morning, there are certain chemicals in your blood called catecholamines that increase your level of alertness. So they go up about 6 o'clock in the morning. They stay elevated until about 2 in the afternoon, then they dip. And then they come back up at about 4 in the afternoon. Now, think about that post-lunch nap that you like to take.

29:35Right.

29:36Anita D’AmicoSiesta, that feeling of, oh man, it's 2 o'clock, I need a cup of coffee. Well, it's not because you had a big lunch. It's actually because of the catecholamines in your body dip around 2 o'clock in the morning and come up about 4, and then they stay up there until about 10 o'clock at night. And from 10 o'clock at night until about 6 in the morning, they go down dramatically. Now, there are all kinds of research out there that shows that human performance follows your circadian cycle. If you are— whether you are driving a truck or piloting an airplane or or does sewing, right? Your performance is going to dip when those chemicals in your body dip. And so when I saw this work that said that late-night commits have more bugs, it was like, well, yeah, they did it when they were the least alert physically. That's a physiological human factor that affects their job performance.

30:37Chris RomeoSo you're basically saying that we should be sleeping from 10:00 PM until 6:00 AM each every day if we want to have that optimum performance?

30:47Anita D’AmicoThat's right. You actually should be. And now there are some people who can change their circadian cycles, but they're few and far between. So people who are on a permanent shift, some of them are able to change, like if they always work nights, but most of them can't. And this actually becomes a physiological and mental stressor on them. There's another reason— excuse me?

31:14Chris RomeoDoes age play into this at all? Like, I'm thinking about my technological career, okay? And so, I got into computers when I was really young. It was a long, long time ago. But I used to be in more of a rhythm where from, you know, 10 PM until 2 AM was like, you know, just an optimum time, felt like an optimum time. Of course, I was young, maybe I didn't know any better. But does any of the research say that there's an age factor that impacts this?

31:45Anita D’AmicoNot that I know of. And not that I know of. And even though I, you know, I believe all the things that I just said, I personally violate this on a regular basis. If I'm in a groove, you know, I mean, I could be at 2 o'clock in the morning and be writing. And who knows whether it's— it may not be as good as what I wrote at 2 o'clock in the afternoon. But I violate this on a regular basis. Getting back to your original question about the change in schedules, there's people working in the pandemic, and they're taking care of their kids, and they're working these odd cycles. So there's another piece of, uh, research that I think is very relevant to software engineers and to anybody really, and that is the amount of hours that you have been awake. So, uh, it's a very strong effect that if you have been awake for 11 hours or more, after that point you're— or sorry, 17 hours or more.

32:55Chris RomeoYeah.

32:58Anita D’AmicoAfter that point, your performance goes down. And really, 17 hours is, if you know, it's not that much. I mean, 16 hours is okay, you're ready to go to bed. So, if you got 8 hours of sleep, that would mean, okay, it's just about bedtime. As soon as you start working past that 16th hour into the 17, 18, 19 hours, Without sleep, performance goes down. Now, I'm not saying it does with software engineers, but in other areas, like in driving, piloting an airplane, healthcare, there are notable decrements in performance. So there's no reason to believe that it would be different in software engineering. In fact, one of the studies showed that the work that you do after you're awake for 19 hours is about equivalent to a blood alcohol level of about 0.05%, where you have kind of like a buzz on.

33:58Wow.

33:59Anita D’AmicoYeah. And it's such a strong effect that the Department of Transportation says that a truck driver can't drive more than 11 hours. And when you look at the medical community, the medical community has rules for their residents that basically say, you know, you can only work no more than 80 hours a week. And when you break that down, it comes down to like 11 hours a day. So I would caution any software engineer or their manager to make sure that they are not working more than 11 hours a day, because your code's probably going to affect— be affected by it. Yeah.

34:38Chris RomeoSo I want to get your take on one other situation before we, before we bring it back, because I definitely want to land the plane. There's another aviation reference. I want to land the plane on that, some of those tips that we have for software developers and managers. But maybe give me a quick, quick, some quick, uh, data on this other situation where we always talk about developers and flow, like especially in a DevOps world.

35:03Anita D’AmicoYeah.

35:03Chris RomeoLike, you know, it might be if somebody's in the zone, but, but, but flow seems to be the DevOpsy— I don't even know if that's a word, DevOpsy— but the DevOpsy kind of word is like the developers, they're in their flow, they're doing their thing. Don't knock them out of that because this is, you know, The Social Network with, you know, the movie with, you know, Mark Zuckerberg had his headphones on and nobody— like, he wasn't even acknowledging other people around him because he was so dialed into what he was doing.

35:30Anita D’AmicoYeah.

35:31Chris RomeoSo what's the impact then of being pulled in multiple directions and interruptions and things that have just seemed like they've become even more of a part of our modern everyday world of work is like, I can't get 5 minutes just to sit here and focus.

35:46Anita D’AmicoWell, if you, if you do context switching, it really does affect your performance. So there's been— and look at the open source world. All right. So there's, there's ways of actually looking at this in an open source repo. When— so some of the research is looking at the files that have been written. And there's something called unfocused contribution. And it's an indicator of how much attention a developer is actually focusing on specific files. So unfocused contribution goes up when a developer is contributing not just to that file, but to a whole bunch of other ones. in close succession, right? So they're sharing their time across a bunch of files. It also goes up when there's a lot of unique contributors to a file. And what the research shows is that the more unfocused attention on a file, the more insecure the code is.

36:56Chris RomeoHmm.

36:59Anita D’AmicoIt was, you know, this— you remember the piles of, like, the pile of files that have vulnerabilities and the pile that don't? Well, the pile that has the vulnerabilities is far more likely to have in it code that was developed by developers who basically sprinkled their time across a whole bunch of different files. So, I mean, that's an indirect measure of attention. But it kind of speaks to the idea that you have to focus your attention on just one or two things, or else your code is going to suffer.

37:34Chris RomeoMakes me think there was an article written a long time ago, back in 2009, and it got a lot of attention on Hacker News, the Maker Schedule, Manager Schedule. And it talked about this idea that— and it's not as much from a direct interruption perspective, but from the fact that like someone who's writing code, someone who's a maker has a very different approach to the world. And for them to be successful has to be very different. The manager will stack 8 meetings or 9, 10 meetings a day, one after another. If you do that to a maker or a developer, they'll never get anything done.

38:15Anita D’AmicoRight.

38:15Chris RomeoThey need blocks of time where they just can go, they can just turn everything else off and focus. And it sounds like what the data is saying, from various studies that you're bringing to us here is that someone who is trapped in that world of not being able to stop and focus, they're going to introduce more vulnerabilities in the code that they're writing.

38:36Anita D’AmicoAbsolutely. And we see this, by the way, not just with open source, but we were able to look at 4 software projects and look at static code analysis. So we ran source code analyzers on the code, and then we looked at how that correlated with the level of focus. And we found that the less focused the developers, the more they spread themselves around, the more static code analysis findings, quality and security. So it's a— I think it's a pretty robust, uh, human factor that we're looking at there. And I would say that developers really should have concentrated time.

39:20Chris RomeoYeah, that's— and as a developer, you really, you think, oh, I want to support my other teammates, I want to be available for communication, when in fact you're actually harming your code and potentially the other people because they're not stopping and focusing either because they keep coming to you and people keep asking questions and stuff. So yeah, it's very, very eye-opening. I want to transition and make sure we get enough time because I want to think through and get your list of what software developer managers should be thinking about to care for their people well in regards to making security. But also, I mean, make it applicable for a software developer too. Maybe their manager's not listening, but for someone who could apply these things themselves to make their life better, or if you're a manager, to make your team's life better.

40:10Anita D’AmicoMm-hmm. So I would say I'm going to draw attention to something that I hinted at before. Remember we talked about the Microsoft study? And I said that there were differences. There was a human factor that made a difference between the failure rates between the groups, but it wasn't colocation. What it was was number of developers.

40:38Chris RomeoYeah.

40:38Anita D’Amicothat the more developers that are on a project, the higher the failure rate. And this is something that we see over and over again in the literature, that as you add developers to a team— and it's actually above a certain amount, there's no golden number, but we've, um, the studies that I've looked at seem to have a kind of inflection point at about 8 or 9, that once you get to that 9th developer, uh, you have significant impact on the quality and the security of the code. So for example, if you take a look at the Linux kernel source code, as soon as you get to 9 or 9 developers, there is 16 times more likely to have a vulnerability in the code Hmm. Than with fewer developers, 16 times. In Chromium, once you get to that 9th developer, the code is 68 times more likely to have a vulnerability in it than if you have fewer developers. I mean, that's pretty dramatic. So, and they found in the Microsoft study that what made a difference in the failure rates was number of developers. So in answer to your question of what should managers be thinking about in terms of ensuring that their workplace is developed in such a way that you have good quality and secure code, I would say, number one, limit the size of your teams. Don't, and if you, If you have to have— try not to have more than 9 developers on one thing. Break them out into smaller groups. Another thing is that you should— if you have somebody has committed code after midnight, be sure to check it. But I would make a rule in general that people should not work more than 11 hours. And I would strongly discourage them from working on something critical after 11 o'clock at night. So those are a couple of the guidelines. Another thing that I think is important, and we haven't really talked about, but I know it's important to you, is security culture. So there aren't very many research projects that have been done which actually collected data on security culture in software development, but there are in other areas. So if you go into healthcare, for example, and you look at the number of poor outcomes, which in healthcare unfortunately means— that's a funny way of saying somebody got really sick or somebody died, right? And you actually look at what contributed to that, you find that if there wasn't a good safety culture, you're much more likely to get poor outcomes. And when you go in and you intervene and you actually create a safety culture and you reward for a safety culture, that the outcomes become much better, that the patients get better faster and live longer. So, You see that and you also see it in other areas like in aviation, having a security culture is important. So I don't see any reason why those results wouldn't also apply in software development. And I know that's right up your alley, Chris.

44:30Chris RomeoYeah, and it's when you think about the safety culture, that's an interesting parallel to security because I get an opportunity to work with a utility company in the Midwest. And they have a— their safety culture is top-notch. But the challenge in making that transition to a security culture is they've realized that if they don't have a strong safety culture, people in their company die.

44:59Anita D’AmicoYeah.

44:59Chris RomeoBecause these are the people that are out working on the lines and doing those types of things. They have to— and if they're going to have a safety culture, they can't just have it for the for the men and women driving out in the trucks that are repairing lines. It has to be ingrained in everything that they do so that it is ingrained in the people in the field, people managing the people in the field, you know, the people working in this, in a service center, the people working in the corporate business side. And so that's always going to be one of the challenges when we try to say, could we just take a safety culture and bring it to the world of security? I wish we could.

45:31Anita D’AmicoWell, why not? I mean, because look, the people, All those areas that you just talked about, they're using software. Software is everywhere, right? And so, you know, whether if you're in the power grid, if you're working in power, if you're working in healthcare, all those areas, aviation, there's software behind it. And software is actually part of the safety-critical system.

45:56Yeah.

45:56Anita D’AmicoAnd so the development of that safety-critical software should be treated with a safety culture.

46:04Chris RomeoYeah.

46:05Anita D’AmicoAnd I don't think we think about that enough.

46:07Chris RomeoI'm going to keep working in that direction. That's my goal. I want to see us get to the point where, you know, we have the same approach to security culture that you have, whether it's aviation, whether it's utilities and power, that same kind of approach. That's always my goal. Another thing that I had to chuckle when I was kind of thinking about the team size, the the finding or the conclusion you had about the team size. It made me think about Jeff Bezos had the, you know, I guess he's not the CEO of Amazon anymore, but he had this thing called the 2-pizza rule.

46:42Anita D’AmicoYes.

46:43Chris RomeoNo team could be bigger than being able to eat lunch with 2 pizzas.

46:49Anita D’AmicoPizzas, yeah.

46:49Chris RomeoBut I think that's the same thing that you're— I think 9 people, what the research has shown, it's pretty consistent with what—

46:56Anita D’AmicoPretty consistent.

46:57Chris RomeoWhat Bezos did at Amazon.

46:58Anita D’AmicoAlthough the developers I know, that's about 6 pizzas, 6 people for 2 pizzas.

47:04Chris Romeo6 people for 2 pizzas. So maybe we got to make a slightly modified rule, but I was thinking about that. Well, Anita, thank you so much for taking us through and sharing the various things that you've uncovered in this research. I know it's given me a lot of things to think about. I think there are some really practical things here that I hope a lot of our audience as software developers can take back to what they do in their daily jobs, those that are managers, and listening to this, hopefully they can take this and say, not only does it make our software more secure, but it makes our— the lives of our developers just better. Like, if you're— the rules you're practicing here are a better quality of life, it's better work-life balance, and it just happens to provide more security. So there's a, there's a lot of positive things here. But thank you so much for taking the time to walk us through this, and we look forward to having another conversation with you in the future.

47:54Anita D’AmicoThank you. I'd be delighted. Nice to see you again, Chris.

47:56Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, security is a journey. Not a destination.