Cindy Blake — Aligning security testing with Agile development

0:00Chris RomeoCindy Blake is the Senior Security Evangelist at GitLab. Cindy collaborates around best practices for integrated DevSecOps AppSec solutions with major enterprises. She's proud to introduce her new book, 10 Steps to Securing Next-Gen Software. The book combines her cybersecurity experience with a background in lean and software development and simplifies the complexities of today's software evolution into pragmatic advice for security programs. Cindy joins us to discuss how to align security testing with agile development. For season 7 and beyond, we've launched our YouTube channel, Application Security Podcast, where we post video feeds for all our episodes. You want to check it out, as many interviews now have demos included where we capture screen during the interview, and you get to see our smiling faces. We hope you enjoy this conversation with—

0:55Cindy BlakeCindy.

0:55Robert HurlbutCindy Blake.

0:56Chris RomeoYou cannot hack yourself secure. Everyone wants to focus on the offensive side of the equation. The challenge is that developers get bored with hacking broken pieces of code after a while. Sure, it's a shiny, cool new thing in the beginning, but how about one year later? At Security Journey, we focus on long-term sustainable security culture with the developers as defenders. Our approach integrates experimentation together with learning. We believe that developers need hands-on experience, but not at the expense of fundamental knowledge. Visit www.securityjourney.com to sign up for a free trial of the Security Dojo or schedule a demo. Hey folks, welcome to this episode of the Application Security Podcast. This is Chris Romeo. CEO of Security Journey and co-host of this podcast. I'm also joined by Robert Hurlbut. Hey, Robert.

2:03Robert HurlbutHey, Chris. Yeah, good to be here with you, a threat modeling architect.

2:07Chris RomeoAwesome. And so our topic today is aligning security testing with agile development, and we're joined by Cindy Blake. And so Cindy, what we like to do is just jump right in with the security origin story question for you. So if your security career was a comic book, What would episode 1 look like as you had your security origin occur?

2:31Cindy BlakeEpisode 1 of a comic book. Wow. That's an interesting way to phrase the question. Um, I don't know. I was a developer long ago, um, way before Agile, and, um, got into the security space at Fortify, actually, HP more broadly was involved with Tipping Point, Fortify, ArcSight, all of that, and then moved into Fortify to focus on application security. I think the intersection between AppSec and DevOps for me started when the GM said, you know, there's all these DevOps tools and we know we need to interface with some of them, but we don't know how to prioritize which ones. I did a research project with a third party, and we did some interviews and good primary research and found that there were 50 different DevOps tools that people were using. We needed to intersect with them, but it was challenging. That was kind of the thing that sparked my interest in having DevOps tools developers better enabled for application security. And so at GitLab, that experience and project really lent itself to what I'm doing now.

3:58Chris RomeoWhich languages were you coding in as a developer?

4:02Cindy BlakeNow, that is kind of a biased question because that's going to give away my age.

4:09Chris RomeoOkay, okay, you don't have to answer if you don't want to.

4:12Cindy BlakeIt was COBOL.

4:13Chris RomeoOkay, which is at this current day and age, a very popular programming language. Like, you could travel to certain parts of the country and jump right in and be making $500 an hour writing COBOL of all things.

4:29Cindy BlakeYeah, actually, the group that I worked with, there was a whole group of us that were hired out of college that worked at Texas Eastern in Houston, and we've all been reaching out to one another on Facebook going, hmm, maybe we should dust off our free evening activity.

4:45Chris RomeoYeah, you could go save an unemployment system somewhere with some new COBOL code. Crazy that COBOL still runs the world, but it is. It still works. That's the thing. If it didn't work, nobody would use it anymore. It exists because it works. It tells us that as developers, when we're thinking, nobody— I'm writing this code. 5 years from now, this will be forgotten.

5:06Robert HurlbutHa!

5:07Chris RomeoThat's what you— you wish it was forgotten 5 years from now. It'll live on somewhere.

5:11Cindy BlakeRight. So, so write it securely because it will continue.

5:15Robert HurlbutYeah.

5:17Chris RomeoSo, so do you think— is any of the code you wrote still— you think it's still running somewhere?

5:22Cindy BlakeIt could be. Who knows? That was oil and gas pipeline company, and it's been bought and sold many times. So who knows?

5:32Chris RomeoThere's a chance some of that code's still out there. We had a guest one time who had to reach back after— he was a developer, had written some code, and then it was something to do with like his alma mater, and he happened to reach back out and found out the system he wrote was still in production.

5:48Cindy BlakeRight.

5:49Chris RomeoAnd he had written it as a student, like knowing nothing about security or really web architecture at all. And so he had to reach back out and say, yeah, we might want to do some tests or something to this because there's probably some SQL injection. Built into this code that I wrote. Well, the topic we're going to get into here is really what we talked about at the beginning, aligning security testing with agile development. And so when we first were chatting about this conversation, the security testing angle of it really caught my attention because I think that's definitely a part of our tool suites that a lot of times we take for granted as far as like what we're going to be able to do there. So I wanted to start by asking you, Cindy, when you say AppSec scanning, embedding in development. What do you mean by that terminology?

6:36Cindy BlakeSo, it really— when you think about the processes for development, security really hasn't historically been part of that, part of that phrase, part of that workflow. You know, it's been a very siloed effort. With DevOps, you know, you've got You've got changes in roles where people are more cross-functional in nature. Security has a natural fit into that because while traditionally security people, professionals, are the ones who would find vulnerabilities in code, it's the developers that have to fix it. In the spirit of being more efficient and doing things once as opposed to redoing them, wouldn't it be great if you could empower the developer to find and fix vulnerabilities while they're working on the code? That's the holy grail. Getting there has been very challenging for people because application security has been around for quite a while and there's some ingrained, entrenched methods, and, you know, that being a catalyst for change, DevOps has pushed in the right directions, but it's fighting a bit of a battle in terms of tools and workflows and roles that have been entrenched for some time. Bringing all of that together to have a united workflow is really What are some of the tools then that you're describing there?

8:29Chris RomeoLike, what are— when you're describing those tools, are they things like dynamic application security testing and static application, or are you thinking of other things? Or kind of what's your description of the tools that fit into that picture?

8:43Cindy BlakeWell, because historically these 2 efforts have been kind of siloed, each has had their own tools, right? So the developer has had source code management and automation with continuous integration, continuous deployment, and that has not historically included security. Security has had their own tools, like static and dynamic analysis. I think the advent of IAST, or interactive application security testing, has been an effort to try to shift those scanning techniques left and get them in the hands of the developer, but it's come from a security perspective. I think the tide's turning and we're beginning to see that it's not a matter of shoving security tools over to the developer and asking them to use those tools. It's more of coming at it from the developer standpoint and how can you get into the developer's native workflow and bring that security perspective, but also bring automation and try to really remove as much of the busywork as you can.

9:50Chris RomeoWe had another question about why have so few succeeded in achieving this, but I think you kind of gave that answer in the description of the separation between the source code control system and the classic security tools. Is that what you see as some of the challenges as to why this hasn't worked together?

10:12Cindy BlakeWell, yes, and another piece I would add to that is application security is just plain hard. It's never been as ubiquitous as something like web application firewall or network security or some of those things that you can just plug into a network and do a little bit of configuration and it goes. Application security really requires that people, process, technology all come together. There's an educational element. There's an element of Because of that separation in roles, you know, there's finding and there's fixing. And I think oftentimes the security profession in general gets so fixated on finding the vulnerabilities that they forget about the fixing part. And, you know, I think what we're trying to do is turn that around a little bit and focus on let's fix things before they ever progress. I think that those kind of go hand in hand.

11:20Chris RomeoWould you, when you're thinking about this kind of separation, I'm going to go back to that same language of kind of the separation from the source code control system to the classic security tools. In your experience and kind of what you've seen in the industry, do you feel like security gets development? At this stage or not?

11:42Cindy BlakeThey're starting to, but I would say, you know, just like any good marriage, it takes two, and dev needs to get sec as well, right? And I think that that's happening at the same time. So, you know, I did a focus group with a group of developers last year, and I asked them what motivates developers to craft secure code. And what I was looking for is, Is it a carrot or a stick? Is it metrics that motivate them or peer pressure, management pressure? What is it that motivates them? And I was surprised to learn that every single one of them in the group was motivated most by their own personal reputation. Nobody wanted to be— that person that brought their company down because they were the one that delivered the insecure code that was exploited. So, that's a huge motivator. So, dev is starting to embrace it. In fact, we've had DevOps groups that have gone off and purchased GitLab, including our— and used our security capabilities and said, okay, now we're going to unplug Fortify. And The security team was like, whoa, whoa, whoa, you can't do that. So, they want to do it. They want to do security, but they want to do it in a way that works within their workflow, that doesn't inhibit their velocity, and that doesn't create extra work. At the same time, security is beginning to understand that— well, I think they've understood for a while— they'll always be outnumbered. There will always be more developers than security people. They've started to understand, though, that if they can harness development as an enabler, then they can focus on a couple of things. One is more of the value-added effort of helping to resolve the vulnerabilities that are challenging, as opposed to the busywork of, let me run this report, get 10,000 vulnerabilities, and then spend all my time prioritizing, vetting false positives, and false positives and triaging it back with, with dev and following up every few weeks to make sure, you know, it somehow got into their next sprint. So, they're beginning to realize that they can focus more on value-added effort if they can leverage some automation with the development team. Then, they can also focus on what should the policies be, because when you automate, you can automate to the policy. And so, you know, and how effective are my policies? Are they correct? And one of the things that people have found with automation is they'll take a policy that they believe has been followed pretty closely. And then when they automate it, they realize one of two things happens. Either everything grinds to a halt because their policies are very, very strict.

14:49Robert HurlbutRight.

14:50Cindy BlakeAnd, you know, the other side of that coin is the way— what's been happening in a manual environment is everything's become an exception, right?

15:02Chris RomeoMm-hmm.

15:02Cindy BlakeSo the policies aren't really being followed. They're being worked around. And when you automate it and you now have that transparency and that visibility, you see that, wow, okay, this wasn't really reality. And so maybe I need to rethink my policies and recalibrate them.

15:22Chris RomeoYeah, the term that I've been saying way too much lately, but I'm going to continue, it's my— these are my 2 words for 2020, and that is developer empathy. I think as security people, and our audience is probably about getting sick of hearing me say these 2 things, but I don't care, I'm going to keep saying them. Developer empathy, though, it's For those people that, such as myself, that primarily come from the security background, we really have to walk a mile in the shoes of the developers we're serving to understand the challenges. And I've told a couple different stories throughout the time of the podcast here about where my eyes have opened. You know, the first time I had a third-party library vulnerability in a component for a cloud service that I was responsible for.

16:08Cindy BlakeYeah.

16:09Chris Romeoin our own little startup here and looking at that and going, you know how many times I've told developers that this is not acceptable? And guess what I'm about to do? I'm going to make an exception in my— I'm going to filter this out because there's no solution today and I want to keep shipping code into production. And I'm like, now I feel— I realize the pain, some of the pain that I've generated in the past by saying, no, I'm from security and this is how we have to do it. That's developer empathy. I think fits in line a lot with the things you're describing here about how we get closer to meeting the developers where they are. Instead of saying, we have to do this because we're security and we know what we're talking about, it's more about how do we plug into their world and give them the things they need in their space.

16:51Cindy BlakeAbsolutely. If you can use a tool that works for both, that works for the developer and has a security perspective, you can get everybody on the same page. and improve that empathy. For instance, if you have a single source of truth and everybody sees the same information about the vulnerability, and you've got a list of those vulnerabilities, when the developer goes through and looks at them, they might dismiss something and say, it's a false positive, or, it doesn't apply to me, for some reason. document it, what have you, and then they don't want to look at it, right? They want to set it aside and move on. The security person, what we found is those are actually the things that they care the most about. What is the developer dismissing? Let me look at those and make sure that I don't have either an education issue or, you know, a pattern here or something.

17:48Robert HurlbutYeah.

17:49Cindy BlakeIf you're looking at the same thing, it's just a matter of flipping that toggle and that filter. you know, and yet still meet the needs of both.

17:58Chris RomeoYeah, yeah. Robert, what's your experience been between kind of— as some— because I know you're somebody that's— that came from the development background and is now primarily in the security world. What's been your experience with the tension between the two, and have you had better experiences because of your strong development background as a security person? Have you felt Have you felt better, you know, felt for the developers you're trying to influence as a security person more?

18:27Robert HurlbutOh, sure, sure. And there was some tension, I guess, some time ago, maybe 10 years ago or so, 15 years ago. But I think more and more these days, I don't see it as much. I see more developers just want to write good code. And what does that mean? Sometimes it does mean secure code. Or, you know, maybe code that tests well. And that again may include security as a part of it. So I see more developers are not just trying to get it done, but they also want to make sure it's secure as well. At the same time, I also hear occasionally— well, certainly when I was a consultant— that They may say, well, I want to be more secure. I want to write more secure code, but I'm also hitting deadlines. I've got to get this done, and, and that's making it much more difficult to, to get it done as a result because I have to stop, make sure it's secure, all those kinds of things. So I have heard that as well, and certainly some empathy there. I understand that. I've been there myself. Where you're under a deadline, you want to make sure it's secure. If you don't have all the tools, you don't have all the processes in place to, to do both, it can make it difficult. Which one do I choose? Sometimes is what a developer feels like they're running up against.

19:57Chris RomeoSo Cindy, what do you think are the best practices that work then? I'm certainly— I just enjoyed this conversation we just had right now about kind of the developers and Robert's perspective, your perspective, but Now, when we start thinking about how do we— what are the best practices to actually address this issue that we've identified? What are some of the things that you've seen?

20:19Cindy BlakeWell, I think a lot of people have tried to pull scanning results into their CI pipeline, and you've got to think about where DevOps comes from and the whole DevOps toolchain. Going back to my days at Fortify when I did that, that research project many years ago, back then there were, you know, 50 different tools that people were using. There are still 50 different tools that people are using.

20:45Chris RomeoMm-hmm.

20:46Cindy BlakeThe DevOps toolchains are really ugly. So, it's been challenging for them to pull in security. What I found as a best practice and what works best is 2 things. One is you need to pull it into the pipeline before the code is merged with anyone else's. So, on the feature branch, that's where you should be doing the scanning. And what that allows is, if I'm scanning my feature branch from— given that this is probably a security audience, let me translate for just a moment. As a developer, I'm going to check out a piece of code, and I'm going to make some changes to it. That's a feature branch. I'm doing— I've branched off of the main code base, and I'm working on a piece that I'm working on alone here at the moment. If I can run my scan on that, what that means is if a vulnerability is found, I created it. It's not Robert that created it. It's not Chris that created it. I created it. So from a pure visibility actionability standpoint, it's much, much cleaner and clearer that I'm in the position to correct it. I still may not be able to correct it. To your example, sometimes there's just not a solution. But if I'm able to, now I'm in a better seat, better position to do it. The other best practice is if you can use a single tool that gets both the developer and the security person on the same page, then it's a little bit less adversarial, right? You're both working towards the same goal. You're seeing the same thing. You don't have to get lost in translation between what the security tool says and what the developer sees, right? Because the hardest thing is when you have a vulnerability finding and the security team finds it and says to the developer, I know it's your code, which sometimes they don't, but if they get far enough to say, I know that it's your code, but they can't tell you where exactly or why, that's very frustrating. I mean, imagine trying to figure that out. So if you're both looking at the same thing and you can see exactly where it came from, you know, those are the best practices. So where you get the results is so much more important than the results that you get. Security people get really hung up on finding everything. It's better to find it and have a good context for that remediation than to find it and just be frustrated because I have all these vulnerabilities and nobody's fixing them. You know, so where you have the results and getting everybody on the same page would probably be the 2 biggest initiatives that I've seen.

23:31Chris RomeoAnd that really comes into integrating your security testing and your automation into your source code management system, though.

23:41Cindy BlakeInto your CI, actually. It's, it's into the continuous integration because you can do source code management without necessarily doing CI. And, and it's really that continuous integration is where that magic happens, because that's where your pipeline is and your automation. If I've got this code now, what do I do with it? Here's all the steps I want to do with it. If I send it off to a scanner and I don't get those results back to the developer in their pipeline, it's branching off from the process. It's, it's creating another path. And you really want to have You know, that single path. I liken it to— a good metaphor for this is the cell phone. Think about how many times you use a real camera anymore, right? The camera might be the best tool for taking pictures, and you really want to save that for, you know, like your family pictures, your baby pictures, whatever that you're going to take once a year. But you don't often— the day-to-day pictures that you want to capture is just part of your living experience and you want to share it with your friends and family, you don't use a camera for that anymore because it's not integrated. This is integrated with your, you know, with your social media, with the internet, the way you store these pictures. And that's so much more important and beneficial and useful that sometimes we forego what is like the absolute best single-point solution for that integration.

25:11Chris RomeoYeah, yeah. We're in here in the— at the podcast, we're normally quite non-commercial, but I'm going to ask you a question because I want to understand how GitLab actually works. Okay, so when, when you're talking about doing these kind of security automations from a GitLab perspective, Is GitLab kind of mapping out the pipeline and then including the security tools in the pipeline side, or are you doing some things in the source code management side?

25:42Cindy BlakeIncreasingly both, but I have to say, most people think of GitLab for our source code management capability, but we also have CI. In fact, Forrester put us at the top of the wave for CI. It's not something you'd settle on. It's a leading, market-leading product.

26:02Chris RomeoYeah.

26:02Cindy BlakeAnd developers love to use GitLab. So, if security can take, kind of ride those coattails, it's a path of least resistance, because developers already like to use the tool. But the security scanning fits within the CI part of our capability, just as, you know, if you look competitively and you, Azure DevOps, GitHub, it's within the CI scanning. It is really— I mean, the CI workflow is really where security scanning needs to reside.

26:36Chris RomeoOkay. I guess the last question then is, from your experience, what have you seen as far as the velocity improvements from people integrating and doing things the way that you've described here? using automation and working in the pipeline.

26:54Cindy BlakeYeah, we've seen countless customers that have really dramatic improvements from, you know, pushing code to production once a month, once a week, to pushing it multiple times a day. And so that's why GitLab is so popular as a DevOps tool, and the fact that the security is embedded within that is really the secret to getting that congruent process and that velocity where the security piece has the same velocity as the, as the development pipeline.

27:31Chris RomeoVery, very cool. So if you, I guess, from a last kind of conclusionary statement or key takeaway or kind of a call to action, what would you leave our audience with as far as kind of one thing?

27:46Cindy BlakeI think the one thing is the mindset is a little bit different here. It's not about finding everything. It's not about scanning my entire code base. It's about embedding security into the workflow and the process. So you're scanning the new stuff, you're scanning the changes, the delta, and you're looking for what new vulnerabilities have been introduced. So I'm kind of stopping the bleeding, right? I'm saying I might not be able to fix everything that's been out there for 20 years. But I'm going to prevent any new vulnerabilities from being created and critical vulnerabilities from reaching production. I'm going to do it by retooling my development workflow to include security.

28:28Chris RomeoYeah, I think that's an important thing for a lot of folks to understand because so many people do suffer— security people do suffer from the 20,000 findings, and we just need them all fixed by the time you ship. To production, which we all know in this day and age is just not something that's going to work in the world. And I like the way you put that about, you know, it's moving, it's about making things better moving forward. And that's, that's how we, that's how we build more secure software and do it in a sustainable way across the board.

29:00Robert HurlbutThank you.

29:00Chris RomeoCindy, thanks for being with us today and sharing your experiences here. We, we definitely enjoyed your perspectives and Look forward to chatting with you again in the future.

29:10Cindy BlakeGreat, thank you. It was a pleasure.

29:14Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, security is a journey, Not a destination.

29:36Robert HurlbutNot a destination.