---
title: "Geoff Hill — AppSec, DevSecOps, and Diplomacy"
url: https://appsecpodcast.com/geoff-hill-appsec-devsecops-and-diplomacy/
date: 2020-01-09
duration_seconds: 2213
guests: ["Geoff Hill"]
topics: ["Building an AppSec Program", "Security Culture", "DevSecOps and CI/CD"]
audio: https://www.buzzsprout.com/1730684/episodes/8122618-geoff-hill-appsec-devsecops-and-diplomacy.mp3
transcript: true
---

# Geoff Hill — AppSec, DevSecOps, and Diplomacy

*January 9, 2020 · 37 min*

with [Geoff Hill](https://appsecpodcast.com/guests/geoff-hill/)

on [Building an AppSec Program](https://appsecpodcast.com/topics/appsec-programs/), [Security Culture](https://appsecpodcast.com/topics/security-culture/), [DevSecOps and CI/CD](https://appsecpodcast.com/topics/devsecops/)

[Audio](https://www.buzzsprout.com/1730684/episodes/8122618-geoff-hill-appsec-devsecops-and-diplomacy.mp3)

## Show notes

Why can a technically sound DevSecOps initiative fail before it changes how anyone works? Geoff Hill joins Chris and Robert to discuss the diplomacy behind application security transformation. He describes security as an activity throughout delivery, not a product bolted onto a pipeline, and shares lessons from tool integration and organizational resistance. The conversation moves from security champions to the managers who control their time, showing why persuasion, listening, and political capital matter. Geoff explores the fears underneath resistance, including job security and breaking working systems, and the danger of moving faster than an organization can absorb. He closes with practical starting points: understand the existing workflow, establish repeatable development and deployment practices, and introduce security through achievable improvements.

The Application Security Podcast is brought to you by [Security Journey](https://www.securityjourney.com/).

About Security Journey  
Security Journey provides application security education for developers and everyone in the software development lifecycle.  
→ [Learn more about Security Journey](https://www.securityjourney.com/)

Connect with Geoff Hill:  
→ [LinkedIn](https://www.linkedin.com/in/geoffrey-hill-tutamantic/)

Mentioned in this episode:  
→ [Veracode](https://www.veracode.com/)  
→ [Ansible](https://www.redhat.com/en/technologies/management/ansible)  
→ [Splunk](https://www.splunk.com/)

Chapters:  
00:00 Introduction  
02:00 Defining DevSecOps across the lifecycle  
04:57 Making security native to delivery  
07:19 Lessons from failed integrations  
12:17 Security champions and management support  
16:34 Diplomacy, persuasion, and political capital  
18:54 Listening before proposing change  
20:16 Transformation in nonagile organizations  
25:44 The fears behind resistance  
29:46 Removing barriers through practical experiments  
32:59 First steps toward a working pipeline

## Transcript

*6,988 words · assemblyai*

**0:00 Chris Romeo:** Jeffrey Hill is an AppSec DevSecOps leader and architect. Jeff joins us to discuss his experiences rolling out DevSecOps in both agile and non-agile practicing shops. We hope you enjoy this conversation with Jeff Hill. You cannot hack yourself secure. Everyone wants to focus on the offensive side of the equation. The challenge is that developers get bored with hacking broken pieces of code after a while. Sure, it's a shiny, cool new thing in the beginning, but how about one year later? At Security Journey, we focus on long-term sustainable security culture with the developers as defenders. Our approach integrates experimentation together with learning. We believe that developers need hands-on experience, but not at the expense of fundamental knowledge. Visit www.securityjourney.com to sign up for a free trial of the Security Dojo or schedule a demo.

**0:53 Robert Hurlbut:** Hello folks, and welcome to another episode of our Application Security Podcast. This is Robert Hurlbut, and glad to be with you here today. I'm a threat modeling architect, and I'm also joined by my co-host, Chris. Hey, Chris.

**1:20 Chris Romeo:** Hey, Robert. This is Chris Romeo, CEO of Security Journey, and really excited to be here again to talk about DevSecOps, whatever that is.

**1:33 Robert Hurlbut:** And also we have with us today our guest, Jeffrey Hill. Jeff, thanks for joining us again.

**1:39 Geoff Hill:** Yeah, no problem, no problem. I'm on the hunt from Skynet, so I'm glad to be here.

**1:43 Robert Hurlbut:** And Jeff, you were with us before on another episode, season 4, episode 26. That was Rapid Threat Model Prototyping, if I remember correctly.

**1:52 Geoff Hill:** That is correct, yeah. And I had a great time talking to you guys about that. Things have moved along since then too, which is kind of interesting.

**2:00 Robert Hurlbut:** Oh, that's great. That's great. Well, as Chris said, today we're going to be talking about DevSecOps, and we have had a few people talk about it, or DevOps in general, on our episodes here, on our different talks we've had. So today, just curious as we get started here, what is your definition, or what do you think about DevSecOps? What's your take on it?

**2:21 Geoff Hill:** It's interesting you say that because I'm currently right now stuck in a— well, I wouldn't say stuck in a situation, but deep up to my neck in a situation right now where we're rolling a a group that has had no experience outside of waterfall, or what I like to call waterfail, and they are trying to roll into the modern century by doing this. They had a problem defining what the Sec part of DevSecOps was. Basically, what I told them was, if you think of it, the most basic form is that you want to have security introspection at each stage of the development and release journey, as it were. That not only is talking about checking the code, but also you want to be making sure that all the different devices, for example, if you're running stuff on-site, that all the different devices are actually broadcasting and you're checking to make sure that the logging and auditing of those devices is going along well too. Because you don't want, for example, if you had an Ansible tower get hacked and get attacked successfully, that would be disastrous. You need to have all these things communicating. Now, what the DevSecOps means, like the The DevOps idea is that you have a highly repeatable process with incremental moves that are all holistic. The SecOps part is just adding security into that, but in such a way that, at least my belief is, is that people stop talking about security and they just start doing stuff. Because one of the things that used to bug me all the time in the past was that whenever we rolled out security engineering, they said, make sure you make security front and center. I said, why? You're just going to scare everybody. Just make it another task that people do and remove the fear that people used to have and just make it just another thing because otherwise people put it up on a pedestal and they'll try to avoid it sometimes, or they'll pay too much deference to it and it's ridiculous in a way. The DevSecOps to me is just a smooth operation of build, test, deploy in a repeatable manner with incremental parts that just happens to have security oversight.

**4:26 Chris Romeo:** Yeah, and I'll throw something out here too, Robert, on the DevSecOps front. And I guess one of the things that's always kind of bothered me with this moniker being that it is DevSecOps, it was actually Julian Vehent, who's at the Mozilla Foundation and was a guest of the podcast a few seasons ago. And he had this really funny tweet where he came out and he listed like every derivative, like DevSecOps, SecDevOps, OpsDevSec, OpsDoubleOps. And at the end, he said, why don't we just call it DevOps with security?

**4:55 Robert Hurlbut:** Yeah.

**4:57 Chris Romeo:** Like, why don't we just have DevOps and security be one? And so that's what— it's been one of my kind of soapboxes as well, is just the fact that, hey, you know, it doesn't even have to be— it's DevOps, and DevOps should just have security built into it natively. There shouldn't even be a discussion about it. So that's kind of my take. I don't know, Jeff, what you think about that, if you violently disagree or what. I don't know.

**5:18 Geoff Hill:** No, I agree with that because— and I think that's based upon my personal bias and my personal experience that I've had. is too many times if they try to do all this slapping a sec on top of it or in front of it or around it, which almost sounds like an old Monty Python skit for some reason. But if they do that, then I don't know, they're missing the point. The point is, well, we want to have a smooth DevOps operation which happens to have security built into it. Let's not focus too much and get too hung up on where security sits in it. It then becomes semantic, if you know what I mean.

**5:49 Robert Hurlbut:** Yeah, that makes sense. I'm thinking the same thing is we want to say DevOps and it's already at some point, it's going to be implied or understood that security is a part of that just as much as everything else.

**6:02 Geoff Hill:** That's a good thing.

**6:04 Chris Romeo:** Exactly.

**6:05 Geoff Hill:** I think the 3 of us agree with that. That's a good thing because how many times in the past were we crying away in the wilderness about the reason people were saying like, we're not putting security in. We were saying you need to incorporate security, you need to embed it in your systems, and then they always add it at the end. all of a sudden we have the chance to, so let's just make it DevOps with security.

**6:26 Chris Romeo:** Yeah, now with all 3 of us kind of agreeing here, I was just sitting here thinking, is there any way I can take the opposing side in this argument? And I can't do it though. I can't find myself being, well, we don't need security in DevOps. Like, I'd fall out of my chair and be laying on the ground, so we'll just have to continue.

**6:43 Robert Hurlbut:** All right, well, you've had some experience with implementing this, whatever we want to call it, DevOps with security or DevSecOps or whatever. You've implemented that in a few situations, a few shops, so let's talk about— and probably the most natural way to get to DevOps is probably from a more of an agile perspective.

**7:06 Geoff Hill:** Definitely.

**7:07 Robert Hurlbut:** So let's talk about that in your experiences working with teams that are already following some kind of agile process and now they're trying to introduce this DevOps and some security along with it.

**7:19 Geoff Hill:** Yeah, I mean, I don't know if you guys follow through with this, but I think more and more as I get— as I move more and more into experiences and all that, that I'm tending to talk to people in terms of failures that I've experienced or failures that I've seen as opposed to successes. And the reason is because you describe a failure, you make it sound kind of funny. Sometimes they're ridiculously funny, sometimes not. But then people remember failures pretty quickly. Then they say, oh, I won't do that again, or, I won't do that at all. Whereas if you describe a success, sometimes successes are because of a unique set of elements that came together, and sometimes you're not able to replicate that. Now, in terms of some Agile shops, there have been some shops in the past where— there's one which I— most of them I probably won't be able to name, but one in particular that happened to be a telephone, a phone manufacturer. They had a fairly small— well, they called it a CI/CD development team at that point because they were using a proto continuous integration, continuous deployment model. They were using Agile from year dot, basically several engineers and all that right there. To get them involved in rolling out security into it, we didn't call it DevSecOps back then because DevOps didn't really exist. It was more CI/CD. But we discussed, in many ways, how we could get the security element put in, in such a way that it wasn't going to hold up any of the development, and in such a way that we didn't have that kind of big red button, old school gates where the security person had to review things. What I found in agile shops as opposed to non-agile shops, which was the tendency that the agile shops would look at things and say, well, we're getting data out of this, whatever this is, a certain system, subsystem. How can we go about pouring that data into the next step, writing some code around it, interpreting that data in such a way that we don't have to make that decision, a baseline decision is made, and it goes to the following step after that? If we need to at some point, yeah, we do a review, but the human being is no longer the sticking point in the loop. Because as humans, I mean, you guys will probably agree with this, we have dozens of different things during the day that we're doing, and so we don't really want to become that single point of failure. We'd rather have these processes move along, and if there was an extreme event happening such as 2 standard deviations away, yeah, then maybe that should be a slap down the red button and stop the process. But for the most part, it should happen automatically. Things that we discovered, like one of the things I enjoyed about working with this particular Agile shop was we did a lot of spiking. We did a lot of exploration of how to go about integrating the tools, which were relatively crude back then. We used Fortify, for example, but how to develop that, how to get the output, how to get the input, how to integrate the tool in such a way when the tools weren't designed to be fully automated originally. A lot of those tools back in the day weren't designed to be, and you had to do a lot of code around the front and the back to get them to behave. It was interesting, it was fascinating to do that because I had a lot of takeaways from there. Then when I came to the next one, which is working in non-Agile shops, which funny enough, I did more work in Agile shops years ago and more work in non-Agile shops recently. I think a lot more people, at least from my particular viewpoint, are seeming to move away from waterfall and go into Agile and starting to work that way. One of the experiences would be that they We're constantly looking for ways to automate things and to close the loop. One of the other things that I enjoyed about working with Agile shops as opposed to non-Agile shops was I didn't have to explain the whole fact that the output of one tool is fed into the input of another tool to help you to make a better decision or to help to push things along. One of the things that I found with on-site versus remote, was there's a huge discussion about where to put all the tools. I'm sure you guys have run into this. It's not a simple factor of saying, well, we're going to dump all the tools in prod and run everything out of prod, not by a long shot, especially if you're working in a finance industry firm or something that's highly regulated. That won't float. In fact, that'll be a huge mistake. In the current position I'm in, we're trying to work our way out of that. Because we found that originally everyone dumped everything in prod, in the production, and made a whole bunch of different exceptions to make connections. Then we said, well, that's actually not very secure. Now what we're trying to do is we're trying to re-architect that approach right there, which is a very interesting thing as opposed to running everything remotely on a cloud-based environment.

**12:17 Robert Hurlbut:** You're talking, which is definitely part of it and a question, a lot about tools, integration of tools, and so forth. But what about the process and the people and that aspect of agile shops? Did you notice any issues, any difficulties in moving them to ways of thinking or different ways of thinking?

**12:41 Geoff Hill:** In the agile shops, I think the biggest resistance I got was getting security champions set up. That was a huge resistance. I don't know why. But in the previous 3 places that I worked at, Some people picked it up, like mostly the engineers would love the concept of security champions. Mostly the managers didn't. The managers got very defensive. They got very protective about their people and their team. They said, well, I don't want the person on my team doing more work. Why should they do more work when you're here? I have to gently explain to them it's a way of extending myself because I can't attend every standup. Of course, it's funny, then they get a little bit hurt that I couldn't attend their standup. That required I think that it's interesting. To back up a little bit, I found that my role has, since I've started doing AppSec, has morphed from an AppSec-y type role to a mixture of AppSec and diplomacy because I found that— and this is talking about the process and the people— I found that sometimes that's more important than the technology. Several projects ago, or several companies ago, I worked in a company that loved bright, shiny objects, so we'd just buy kit. and throw it into the Agile process and see if it worked, never bothering to check if it actually had a process to work with. Inevitably, they would fail, or we worked mightily to get them to succeed. I took a lesson out of that because after having 4 projects nearly fail because of them slapping these processes together, I finally got annoyed and talked to the project manager and said, you've got to sort this out during scoping or we can't go forward. Then the lesson learned from there was getting the people to actually form a process to use the tool. Then we got into the whole naughty bits of, okay, let's break down the activities we do during sprint, pre-sprint, and post-sprint activities. Okay, so what are we going to do during pre-sprint activities? Let's break that down. Then once we got that down and we said, look, the security champion is going to help you during these activities because the security champion will bring forth, will be basically leading the charge there, and will have a knowledge of what the security backlog is and will have knowledge of what the current threat model looks like and introduce that to the team and will help the team to basically t-shirt size any issues that come along that have a whiff of security to them. Another process that I brought in strongly, that I strongly push, is that in shops where you do some form of creating the story, when you go from the epic down to the story, or other shops where you create some kind of a business requirement, then at that point in time, the security person should be up front and center and figuring out the security attributes of each one of those epics and/or business requirements and attaching them as part of the story or attachment as part of the epic, because then they will float down and they will make the life of the security champion a lot easier when they go through because they will see this metadata there and they will say, oh, well, the SME has already looked over this, and has figured this out that there's going to be a huge element of access control here. As a security champion, I agree with that. We're going to look at the threat model. Okay, that verifies this thing. So now what we can do is within sprint, we can talk about the different access control methodologies we have, and we're going to push those forward. Then each sprint, at the end of each sprint activity, the security champion makes sure that these things are put forward. So getting the Agile people together, I think the biggest pushback for me in terms of process was getting the managers to come along with it, as opposed to— and sometimes a Scrum Master, but as opposed to the actual engineers. They bought into the idea. I mean, I think it was just because the managers didn't like the security guy coming up and asking more time for people until I waved beer, money, threat of Skynet over them, that kind of stuff.

**16:34 Chris Romeo:** So, I did pick up on something you said there, and the name of my third book is going to be AppSec and Diplomacy.

**16:41 Geoff Hill:** Yes.

**16:41 Chris Romeo:** It's going to be like, I don't know, they'll probably sell 4 copies and 2 of them will be to my mom. So, I mean, it's—

**16:48 Geoff Hill:** I'll buy one off your mom.

**16:50 Chris Romeo:** Well, no, you got to get me 3 copies here at least. Coming soon, coming in 2027 to Amazon, AppSec and Diplomacy.

**16:58 Geoff Hill:** Coming into no near you. Excellent. I mean, but it is true. And now more than ever, we as security professionals We're front, face, and center. I keep telling this to people that we have to have that ability to interface with different groups of people who have different ideas, who have different lists of things and what they want to do. We've always been a soft power group. I've never been in any AppSec or any security group where we've actually had real power. The power has come down to me to persuade the people to go my direction. Then to build up enough political capital that if I needed to, I could pull the button, or I could push the button or pull the cord or anything like that. Generally, I never did because I never wanted to. You build up this list of credibility, and then people would start doing stuff because they wanted to do stuff with you as opposed to having to. That was usually the turning of the corner, especially in agile shops, getting the people to work and then once they realize you understood how their Scrum worked or how their pod worked or whatever the case was. Once you got them to understand that, then they seem to open themselves up to you and allow you to then incorporate more security practice into what they were doing. Then once they realized that doing this work and, for example, running a number of tools in IDE and fixing the stuff before it even got into the main build branch was actually good, Once they realized that, then they jumped all over it. Then I felt like there was less burden for me to go into to punish people because they were doing it on their own volition. They were realizing that if they fix these issues and it never got into the build stream, that then there'll be less work for them, which is fantastic feeling when you get that. That's actually a proper win.

**18:54 Robert Hurlbut:** Well, I picked up there and it's along the lines of diplomacy still. But, but that idea— and in fact, I can remember, um, we've had Brooke Schoenfeld on a few times, and, and he's also mentioned this idea of, first of all, listening before talking.

**19:10 Geoff Hill:** Yes.

**19:11 Robert Hurlbut:** And, and speaking to a lot of people and understand where they are, what their issues are, what their problems are, so that, uh, rather than just come in and say, I've got all the solutions, now listen, uh, listen to everyone else first and find out where they are, and then how you can intersect with what you have some things you want to bring in, what you want to help them with, and so forth. That helps to get understanding and ultimately buy-in, which is what you want.

**19:39 Geoff Hill:** Yeah. It stuns me still for how long Agile has been out there, for example. I'm not going to talk about DevSecOps or DevOps or DevSecOps or anything like that, but Agile itself, that it still stuns me that people who are even running Agile shops that they still are trying to silo security and not trying to— like, I would think that you'd want to build a team which has security specialization or security in it from the get-go. But then in order to do that, then you actually have to talk to people, and siloing security is not a very smart thing if you want to actually talk to the people and get the information across. You know what I mean?

**20:16 Robert Hurlbut:** Well, that sort of leads us into our next one. We talked about agile quite a bit, and certainly you would As we said, we think that they would probably jump on this quicker, and maybe those who are working with it versus managers may be a little faster, not sure. But what about those who are not in an Agile shop, a non-Agile shop? What are some of the things that you've seen there in your experiences?

**20:41 Geoff Hill:** I'm going through some bitter experiences at the moment. I wouldn't say bitter, actually, it's the wrong word, but complicated experiences because sometimes with non-Agile shops, when they want to get the jump, they pull in external consultancy to help them do that jump. The problem is the external consultancies don't necessarily have the same outlook vision as the company does. The company wants to bring an external consultancy in to get them up to shape, get them up to speed. The external consultancy wants to bring more consultants in. As a result, you get a lot of potential clashes, you get a lot of tension that rises out of that, and you get situations where This is speaking from— this actually is like a non-Agile shop situation where they're trying to go Agile and they're using an external party to do it, that the external party doesn't necessarily listen to, going back upon our previous thing, to what the waterfall people are saying in the non-Agile shop. They're not necessarily looking at how can we go about moving these people over to Agile without causing a lot of disruption.

**21:51 Chris Romeo:** Right.

**21:52 Geoff Hill:** I've seen a trend. I mean, I've seen projects fail because people have moved them too fast from non-agile into agile. I've seen external consultancies come in and just make a complete mess of things. Then, of course, they're like, bye, see you later, and they're gone. Then, that's years later and no IPs left or anything like that, which I think is a shame because then it makes the job of the security person that much more difficult. because the security person is seen as part of the problem and not the solution. Again, we get back to the whole diplomacy bit. In terms of a non-Agile shop, what you want to do is you want to approach it in saying, look, I want to integrate security into your DevOps process, as opposed to, I want to be DevSecOps. It sounds like you don't want to be that way. You want to make it sound a bit more natural in that sense. Then what you do is you sit down and you talk to the people who are spearheading the move. So you want to talk to the people at the top and you want to get their buy-in. It's the usual same tools we did beforehand. Once you get their buy-in, you explain, here are the benefits of it. How can we go about— now you already have, you're already using X, Y, and Z tools. How can we go about using these tools to your benefit in a new higher speed or higher revolution development environment? So another thing that I shy away from doing, and I'm sure you guys agree with this, is to not push new tools on them if they have tools that do the job. And I've seen that happen because it just causes more chaos. You want to go in there, you want to say, okay, here's a view of the tools you have currently. How can we make these tools work to your benefit in a DevOps environment? A lot of tools right now are moving towards that, so it's not as difficult as you'd think. If you're using, for example, some of the older static analysis tools, for example, Veracode is something that they've been moving incrementally into the DevOps world. Therefore, it's not as difficult as it used to be to integrate in Veracode. We're actually using it. We're actually doing it quite successfully. Then it's just getting the people over that first initial fear of DevOps. What is this DevOps magic, this wizardry you're talking about where you push a button and everything happens? What you have to do is you have to, in that particular environment, you have to explicitly go out there and and educate them on what spikes are. They get the idea of what a POC is, but a spike is a very fast, let us see if this fails, let us move along with it. Then once you give them an idea, the fear goes away and they see how the security can work because they see an instant solution that happens out there. You might do a spike, for example, like we did recently, is where we ran a very small pipeline, and that pipeline literally had a build and it had nothing else, and then it had Veracode run against it, and the Veracode results come at the end. That's it. But we wanted to show them that it was possible. And then all you have to do is take the results of that. And now what we're working right now is we're showing them how we can take and we can pipe those results into ArcSight and we can do some funky stuff in ArcSight. And then we can produce— and then, you know, another thing would be the next step would be producing dashboards or producing integration in with Slack, for example. But we do it one step at a time. not the whole thing. First of all, because we don't want to boil the ocean, we want it— we don't want them to be completely befuddled. But second of all, because if we show them each distinct link, it's no longer as magical as it used to be. They can actually see the benefits of all the different links, and they can see where they can get benefit. And sometimes they come back with some very interesting observations, like what they can use the information for. A couple of people came back and took a look at the the Vericode output and made some suggestions about where we could go about outputting it to. I thought, I can't go into any more detail, but I thought that was very interesting. That's not something I would have noticed. You're constantly learning from people that way too, whether they're agile or non-agile shops.

**25:44 Robert Hurlbut:** You used that word fear, and I kept thinking about that. What do you think the fears are? Have they expressed what we're afraid of, or just curious on that one?

**25:55 Geoff Hill:** I think part of it is fear of the unknown in the sense that if you've been doing waterfall your entire life and then people come in— well, there are several aspects to that. Aspect number 1 is you've been doing waterfall all your life and you fear you're going to be turned into a dinosaur and put out to pasture. That's point number 1. Point number 2, it's kind of like the old mainframe guy or something like that, a crazy old head over there with a ponytail, that kind of thing. So point number 1. Point number 2 is that you fear that you're going to break a lot of things and you're going to put your job in jeopardy. That's another set of fear that people have. They fear that with all these new processes, that the processes aren't going to work, and they won't be able to go back to the old waterfall processes because they're going to be in this weird semi-state. As a result, they're going to get thrown out because people are going to look at them and say, well, it doesn't work, and you're out. Those are the fears you're going to try to allay because you say, look, if we work well and we do spikes, then we can take each one of these elements right here. We keep it very simple. That's another thing too that I've pushed forward is one of the guys wanted to push his highly complex array of systems and applications into the DevOps build. I said, look, we've got to crawl before we walk. If you do all this and we try to do it all at once, we're going to fail and the project is going to fail. We have to scale that back, and we have to do these things one at a time. And then we get the output of that. We find from the output what the benefit is. And then once everyone gets used to that, then we introduce another tool. And we integrate that in and do the output and input. We're still answering questions right now of whether to break the build on a Veracode run or whether to let it go. We're still asking questions, and we're still doing this for testing of, whether to put the Veracode in a separate pipeline or have it run at the end of the pipeline, that kind of thing. We're still trying to work that out. We have a very strange setup where we're actually getting code from another entity that's being put in, and they're not working on the same Git repo. The interesting thing is they're actually sending code to us, and then we're taking the code, and essentially it's like a fork of the code, but we're getting the fork as opposed to pulling the fork. So it's making things a little bit complicated because we then, since they're using a different, uh, a different SAST tool, then we have to get a baseline from them. We have to run our own baseline once we compile the code, and we have to compare the baseline. So now what I'm doing is I'm working out a mapping table between Fortify and Veracode, and that's proving to be kind of interesting, kind of a bit of a challenge because Fortify keeps changing the rule sets. But it is interesting, and it's something that I've never encountered beforehand. Then, of course, what I've got to do is I've got to keep things simple because of that complication right there. If you start throwing in a bunch of different tools, I'm bound for failure, I'm destined for failure because I won't be able to provide any way of capturing that, or maybe I won't even be able to put the tools in properly, and we'll have a build that'll take forever to build. These are all real-world considerations. For example, in a non-Agile shop, one of the biggest things we have is environment. We have a bunch of static development, build, and production environments. Well, we have one production environment. We have a number of other environments. The big question is, where do we put the tools? Because obviously, if we put the tools in certain environments, then we risk the bad guys getting access to those environments if we don't watch ourselves and we don't do proper configuration. That is turning out to be a a very interesting challenge and something that I never expected I would get into because my background is application security, and here I am working with networks guys, which is fascinating.

**29:46 Robert Hurlbut:** Yeah, I agree. You open yourself up to lots of things, I think, once you start diving into the DevOps world. I mean, that's sort of what it's about, right? Some of those barriers that used to be there are broken down further and further so that we're communicating more than we ever did before.

**30:04 Geoff Hill:** Well, it's interesting because I look at it and I look at how we used to build things and how we're now building things. In many ways, we're going back to some of the old ways we used to build things, such as doing the prototyping, the rapid prototyping, having these prototypes fail. Somehow, it seems like in the last decade, we forgot all that, and we're relearning it. The other thing too is the devolution. It's a little bit of a rant of mine about the devolution. Initially, when you had the shops, the shops were highly controlled by a pretty vertical structure, and the vertical structure said, you're going to use this toolset, you're going to use this programming language, and that's the way it's at. I've gone to the other extreme where I've gone into an Agile shop where literally, it was a fintech shop, where they said every team had its own set of tools, and every team had its own set of toolsets and its own languages that it was developing in. I thought it was wonderful up until the point that one of the guys told me it was a nightmare to maintain. He used an example, there were a couple developers who decided that Rust was going to be their tool, their language of choice. One day, they realized, hey, there aren't many of us. The 2 of them took off, left the team and went off and found employment somewhere else. The team didn't know how to maintain the Rust, so they had to rewrite it all in Java. From a security point of view, it wasn't gray hair inducing, but it was pretty close because they They wanted to pull in these different tools, not toolsets, but these frameworks for Java. I had to keep on slapping them back and saying, look, you can't just do that. You can't just pull a third-party framework out of the ether and just start using it. That proved to be— I really wanted to go back to Rust, but we didn't have any capability of doing it because we had no people who knew how to develop in it. Cry me a song, right?

**31:53 Robert Hurlbut:** All the things that can happen.

**31:56 Geoff Hill:** Yeah, well, and so, you know, one of the things I don't think that people, when they talk DevSecOps, is they don't actually think about locking down— when they lock down the actual servers, are they actually checking the auditing and logging of those servers? I've seen this a number of times now, which kind of surprises me, is that I say to people, well, guys, are you actually logging what's going on in the build engine? And they say, no, why?

**32:21 Robert Hurlbut:** Yeah.

**32:22 Geoff Hill:** Or they're doing it on a completely separate event server or something like that. I said, well, I think security would like to know about that. Then we'd have to rework things and get it sorted out. If it gets, for example, we had one that got sent out to Splunk and we had our own separate section in security where we could actually look at things, events that came through and break it down. But initially, they didn't even think about that. It's interesting from that point of view too, that they didn't think Why wouldn't you do that? That kind of thing. In fact, some of these things like Ansible Tower are so powerful that you really need to keep an eye on them.

**32:59 Robert Hurlbut:** Well, Jeff, it's been great to have you here. I was curious if you could maybe— we've talked about a lot of different experiences here with different shops and so forth. I wonder if you could summarize for us as best you can. I know we've talked about, like I said, a lot of things, but someone getting started with this, whether they're an agile shop, non-agile shop, or anything in between, What should they do to get started?

**33:22 Geoff Hill:** Well, I think the very first thing that they need to do is they need to lock down the process. Let's say they're getting started in DevOps, then before you think about any security, you'd need to lock down the process of how you're going to be publishing, basically how you're going to be getting the code from the workers' machines up through. It means you're going to have to basically decide what your Git flow is going to look like. These are all, while they're functional things, they actually tie into security. Get an idea of what your developer workstation is going to look like, whether you're going to use a VDI or if you're going to use a VM or if you're going to use just powerful machines. When you get that idea, get an idea of how the developer is going to get the code up into their Git, the Git flow, and work this all out before you start anything. Then get an idea from there, Once it gets into GitFlow, how are you going to go about building it? Are you going to do a daily build? Are you going to do an on-call build? That kind of thing. What is your frequency going to be? The standard DevOps stuff. But then once you get that down, then you start looking back and say, okay, how are we going to touchpoint security into this? Do the bare minimum to start off with. Do not try to do too much. At the very minimum, you are going to want to provide the developers with something that they can check their code with, in their IDE. If you want to go to the next step up, then you say, okay, now we want to figure out, we want to get a tool that can get us in the build. Now, preferentially, I would say on the pull request, as part of the whole, as part of the whole, you know, okay, the list of people or approvers, one of those approvers would be, you know, whatever the SAST tool is, and you'd set limits on it. And you say, it's not going to approve that pull request if the SAST tool runs through and says you've got X number of issues. What it's going to do is then it's going to force the developer to go back into their IDE, fix the things, but then eventually the developer is just going to start using the tool in IDE because they know they're not going to get past that barrier right there. Then afterwards, you're going to do a belt and braces and you're going to say, okay, at some point in time, you need to discuss if you don't mind the timing of the build, then you might want to put the SaaS tool in, in, in the pipeline. If you're kind of concerned about that, then you create a separate pipeline and you run it. And these are things you need to do, you need to spike out. So the recommendation is don't throw things together and do stuff, be kind of methodical about it, and then go through and spike each one of the different parts as you're building up your, your DevOps pipeline. And I would say Keep it simple. So start off with one tool and then add other tools onto it. Try not to make a complex toolchain because it just means you're gonna have to maintain a lot of tools.

**36:06 Robert Hurlbut:** Good advice. Keep it simple. All right, Jeff, well, again, thank you for joining us today. Really appreciate it. Enjoyed this conversation. Hope we'll have you back another time to talk some more about threat modeling and other things going on.

**36:21 Chris Romeo:** Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertHurlbut. Remember, security is a journey, not a destination.

---

Source: https://appsecpodcast.com/geoff-hill-appsec-devsecops-and-diplomacy/