Chris Romeo -- Security Community at Any Scale

0:05Chris RomeoThe Application Security Podcast. Here we go. On this episode of the AppSec Podcast, Robert interviews yours truly about security community. So I talk about some of the experiences that I've had in the past doing security community in large organizations, and then Robert keeps asking me, how do we apply that to small organizations? So you get the best of both worlds from a security community perspective, how to make it work for large companies, but also how to make it work for really small companies. We hope you enjoy.

0:53Robert HurlbutSo today we're going to talk about, in this podcast, security community. This is Robert Hurlbut, and I'm here with Chris Romeo, and thank you for joining. So Chris, tell us a little bit about your background in terms of security community and what does that mean to you and, you know, some of your experiences.

1:19Chris RomeoYeah, so in my career, I worked at a large technology company for roughly for the last 10 years. So I left that large technology company this past January. But one of the things that I did in my time there was I really focused on building this thing that we call security community. So I think of security community as inside of a large organization or maybe even inside of a small organization, How do we get people that are passionate about security to come together and to encourage each other and to teach each other and to motivate each other with the single goal of saying, how do we make the products that exist within our company better from a security perspective? So when I think of security community, that's what I'm thinking of is how do you bring those people together and really fire them up and get them passionate about security?

2:14Robert HurlbutOkay, that makes sense. So let's say you have a company that has just a few people that are interested. I mean, what would you say to them in trying to get started? And yeah, what would be the first thing they need to do?

2:28Chris RomeoWell, I think it's great to think about it like this. If you have that one person that is passionate about security, that's all you need to start. So if somebody's listening to this podcast and they're thinking, oh well, you know, That's great, he's talking about a large technology company, but I'm a single person who's excited about security in a company of 100 developers. My advice to you is that's okay, that's good. What you have to do at a smaller scale are some of the same things that you do at a large scale. It's about how are we going to initially reach people and let them know about the benefits of making more secure products. And some of the same things that I would do in a large company, you can do in a small company. One of the things that I find consistent between large companies and small companies is they usually both eat lunch. And so what you can do is over a lunchtime session, which might even be easier for a small company because you can order a couple of pizzas and invite 20 people to come and listen and have a little conversation about security. And the idea is in the beginning you just want to make them aware and try to answer the question, why do you need to care about this? If you don't. And what you'll see is some people will start to step forward and be more passionate and more interested in improving security, and that can become your core group for how do I do something bigger.

3:51Robert HurlbutOkay. So, let's say that you're starting to put this together. What are some of the values that are— why do you need that anyway? I mean, we talked a little bit about that, but if you could just a little bit more. Let's say myself as a security person in the company, they're interested in this. What's the value of me relating to others and being a part of this community?

4:14Chris RomeoWell, the value— there's some value for the company and there's some value for the individual. The value for the company is that the more people that get crazy and passionate about security, the better you're going to get at actually finding security vulnerabilities in your products. So if you have a company where people could care less about security, they're not gonna have any interest in trying to find any problems in the products. So when they get that passion, you can drive down the amount of time it takes to fix security problems that are found or those types of bugs that are open. You'll start to get people that are actively applying security into what they do in their day-to-day job. So that's kind of the institutional value that you get out of this type of a security community effort. On the individual side, listen, if you're not in the security business right now, if you're somebody who's a developer or tester, I got news for you. One of the hottest places in the job market is jobs that have the word security in it. And if you already have development skills and a little bit of passion and a little bit of interest for security where you can go and learn some things, developers that speak security are very, very valuable. You will not be— if you reach that level of knowledge and expertise, you won't spend very long waiting for a new job opportunity to come your way, 'cause there's just not a lot of people that fit into that category right now. So that's really the benefit for the individual is advancement, additional learning, and just doors that will open up to you to take your career in a different direction. You may have thought, I'm gonna write code every day for the rest of my life, but I got news for you. You might be able to take that knowledge and love of writing code add some security into it, and have a whole new thing that you can do.

6:04Robert HurlbutWell, yeah, that's a great value, and I can relate to that as well. It's been great for myself to continue to be in security and switch from being development only to combining it with security and ways to help other developers. So I can definitely agree with that statement.

6:22Chris RomeoYeah, and you speak development first, which I think is what a lot of the developers out there— that's the advantage that they can really bring. And what I find is when people come to development after they've already spent years and years in security, they just, they just don't have the same core knowledge about how to actually write code, how code is actually tested, how it's built, what are all the things that you're dealing with. So Robert, I think like in your example and your experience, you have a very unique perspective because you were a developer first and then the security light bulb went on for you. Versus the other direction.

6:58Robert HurlbutRight. And that, yeah, that made the difference. If I went the other way around, I think I would probably be lost in many ways. So yeah, I'm glad I went the developer to security route for sure. So just curious about— now let's talk about, you know, we built the community. I'm assuming that there's also some roles within that community perhaps, and I've heard these terms. security advocate, security champion. Tell me about those. What are those roles, typical roles? And maybe those are not the only ones, but what are some of the roles that some people may have in a security community?

7:36Chris RomeoWhen you're building your security community, it's good to have a role or a title or a name to call the people that are really passionate about security. I've used the term security advocate at my previous job. Other people say security champion. Other people say security guild members. There's lots of different words that get used to apply to this role, but it's actually all the same. It's all the same role. These are the folks that are passionate. You want to— because they're doing things like championing security. They are advocates for moving security forward. So this is the most primary role, and it's for those that are already bought into the process of doing security. So you give them that title and you call them a security advocate. I've had a lot of success in the past by making that a virtual role. And what I mean by that is the person's job description doesn't change. Their title in your direct— in the organizational directory does not change to security advocate. Their title remains software engineer, hardware engineer, product manager, whatever they are. That security advocate is like a badge that gets applied to their existence. And as they start to think more and be more in-depth about security, they can bring security into their development world, into their product management world, instead of just being a security person off to the side. So that's really the first kind of role. The second role is, I think of as everyone else. I think that everyone has a role in being part of the security community. Some people need just a basic level of awareness applied to their world. So, for example, a developer, I believe, needs some amount of security training to— if it's a web developer, they need to understand what is SQL injection, what is cross-site scripting. These should be things that they live and breathe because they know They truly know what they are. They should understand the secure coding principles of the language that they use. So that's— but that's more into the everyone. They don't necessarily have to love security and want to make it their life's pursuit to have an appreciation for the OWASP Top 10 or for secure coding principles. So everyone also fits into that security community because I truly believe that everyone in an organization has a role in making security better.

10:09Robert HurlbutOkay, so in terms of a security champion, I know I've read a little bit about that, I've written a little bit about it as well, in terms of how do you identify a security champion, let's say, on a development team or other kinds of teams. Do you have any thoughts about that?

10:24Chris RomeoYeah, definitely. So you have to— you're not gonna— it's very difficult to instill passion about a topic in somebody. So what you have to do is you have to be on the lookout for the people that are already interested and have some experience or have some knowledge of security because they've gone out and studied it on their own. So in the beginning, it's important to, to find those people and tap into their— the passion that they already have. Because if you try to, if you try to teach somebody, you can't really teach passion. about a particular topic. You have to find the people that are already into it and use them as the foundational building blocks of your program.

11:08Robert HurlbutOkay, that makes sense. And that's my approach as well. I look for the people that are really passionate about security, interested. Sometimes I've seen also, if you have a lunch and learn, see who shows up.

11:22Chris RomeoYeah, being in the room is definitely the first vote towards I might be interested in this and want to do something more with it. So yeah, I definitely agree with that.

11:32Robert HurlbutExactly. Exactly. Okay. So along with those lines, let's talk about building that security community in an organization. So we've talked about how important it is. I think we understand that and the value that you get from this to the organization and to the individual and some of the roles that people might hold. Give us some examples of how could a security community be built into and grown into an organization.

12:02Chris RomeoSure. So the first thing that I do when I'm going in to help somebody build their own, their internal security community, is you have to, you have to set a monthly meeting or a monthly training cadence and get people used to getting together at least once per month to hear about updates about what's going on with security externally. Also, what's happening, are there any changes in our process or the tools and things that we're using? So, you have to establish that monthly training session. And so, when I, with the previous organization that I was working with as an employee, the monthly training sessions really became something that people looked forward to because as I had more and more people interested in it, I went out and started reaching out to different folks from the world of information security, external people. And I said, hey, will you come and speak at our monthly training session? I had Dave Kennedy come in and do a quick talk. I had Jeff Williams come in and do a quick talk. Katie Mazuras, who used to be at Microsoft, had her come in and do a quick talk. And so that really energized the community. We brought in some really high-level people who speak at all the conferences. And the catch is, when you ask them to just do a quick 30-minute talk over a web conference where they don't have to travel, almost anybody will do that because they want the exposure to be able to talk to a company that they haven't done anything with before, or maybe an industry that they're not so familiar with. And you're not really asking them to get in an airplane and fly for a day and then stay overnight in a hotel and then come and give your 30-minute talk. You just They can do it right from their office. So, monthly training really becomes where it's at as a great place to start as the first example of how you can build up your security community.

13:53Robert HurlbutOkay, great. So, what's next? You've got this in place, monthly cadence and so forth. What's next?

14:01Chris RomeoSo, I think that the next thing that I like to think about is how are we going to get everybody in the organization to begin to engage? As we start to see some people that are passionate step up, What's our plan for how are we gonna connect with the rest of the people across the community? And so the connection that I'm looking for there is how are we gonna teach them the basic lessons that we want them to learn? And so what I did at my previous organization is we built an application security training program that focused in on the secure development lifecycle and all the pieces and things, but also talked about who's trying to attack you, why are they trying to attack you, what are some fundamentals about security? So that really, as a second step, provided people that didn't— maybe they had the interest and they said, oh wow, security sounds like something that's really cool, I want to get into it, but they didn't have any foundation in it. The training experiment or experience kind of gives them a place to learn many things about security and really start to drive themselves forward and gives them kind of a starting point.

15:10Robert HurlbutOkay. So if I understand what you're saying is put some kind of training in place, and that could be a formalized way of doing training, or could that also be providing perhaps books or subscriptions to videos of some sort or things like that? Is that some other ideas for training?

15:28Chris RomeoYeah, certainly. There's lots of great resources out there in this day and age that you can provide for people from a books perspective, from a blogs perspective. Lots of different security training that's available that, that will connect with the individual. So yeah, those are things that you can, that you can provide as, as resources for those people that are just starting to get interested in, in the topic. You can also marry the, the idea of the lunch and learn together with some of those resources. One of the things that we did at a, at this other company I worked with is we set up a lunch and learn session where we actually went through the CSSLP. This is the Certified Software Lifecycle Professional certification from ISC². It's pretty high level. It's got a lot of detailed things. We actually set up a lunchtime session where we would, at a certain cadence, we would go through a chapter per couple of weeks or so. and really dive deeply into studying that thing together. So that's another example of the community angle.

16:40Robert HurlbutOkay. So then at the end of that, just as an example, that certification you took people through, so even if they didn't maybe go for the certification and take the test and so forth, they at least had common knowledge. Everybody had this knowledge, same page if you will. And so then everybody's more or less ready to go and knows what, you know, some of the basics were, if I understand correctly.

17:08Chris RomeoYeah, definitely. So, and not everybody went and took the test after, and that's okay. We kind of figured that was how it was going to be. But like you said, they went through the process of learning with us, so they got some positive things and impact came out of it. And I'm sure it changed the way they thought about certain things. So it was a win-win. Some people went forward like myself and actually took the test and received our CSSLPs. Others just took in the information and nobody was upset at the end of this and thought it was a waste of time. Everybody got some good stuff out of the process.

17:39Robert HurlbutOkay, great. Any other things that you did in your example there to build in security community?

17:50Chris RomeoOne of the things that is— and this is not something that you do in the beginning of building your security community. You have to get to a certain critical mass to be able to do this next thing. And that is begin to have internal security conferences. And you probably can guess why I'm saying this is not something you do very early on, because it takes a lot of people resources to pull off a conference. It takes a lot of— it takes a good amount of budget to be able to do it. But really, when you reach a certain maturity in your security community program, An internal conference provides so many different things. It provides— and when I say internal security conference, I mean the majority of the speakers are coming from inside your organization. So you may have a few, and what I used to do is I'd invite a few external people to come in and be kind of keynote speakers that could talk about things that were happening in the industry. But primarily, this is about how do we get our internal security passionate people an opportunity to take the stage and talk about something that they love and encourage other people to study that particular topic. It might be malware, it might be enhancements to the secure development lifecycle, it might be a case study of here's something that we did, here's how we applied the secure development lifecycle steps to agile a little bit differently, and here's the results, here's the positive impact. So it can be a lot of different things, but the point is, That internal conference is about how do you build up your internal people. One of the other things that happens when you're able to host a conference yourself is the networking opportunities are the biggest things because you really are giving people a chance to connect in a face-to-face manner and just have conversations. And it's just crazy what happens when people from different organizations, if you have a big company, you might have different business units and things, When different people from different business units are able to meet each other, have a conversation, it's— I heard so many stories in my time where 2 months later they would have some incident come up or some challenge they were facing, and those people would connect behind the scenes. And because they had already met face to face at the conference and established a relationship, they could then share code, or they could share— or they could help each other out to to improve things and, and fix whatever the problem was that they were facing because they had made that connection. So when I, when I run an internal security conference, and I'll tell you kind of a, one of my tricks of the trade, and, and I learned it from a friend of mine, Ove Model in Norway, you don't ever put tables in a conference room, in the room where you're hosting the event. Because Robert, you know, what do engineers like to do when they come to a conference if they have a table?

20:47Robert HurlbutSet up the laptop.

20:49Chris RomeoThey like to set up their laptop. And what a lot of folks like to do is they like to set up shop for the next— they'll spend the entire day without moving. So what I do is one, I don't put tables in the room. Okay, so if you want to work on your laptop, you're going to be uncomfortable with it on your lap. The other thing that I do is I randomize my schedule throughout the day. So there is no— I don't use the concept of a track. You— so basically, you can't come in in the morning and— if you love hardware security and that's the one thing you want to hear about, you can't come into that conference in the morning, sit your butt down in a seat, and then stay in that seat all day. Because what I do is I distribute the hardware security talks around to all the different rooms of the conference.

21:30Robert HurlbutOh, okay.

21:30Chris RomeoAnd I do this on purpose because I want you to get up and have to move. Because what happens when people get up and have to move about in the hallways?

21:39Robert HurlbutThey see other people.

21:40Chris RomeoAnd they start to have conversations and they start to talk to people. So it's kind of like a forced socialization thing. And every time I run a conference, I'll get somebody— I get some of the feedback will be like, you didn't have tables in the room so I couldn't set up and work on my laptop. And I just nod, say, yes, yes, you're welcome. That's entirely intentional. And I'll take the flak from a couple of people because those people were forced to get up, move around, and actually have a conversation. with another human being about security.

22:06Robert HurlbutWow, that's some great ideas. So let's say, for example, I mean, I like the internal conference idea, but let's say it's a company that says, you know, we don't have the resources, we don't have— that's a big thing for us to do. Are there some other suggestions, like for example external conferences that they might not necessarily sponsor but recommend that a few people go to and then come back and share? Any thoughts on that?

22:30Chris RomeoYeah, so 2 things that— 2 different kind of directions that I'm thinking about there. Certainly, there's lots of external conferences available around. There is— OWASP has user groups in almost every major and even minor metropolitan area. ISC² has monthly chapter meetings where you can go and listen to people talk about security, give talks and things. Yeah, I think, and then there's also the external, the big external conferences like AppSec USA, Black Hat, DEF CON, RSA. You can go to those types of things and bring things back. But for the more budget-conscious company, you're going to be probably, you're going to be better served by looking for what meetups exist in your community. I know here in the Raleigh, North Carolina area, we probably have like 6 different meetups that have some type of security connection to it. There's an OWASP meeting, there's an ISC² meeting, there's ISSA, there's ISACA, there is something called Security Beers, which has no affiliation. What happens is they meet at a local restaurant one evening during— once per month in the evening after work, and people just get together, have a beer, and talk about security. from lots of different companies.

23:51Robert HurlbutOh, okay.

23:52Chris RomeoSo, it doesn't have to be super formal as well. So, that's kind of my thoughts on the conference and the external side. I'll offer one encouragement for smaller organizations that might say, ah, you know, I can't have a conference. That's going to cost a lot of money. Nobody's going to give me space or time or whatever to do it. I just spoke at a gigantic company and they chose to do their conference as a virtual conference. And so what that means is they used a web conferencing solution and they had the different tracks in different web conferencing calls and people could just connect into the talk that they were interested in and that provided kind of a virtual way so that— and then they recorded all the sessions so that people that were in different time zones could actually consume those presentations, listen to them at a later time. But this was a big company.

24:44Robert HurlbutWow.

24:44Chris RomeoThis is not a small— this is not a tiny little company. Even big companies sometimes choose to do it this way just to save on travel and things. So yeah, I mean, you can set up a— it doesn't have to be a week-long event either. The first time you do this, it should be 4 hours, 8 hours maximum, because you'll get an appreciation for how hard it is to fill 8 hours of time in a structured and a way that people actually enjoy the overall event.

25:12Robert HurlbutRight, because you want to— you're taking some people's time to do this. Not everybody wants to go to— it's another meeting, it's another thing I got to go to. But if you can get bite-sized chunks, then that's, I think, a much easier thing to serve and people to enjoy.

25:31Chris RomeoYeah, and then if you want to get crazy with the virtual style conference, there was just a conference here a couple of weeks ago called All Day DevOps.

25:40Robert HurlbutYes.

25:40Chris RomeoThey actually ran this thing. I caught some of the early sessions. They had Europe-friendly time slots, they had US Eastern-friendly time slots, and then US Pacific-friendly time slots. They ran for like, I don't know, 20— I think it was 17, 18 hours or something.

25:57Robert HurlbutRight.

25:58Chris RomeoAll virtually, all using YouTube Live, so basically Google Hangouts that are then broadcasting the slides and the audio. So the point is, and they had like, I don't know, I think I saw 20,000 people or something that participated in that as viewers. So the point is, you don't necessarily have to bring everybody together. You can use technology, especially in the beginning, to be successful with a security conference event.

26:22Robert HurlbutWow. Yeah, a lot of great ideas, definitely. Okay, well, thanks, Chris. I really appreciate just sharing with me and us talking about Security Community today. Thank you.

26:34Chris RomeoYeah, you're welcome. Thanks for listening to the Application Security Podcast. Our intro music is 8-Bit Kung Fu by Boring and TJ, and the outro is Southern Delight by Stefan Cartenberg. You can find us on Twitter @appsecpodcast or on the web at www.appsecpodcast.org.