Graham Holmes — Adversarial Machine Learning

0:00Chris RomeoGraham Holmes is the founder and owner of AOP Cybersecurity, whose mission is to enable organizations to create scalable and effective strategies for trustworthy outcomes. His career includes over 22 years as a leader at Cisco Systems, where he infamously served as my boss for a period of time. And before that, he served in the US Navy as a commissioned officer for 9 years. Graham joins us to discuss adversarial machine learning. We explore the threats and attacks in an artificial intelligence machine learning world and review solutions to address these challenges using trust as a foundation. Please enjoy this conversation with Graham Holmes. Are you trying to build a security champions program? Everyone is these days. One challenge of rolling out security champions is how do we educate all these new folks? Security Journey has your answer. We provide a security dojo environment with level-based security education that gives your newfound champions a path to follow. And the best part? It requires almost zero administration by you. Visit www.securityjourney.com to set up a demo and learn how you can use the Security Dojo to connect with your security champions. Hey folks, welcome to this episode of the Application Security Podcast. This is Chris Romeo, CEO of Security Journey, and I am joined today by Robert Hurlbut, my co-host. Hey Robert.

1:38Robert HurlbutHey Chris, yeah, good to be here. Threat modeling architect.

1:41Chris RomeoAnd so Robert, I— if I remember correctly, you in fact have some cybersecurity educational related news in the fact that you got a master's degree or just finished your master's degree in cybersecurity, right?

1:54Robert HurlbutI did, yeah. Master of Science in Cybersecurity, something I started 3 years ago. Very recently just celebrated that. Actually, it was the next day after what the graduation time should have been. I noticed that it was literally 3 years ago that I received notice that I was accepted.

2:12Chris RomeoWhere was the program?

2:14Robert HurlbutSouthern New Hampshire University. So it was all online. At the time, I thought I would take 2 classes at a time, which was our max, but I ended up taking one at a time. And so it took some time, but very glad to, to get through that and really, really enjoyed it.

2:30Chris RomeoYeah, that's awesome, and congratulations on that. And it's some, I guess, a word for members in our audience who are sitting out there thinking about, hey, I have a bachelor's degree that I got, I earned, what do I do next? There is a future in going through a master's program in cybersecurity. There's a lot of value in that. So I would encourage folks to take a look at that. So let's go ahead and switch gears now and focus on the topic that we have for this episode, and that is adversarial machine learning, something that I don't even have any idea really what that is. So I'm looking forward to understanding that. We're joined by Graham Holmes, who is someone that I know well, who I used to work with at Cisco. And Graham, we always start for our audience sake with the question, what is your security origin story? How did you get into this crazy world of cybersecurity?

3:23Graham HolmesThat's a great question. It's always fun to think about, uh, why I got into this journey because it can be an interesting journey at times. Uh, so 2005, uh, at the time I was, um, leading Cisco's Global University Research Program and I got a call from my boss. Uh, that was kind of strange. He says, I need you to get on a call. 'Cause we need your help in talking to some people to get them to do some stuff that they don't necessarily want to do. I said, okay, what's this about? It turns out that Cisco had discovered through a third-party researcher a flaw in their IOS operating system, not the Apple iOS, but Cisco's IOS. And It turned out that that flaw was pretty significant. In fact, deeper and more involved than the researcher actually found. In fact, it looked like this researcher had found a way that you could develop a worm that could have infected our operating systems and literally would have melted down the global infrastructure of the internet. So that launched a massive effort that was being done quickly, as rapidly as possible, to update every operating system image that we had at the time. So over 400, you know, ultimately over 400 different images were built to protect against this type of attack, and it led to a year-long effort to ensure that the vulnerability was mitigated in everything that we had posted up on Cisco's download site for its operating software. That kicked off a kind of a self-reflection is how do we mature our security practices so that we prevent or avoid this type of reactionary event in the future? And my role was to, during this transition period of building these things secretly, was to compel developers to make changes in their code base that they didn't understand why we were doing it and what the outcome was and convince them that we need to keep this very quiet, close— closely held until the solution was out there so that we could then announce to our customers what we had done to mitigate the problem. So the resulting of that is over this year-plus work, we also started to think about how do we prevent future occurrences of this happening. And my role was to come up and develop a thing called Secure Development Lifecycle. And we leveraged a lot of our partnership with Microsoft and looked at their SDL and figured out how could we take this and apply it to a company like Cisco, which is developing, you know, many different products, hardware solutions, software solutions as well. And so that became the origin story of Cisco's Secure Development Lifecycle, CSDL. which, Chris, I know you are very familiar with and very involved in making it a success, successful program across Cisco.

6:32Chris RomeoOkay, so our topic then is adversarial machine learning. And so I want to start by just having you give us a definition and explain what this word, set of words together, even means. So what is adversarial machine learning?

6:47Graham HolmesAbsolutely. Let's start with kind of the 2 high levels. What's AI and what's ML? Because that's what adversarial machine learning is focusing on. So, AI, if you think of it very simplistically, is the solutions for creating or replicating human decision-making and thinking processes artificially. So, you have an intelligent system that can take in data, understand what the data is telling it, and make a decision based upon that, right? A classic example is classify this image. Is it a person or is it a Is it a helicopter? And, you know, in our process, we look at different pictures and we come up with different things. Say, oh, that's classified as a human. That's definitely a helicopter. That's not even a helicopter, it's a jet airplane. I don't even know what it is, so I throw it out, right? Machine learning is that process of learning. You know, that how do I, you know, what does a human's face look like versus what's a helicopter look like? And how do I tell the difference between those? The algorithm that we come up with in terms of training to understand what is a helicopter, what is a person, is the machine learning process. The decision that, that data then is then may be used to classify, or as a, as part of a solution, say, hey, anytime I see a human face, I should open the door. Anytime I see a helicopter, I should probably keep the door closed. So that's That's a very simplistic look at what's AI and the machine versus the machine learning process. Adversarial machine learning, AML, is focused on how an attacker looks at and might attack these machine learning systems in order to affect the decision-making process in AI. And so adversarial machine learning is truly focused on what are the types of attacks And techniques that are— that an attacker might use? What are their attack objectives when they go after a machine learning or AI system? And how do we understand those unique attacks and how do we mitigate against those?

8:52Robert HurlbutSo it sounds like— and actually, I didn't even— I forgot to mention that part of my concentration in cybersecurity was on AI and machine learning data sciences. So very, very fascinating field for me as well. So adversarial machine learning is really understanding both those areas, right? AI and ML, and how they can be used, if I understand correctly, how they can be used to achieve a goal. Am I right in that, or maybe you can explain it a little bit more?

9:26Graham HolmesSure. So think about this as an attacker would look at an AI system. So let's go, let's make the example in this case. The system is— the AI solution is trying to look at data flow into a network to determine whether it might be anomalous, right? Whether it might be something that's unusual network traffic, whether it might be traffic loads that need to be adjusted, or whether it might be malicious traffic that's trying to ingress into the network. And so you'd come up with a solution to say, I want, you know, instead of having a human looking at this traffic and examining all the data to say, that's malicious, or that's normal traffic, normal normal behavior, you'd want an algorithm to be able to detect when the traffic was likely malicious and what actions it should take.

10:14Robert HurlbutMm-hmm.

10:15Graham HolmesSo, now think about this from an adversarial perspective. If I'm the adversary and I want to inflow dirty data into your network or hide packets of information coming in so that you don't see that malicious data, you don't see my malware or what have you. I want to understand how— what you're looking for, how you're looking for it, and whether I can bypass that classification system. So, you know, as the machine learning system is trained, that's— you have to start with a data training set, say, you know, here's how to classify good data and here's how to classify bad data. Here's what constitutes good data and here's what's bad data. Or, you know, I've used classification. Another one might be using numbers, regression. As opposed to classification. You, you want to say, you know, you train the data and then it learns how to classify. It learns how to detect this anomalous behavior. And as it sees new and different data, it compares that to what it's learned and says, is this more of the same or is this something different? And I need to shape my, my learning so I now have found a new anomalous type of behavior. So, as an attacker, I want to be able to fool that system. And there are a couple, many different ways I could do that. One of them is I could poison your training data, right? I could— if you think about how data training comes about, and how AI and ML systems are developed, I should say, a lot of this is built on open-source software, either data itself, training data, data libraries, or even processing frameworks that are used and shared.

11:56Robert HurlbutRight.

11:58Graham Holmesas you develop your algorithms and as you develop your AI solution. So one attack I might have is say, like, I know you're going to be using this training dataset, so maybe I can corrupt that dataset so that when you start to train, you're training with something I've already manipulated to hide things I'm going to do in the future. Or conversely, I'm going to poison your data stream that you're getting that's actually feeding your learned system and actually doing the real-life processing so that I can— maybe I can— I want to hide what I'm going to do later on, so I'm going to overemphasize certain data that you see so that your algorithm starts biasing towards, this is what I need to be looking for, or this is what I need to not look for. And then later can slip in an attack because I've trained you away from the types of things that I'm going to be doing.

12:49Robert HurlbutMm-hmm.

12:50Graham HolmesSo those are kind of— those are data poisoning techniques or data manipulation type of attacks. Or conversely, you know, I can't attack your dataset, I can't attack your— because you've protected it well, but I can learn from the output of your algorithm how you're deciding what you're doing with that data, and I can infer things about your algorithm that I might be able to exploit later on by attacking your algorithm in some ways through traditional sources and modifying some of your algorithm, or use that as a way I can train my own attack systems to prevent them from being detected by your algorithm. Or from a commercial perspective, maybe I want to steal your intellectual property. You know, I want to infer what your algorithm is doing so I can duplicate that and resell it.

13:44Chris RomeoGraham, is it full— can you reverse engineer Kind of what is it? It is— is it as far as to say you could reverse engineer? I heard you use the word infer a couple of times.

13:54Graham HolmesYeah, that would be— that would be reverse, you know, description of reverse engineering. It may— you may not be able to say, I can rebuild your algorithm, but I can understand what you're— enough about what your algorithm is doing with its input, with the output that it's looking at, that I can reasonably assume this is how it's behaving. I may not be able to recode it, but I can definitely say this is how it behaves and here's what I need to do in order to respond to that behavior.

14:21Chris RomeoSo I don't have as much access as I would like when I— you think about like decompiling Java bytecode, right? Doing reverse engineering of a Java application. If you have access to the actual compiled file, then you can take that and reverse engineer it by actually getting to the individual instructions. But it sounds like from an adversarial machine learning perspective, you're more since you're able to control the inputs and outputs, you can, by looking at the input and then the output, that's how you're getting to your reverse engineer, because you don't have the— you don't have access to the goodness in the middle in this process, right, as an attacker? Right.

14:59Graham HolmesWell, remember that the algorithm is taking output, is taking data, and from the data figuring out what the input should be to get that data, right? So it's not like an input/output in a data Sorry, in a computing process, it's like it's looking at different outputs and saying, okay, what's the algorithm that makes this guy— this thing a face and this thing a helicopter, right? And it creates this algorithm where it's trained to create an algorithm to detect that. So the attacker has to think about, well, I'm not sure I can reproduce the algorithm, but I can reproduce what I think it's doing in order to classify a face versus a helicopter. And therefore, if I can make my data look more like a face, because I want to get the helicopter through the door, that's— here's what I might do, because I understand how it classifies human faces. Maybe I can make my helicopter front end look more like a face by putting a nose on it.

15:59Chris RomeoSo if we think about application security and adversarial machine learning, Since this— a lot of our listeners are really focused on application security, but yet they are also people who like to learn about new things, and so that is why this is such an exciting topic for this interview. What do you see as the relationship between AML and application security specifically?

16:24Graham HolmesWell, when you think about what people are developing right now, so there has been an explosion in AI/ML solutions. A large part of that recently has been driven by taking human processes and activities and having machines do them. You're creating applications that are relying on AI systems to produce results. Think about business systems that you might have in your network that require a lot of interaction with humans to look at the data, to take an action, and to make a decision. Now, what IT departments are doing is they are developing applications using AI to make those same decisions, right? For example, you know, getting in job applications. That, you know, is something that we can keyword search, and we've automated a lot of that. How— what to do with some of those results and how to best get them to people, or how to best automate your offline transaction processes is something that an application developer using AI might simplify or automate that process and speed up decision-making so that the hiring manager has the right— can take the right actions to hire the right people.

17:50Chris RomeoRight.

17:51Graham HolmesSo, this— the applications are definitely tying into AI systems, and one of the examples I wanted to provide about some of the dangers here, and some of the trust issues that come up with this, is developing these applications without fully understanding the security implications, and even the fairness or trustworthy aspects of the applications that you're developing. So one application that was developed for the— by a developer has made the news quite recently. It's a company called Clear Vision AI and— sorry, Clearview AI. And with Clearview AI, this developer came up with an application he said would help law enforcement accurately identify suspects that were captured in some type of video recording capability, surveillance video, you know, an IP camera outside a business or whatever, and claimed that his algorithm had 98.6 to 100% accuracy. His application was being provided to law enforcement as a way to quickly identify suspects and bring them in And again, touting 98.6 to 100% accuracy. Well, when you look at this application developer, and what's happened is that there's many, many red flags that came out on this. One is, this was a developer with a very imperfect record developing secure apps to begin with, and he had many different applications that he had written that had clear security vulnerabilities in them. He contended that his application was 98.6 to 100% accurate, but the testing really had many accuracy flaws. So as he trained his system, there was many inherent biases in it. Specifically, some of the actual racial bias methodologies were found, and the accuracy flaws in being able to determine a person was who it was was not in that high percentage right confidence that he had.

20:08Chris RomeoOkay.

20:09Graham HolmesHe experienced a data breach that exposed his list of customers so that people knew what he had, who he was selling to, and what types of applications it was being used for. And his app was supposedly privately accessible only to his customers, but several different researchers found his Android application and another iOS package up in unsecured Amazon S3 buckets. So this application developer, trying to use AI as a way to help its customers identify suspects, had created a very insecure, very untrustworthy platform for his customers.

20:58Chris RomeoSo when we think about things like OWASP Top 10 and other application security principles and things, you know, we talked about secure development lifecycle earlier. So it's what it sounds like is the systems that we're using to deliver these AI/ML solutions are going to be— you're creating applications that are then delivering this higher-order function.

21:25Graham HolmesYeah.

21:25Chris RomeoAnd so you have all the same problems you have with any application. In that the systems that you're using to do this are going to have web interfaces for administration. And so, all of the same problems that we deal with with any standard web application are all brought forward into something where, just because we're using AI/ML kind of on the top line, we still have the middle of the stack kind of problems that we have to consider.

21:51Graham HolmesYeah, this is absolutely true. I mean, and let's not— as sexy as AML sounds in terms of, oh, something new and cool to talk about in adversarial attacks against our applications, the net is we are still going to have to deal with, as you said, the basic OWASP Top 10 in developing applications. You need to have a strong— you know, an attacker's got many ways to attack a system without having to be creating new, novel, innovative attacks against machine learning and still be successful, right? I can DDoS your application if you haven't provisioned it correctly.

22:27Robert HurlbutMm-hmm.

22:29Graham HolmesI can do an auth bypass if you haven't done secure communications capabilities in it, or protected passwords, or access systems. So those standard attacks will always be there and exist as well. What AML is really focusing on is what are the new novel ones that we need to be thinking about, so as we design and develop and architect these solutions in the future, we need to be thinking about those types of attacks because they will come. We know that as you harden your you're against the OWASP Top 10, as you harden using secure development practices and cut off that attack path, these new novel ones will become exposed if you haven't thought about that. And so that's what AML is trying to do, is get developers to start thinking about things like the data and how do you protect the data from attack, because that's critical in the machine learning side, and how do you protect your algorithms? from ATT&CK as well, because these have— unlike traditional products, there's some really unique aspects to AI/ML that are new and different that are going to have some impact on security industry that people aren't thinking about today. Let me give you one of those examples. What I worry about is if the AI system is making decisions for you. And you've created a flaw. Let's just say it's a defect, not even a security defect. That defect is now in an operating system, which is making decisions that may be impacting your customers, right? Who's making decisions based upon the output of your AI application. What if it's flawed? How would you know it's flawed, and how far back would you go in terms of the decision-making process to understand how the customer is impacted. So, you know, let's get— go back to our silly example again. If this business was all built on ensuring that helicopters didn't get through the door and only humans got through the door, at what point did the flaw result in helicopters flying in through the front door, and what's your responsibility for that? Because you weren't on top of the decision-making process because of the defect in your thing.

24:51Chris RomeoRight.

24:52Graham Holmesin your application. Now extend that to security. If there's a security vulnerability, how would you detect that? How would you know it's impacting decision-making process, and what do you do about it? And both of those 2 examples, defect or vulnerability, the implications for liability, the implications for trust are much more significant than I think we have in failures now in standard development and applications.

25:21Chris RomeoYeah, because when you think about a vulnerability in a modern web application, like, there's really 2 important— there's 2 important things that are— I guess there's 3 important things. There was when it was introduced into the code, when somebody found it, and when we learned about it. Right now, it sounds like with artificial intelligence and machine learning, you're entering a 4th dimension And that is, how far back do we have to go? How far back were decisions messed up, potentially, or we had bad decision-making occurring in this system as a result of that issue that we had there?

26:00Graham HolmesYeah, you're exactly right. Well, you hit when. And then I think the other dynamic in there, too, the 5th is, and what is the implication of that decision? And what kind of impact did it have on the business and/or other applications that use that decision to make as an input source or as part of what solution it was driving. So I think, you know, there's a real— in the past, as you said, if you had a defect, the thing didn't work right. Now, if you have a defect, not only did it not work right, but you didn't know about it, and it made decisions based upon its flawed thinking process. You know, in the past, it was a human who made that, and you could take pretty immediate action on that. In an AI system, it's like, when did it make that decision? How did it make that decision? What's, you know, even what's the forensic aspects of this that you've got to go through in order to understand what the decision was? I can ask you, Chris, what did you do wrong, and how do we fix it? Fix your decision. But how do you do that with an AI system?

27:06Chris RomeoYeah.

27:07Graham HolmesAnd when did you make those decisions?

27:08Chris RomeoYeah, I can't believe we haven't had a Skynet reference yet, a Terminator reference in this conversation. But until now, of course, of course, we asked, we asked the Terminator, we asked the T-1, like the year— I think it was a T-1. I don't remember what Schwarzenegger was in the, in the first movie, what his model number was. But yeah, we just asked him, hey, so how long ago did this decision adversely impact your ability to protect the world?

27:32Graham HolmesThat's right. When did the Skynet decide that dinosaurs or tic-tac-toe is a lot less fun to play with?

27:37Robert HurlbutSo, you know, we've talked about, uh, definitely importance of, of application security. Let's, let's go back a little bit to the AI/ML solutions themselves. We talked about one threat, the model inference, where I can figure out what the model is, and from there I can put together my attack. What are some other maybe common threats, uh, that to be known, uh, in— or that are well known, rather, in these solutions, or maybe even some that we've not talked about as much?

28:08Graham HolmesWell, listen, you know, let's think about it in this bigger picture. Um, again, let's focus on what are the unique things. There's a lot of, you know, standard types of attacks against any kind of application, but when it comes to the unique aspects of adversarial machine learning, You know, there's 2 sets of targets or techniques that are going to be focused. One is going to be on the training set, you know, that data that you teach the system how to tell the difference between a human face and helicopters and a helicopter. And then there's the testing aspect. So once you train it how to make the decisions and how to look at the data, the next thing it's going to do is it's going be constantly bringing in new data and using that training to actually make operational decisions, right, or do operational things. So the 2 techniques the attacker is going to take is, how do I affect the training process, and how do I affect the testing, the inference process, and what are the things I would go do? So, you know, in training, What I want to do is do, you know, 2 types of attacks. I want to go after your data, gain access to your data so I can see what you're doing to train, train, and I can hopefully understand how you look at the data and whether and how I might be able to evade my— have my attack evaded because you're not training using a dataset that's likely to see me. And then there's poisoning. on the training set, which, you know, I know you're getting data in, but I want to try to poison the data that you're using to train it. So it's going to come up with a bad solution. We talked a little bit just before. I'm going to manipulate the data that you have. I'm going to, I'm going to poison indirectly the data that you're getting by providing some data you would not expect to see or things, data points that are outliers that would cause you to adjust your— the accuracy such that you may not see me. Or I might flow you data that I want you to get so I can bias the decision-making that's going to go on in your training process. Then on the testing side, I got 2 things I want to do. I want to, you know, I want to evade. So I talked a little bit about that. How do I ramp up the frequency or down the frequency or cause large changes in your dataset so that people don't, you know, that you're not getting an accurate solution. You don't see me in the noise. So there's some things I can do that. And then I can also, as I mentioned, learn more about what you're doing and infer from that how you make decisions so I can figure out how to do things that are going to be outside your decision-making process. So those are the primary 2 things I do against your data. You know, I'd obviously also target your data flow. You know, I'd want to target the sources of data that you're getting and how I can potentially cut those data from coming off. So if it's a telemetry stream, is there a way for me to corrupt that or block that telemetry stream? Or I'm going to go learn about knowledge about how your system works, you know, through black box or gray box or white box. white box attacks. So those, those are the types of the likely attacks that you're going to see. Again, and the types of targets they're going to go after are going to be targeting your machine learning model, whether it's supervised or unsupervised or reinforcement learning, and different techniques that they might use against those. And I'll just say a lot of this stuff is still theoretical because we haven't seen a lot of real-life attacks yet. And so as we study these things and we start seeing some real-life evidence of how attackers are developing the trade, I think it's really going to influence the types of defenses and the consequences of those attacks in the future.

32:13Chris RomeoSince we've covered— I feel like we've covered the attacks well, and I want to make sure we spend some time on the solution into this. And so I'm going to jump ahead and ask a question about You know, what are some of the things that companies that are trying to roll out these types of solutions right now, how should they be addressing risk, ensuring trust? Like, what's the solution to this problem?

32:40Graham HolmesSo, one of the things I've proposed, and it brings together a lot of thinking in the industry and academia, as well as government policymakers, is you think about a trust framework that, as you develop, you need to have. One of them is, it has to be secure, right? Customers are expecting anything we provide them to be secure. So, is your solution robust and resilient? The standard OWASP mitigations and things like that need to be looked at. And as you understand unique attacks against your machine learning processes, you need to come up with mitigations against that to ensure that they're robust and resilient. The second thing is, there's an expectation that what you've created is explainable. that the results that come out of it comply with or conform to what you expect to see, that they're— that you can prove that that result is logical and likely to have been a correct decision, right? So they're going to want to see proofs of that. How do you provide tests to show them that this is— explains how it works and how the results match what you expect to see?

33:48Robert HurlbutOkay.

33:48Graham HolmesYou know, maybe there is bias in the system because you expect that. You want to have bias against certain types of traffic, but you want to be able to illustrate that this is the bias we expect and this is the bias we actually see. You know, the third element is critical. Is this solution fair? And this one's probably the more difficult one. Fairness gets back to ethics. It gets back to cultural expectations. You know, we have different different ways of thinking things are fair based upon where we might live, the region that we're living in, or even our own cultural biases. So how do you explain that your algorithm is fair, that it's biased where you expect it to be biased, but it's impartial where you expect it and should be? Is it protecting human rights? Is it protecting your privacy? Is it protecting your data? So that's kind of the fairness algorithms and proofs that you need to be able to provide. And then I think the next 2 elements are kind of the backend processes that you need to expose, which is you're transparent about what you're doing. You share these results about security, the security of it, the explainability, the fairness of it. And finally, you're accountable to it. You know, when you find problems, you deal with them, you are transparent about the problem, and you hold yourself accountable to getting the solution— that problem affected and mitigated. So those— that framework, secure, explainable, fair, transparent, and accountable, I think those are key things that you should be doing no matter what, but have some unique elements when it comes to AI and ML. And I think the net of this is, is you want to ensure that your customers understand what you're doing to protect them, protect their privacy, their human rights, and the safety of their customers themselves.

35:41Chris RomeoAnd so you've, uh, you're branching out on a new venture here, Graham, to try to help the world solve this problem, right? And so, uh, what's the name of the company and kind of what are a little bit of the details about what you're trying to do as far as helping customers? Because it's obvious you've thought a lot about this, researched a lot, and can really help people that are dealing with these challenges.

36:06Robert HurlbutThanks.

36:08Graham HolmesSo, AOP Cybersecurity— AOP stands for Art of the Possible, built upon, you know, my years understanding that everything is breakable and it's just understanding the art of how it got broken— is the company, AOP Cybersecurity. And we're focusing on helping customers develop secure strategies or trust strategies as they implement AI/ML solutions in their companies. And it's about helping them think about this problem, not just as a security problem, but how it actually impacts all the different elements of the company. I mean, you think about some of the examples we've talked about today, decision-making processes and how you communicate those. It isn't just a responsibility of security, it's not just a responsibility of engineering, but legal. And supply chain needs to be involved in these things too, right? Understanding how to respond to vulnerabilities is going to change. This is no longer just a security problem or an engineering problem, but it's also how do we ensure that our tech support are trained to help customers understand how to diagnose potential flaws in our AI systems, and what do we do about it? Again, legal gets involved in this too. a whole company strategy that has to be coordinated and collaborated in a collaborative way if you're going to achieve those high-level outcomes you want to have, which is secure, robust, fair, explainable, trustworthy solutions that your customers can depend upon.

37:40Robert HurlbutSo here's a— and I know we're coming to the close here, but just kind of an off-the-script type of question. I've been looking at recently intelligent, what they call intelligent virtual assistants, which is an application of AI and ML and trying to take in data, you know, questions and either verbally or, you know, pictures and so forth and try to make decisions and then respond to a person, right?

38:10Chris RomeoTypically.

38:11Robert HurlbutAnd I have a question about, you know, we think about adversarial as somebody who's intentionally trying to break something, but what about the unintentional? And this is the threat modeling side of me asking the question. I recently heard about, and you may have too, about the family that had the Amazon, whatever that device is, Alexa.

38:33Graham HolmesYes.

38:34Robert HurlbutAnd the little girl, 5 years old, she saw her parents, you know, order things all the time while they were out. And while they were out, she ordered a bunch of dollhouses, and all of a sudden dollhouses started showing up at the house.

38:48Graham HolmesYeah, that's a great story.

38:49Robert HurlbutYeah, and so, uh, that was obviously not adversarial, uh, intentionally, but it certainly, uh, countered what, uh, the intent was.

38:59Graham HolmesNow, what you're getting at is another thing I think is a fascinating aspect of AI, which is the unintended consequences aspect of this thing, right? Which is, um, you know, it's one of the things that we're gonna have to have developers and the whole company think about is what is the expected consequences of this thing, but what are those things that you didn't think about? How do you— what's your process for looking at the solution and whether there might be a situation where mom and dad are using something to order, you know, groceries and the kids now interacting offline and you know, when they're not around and ordering dollhouses. You didn't foresee that. So what kind of framework do you need to have when you look at the solution and think about what are, you know, what is it you're expecting to do, but what is likely, what are potential things that are not expected or not desired? And that's adversarial. I mean, it may not, you know—

40:02Robert HurlbutRight.

40:02Graham HolmesAdversarial is not necessarily attacking it. It's thinking, you know, about what is possible with this system and having people who are creative and think about it. You need to have contrarians in your decision-making process. You need to have contrarians in your ethical decision-making process because you need to be thinking about things, you know, that aren't comfortable or that you hadn't, hadn't considered. You need some twisted thinking in some of these processes. And so I think when you think about this framework, there's also an ethical framework here that we've talked about with some in the past and, uh, and some things I was working on is how do you, how do you, how do you have people look at a solution and go like, okay, I know this is how it's supposed to work and I know what it's supposed to do, but are there some things it might do that I hadn't thought about, or there might be some implications for that technology that I hadn't thought about?

40:53Chris RomeoSo, Graham, are there any, uh, resources that you would recommend? Um, I, I think a lot of folks are going to be like me in that we haven't really thought about this topic a whole lot and haven't really— like, so are there any books or sites or blogs or anything, or government documents that are frameworks or something you recommend?

41:15Graham HolmesOh, absolutely. That's a— I mean, that is a really good question because there is a lot that's being developed in this country and Europe and other places. One of the frameworks that, you know, I'd highly recommend And I kind of talked about it, about the adversarial machine learning is something put together by NIST, N-I-S-T. And they have— they have the National Institute of Standards and Technology, I'm trying to remember what the acronym was. Go there and take a look at it. The US AI initiative that was launched in 2019 provided a number of incentives to help fuel thinking about frameworks, to think about R&D, about public and private partnerships.

42:07Robert HurlbutAnd they—

42:07Graham Holmesand there's a— they formed the National Cybersecurity Center of Excellence, and that's available for— it has many documents to helping companies understand the problem. But I'd start with— and NIST is one that's easy to remember.

42:22Chris RomeoYeah.

42:24Graham HolmesAnd I would, I would go there, and I'm sitting here trying to think about the other place that I would go to. Oh yeah, NCCoE, that's, that's the one I was trying to come up with, the National Center of Cybersecurity Center of Excellence, also has a number of things when you're wanting to think about adversarial machine learning, about adversarial issues, is another great place to go. And one of my favorite books on the subject, MIT's done a lot of work in this, they created a foundation a spinoff of Foundation, The Future of Thinking, I think it is, is the name of it. I know I'm going to get that one wrong. But there's a book called Life 2.0, and it's an awesome book as well. I talk about the possibilities of things and getting people to think about unintended and intended consequences of AI in the future. Uh, let's start with NIST, uh, if you want to get— understand the AML problems. And NIST, that NIST framework will also help you, uh, have some number of resources that you can go to as well.

43:31Chris RomeoThat's, uh, great. Uh, Life 2.0, I'm gonna remember that. Check it out. Um, so Graham, from a kind of concluding remarks perspective, if, you know, if we only let you give one key takeaway, coming out of this conversation for our audience? Like, what's, what's the thing that you really want them to take away and, and not forget as they consider adversarial machine learning in what they do in their normal jobs?

43:59Graham HolmesI guess the challenge I had, or key thought for folks is, is how is this solution going to be different for us and our customers? And what do we do to make sure that whatever the solution provides doesn't impact our trust relationship with our customer. You know, how do we ensure trust with a system in which there are new novel things that are impacting our business in ways that we haven't seen before? And, you know, trust is always going to be something that is critical for us to maintain with our customers. Thinking about the The novel aspects of AI and ML solutions means you've got to think about trust in new and different ways because you're going to be challenged in new and different ways with AI and ML.

44:51Chris RomeoGraham, thank you very much for being with us for this episode to educate us and our listeners about adversarial machine learning. And I know I definitely took away a lot of things I didn't understand before this conversation that I have a lot better perspective on now. So, thank you for sharing your expertise with us.

45:08Graham HolmesThank you guys for the questions. They're great.

45:10Chris RomeoThanks for listening to the Application Security Podcast. You'll find the show on Twitter @AppSecPodcast or on the web at www.securityjourney.com/application-security-podcast. You can also find Chris on Twitter @edgeroute and Robert @RobertGarcia.

45:30Graham HolmesRobert Hurlbut.

45:30Chris RomeoRemember, security is a journey, not a destination.