Risk Management and Enhanced Security Practices for AI Systems

Audio-only version also available on Apple Podcasts, Google Podcasts, Spotify, iHeart Podcasts, and many more.

Episode Summary:

In this episode of The MLSecOps Podcast, VP Security and Field CISO of Databricks, Omar Khawaja, joins the CISO of Protect AI, Diana Kelley. Together, Diana and Omar discuss a new framework for understanding AI risks, fostering a security-minded culture around AI, building the MLSecOps dream team, and some of the challenges that Chief Information Security Officers (CISOs) and other business leaders face when assessing the risk to their AI/ML systems.

Transcription:

[Intro] 00:00

Omar Khawaja 00:20

My name is Omar Khawaja. I lead our field security function at Databricks. I spent nine years as CISO of an organization that had 14 business units spread across financial services, technology services, and 14 hospitals just to make things super interesting. 

I sit on the boards of HITRUST and The FAIR Institute, and one of my favorite things I get to do is I get to teach at the CISO program at Carnegie Mellon University.

The last six or seven months have been focused on figuring out how to make AI security easier for CISOs. 

Diana Kelley 0:55

So, Omar, when we were preparing for this episode, you shared that one of the things that's important for security professionals to understand before they start thinking about AI risk management is “what is AI,” which seems almost like a reductive question, but I think a lot of people say they understand AI, but maybe they don't really.

So, what is it? 

Omar Khawaja 01:15

When I started at Databricks, if you asked me what I knew about AI, I would have said, I'm very confident I know what it stands for. And if you pushed me further, I'd say, well, I think it uses a bunch of data, and then magically these models are created and then you get a chatbot. Like, that's AI!

And so what I've found is, you know, when normally as a security professional, you talk to the business or you're talking to IT and they're describing a situation, or they're describing an ask, or a project. You know, our brains are just automatically going through and doing risk analysis and we’re threat modeling, and we're like, okay, there's this piece here, there's this piece here. They’re going to be connected here. There's a bunch of users. This is how the data is going to flow. I think that might be a concern. This might be overprivileged. Maybe we need an extra controller here. Maybe we need some segregation here. 

Okay, got it. So by the time the conversation ends, we already have an idea of the risks. We already have an idea of the threats. We've already figured out two or three controls. And if we have to bring in a security architecture engineer into the conversation, we can give them, sort of, an early version of a package of what the concerns are and what to do and essentially provide some guidance, which is, you know, the job of CISOs and cyber leaders.

And what I realized with myself is I could not do that. Any time someone was talking about AI and they were talking about, yes, we're going to have like some of this training data and then there'll be a gateway, and then we're going to featurize and there's going to be a – I’m like, I don't know what half of those words mean, and I don't know what the flow is, and where's my picture?

And for months I thought that there was something wrong with me because I moved into this new role and somehow I'd lost this ability. And then as I started having candid conversations with other CISOs, all of them pretty much admitted that they were feeling the same. In fact, I was running a workshop with a pretty large bank yesterday with about 20 of their execs, including many from their cyber team, and several of them said exactly that, which is we’ve lost.

We don't have that clock speed when we're dealing with AI like we do with everything else. And in many ways, the closest I can relate that feeling to is when I was an intern. Because as an intern, you come in, you've got new words, and you’re like I think I understand the words, but I'm not sure what to do with them.

I guess I'm just going to sit here and take notes and, you know, maybe in six months or a year or two years, like finally some of this stuff will connect and my synapses will start firing. But for now, these are just straight comments. I don't even know how to stitch them together, get them organized, how am I going to add value here?

And so as I went through that painful period and finally emerged from it, I then went back and with the power of hindsight, I said, well why was it so hard? And the thing I came to is I had no visual, I had no mental model. There was no scaffolding for me to take all of these different concepts and stories and architecture and to layer it on top of some mental model that was familiar to me that I could use as a starting point to say, okay, now I've got these gaps that I need to fill, let's move the conversation forward.

So, having that visual is key and that really is, you know, what are the components that make up AI? And by the end of it, we basically said there's 12 components that make up AI and we created a picture because what I started with is I would ask CISOs, can you draw me a picture of the AI?

And they couldn't. So I thought it would be nice for us to be able to say yes to that 100% of the time. 

Diana Kelley 05:05

Yeah, I love that you can talk about that. You know, you felt like you went back to being an intern in terms of level of knowledge. Because I find that too, you know, I've been in IT and security for decades now and it's pretty hard for me to feel like, wow, I am just really not getting it.

But that was when I first started getting deeper into AI and ML security. There was so much that I had to learn and I love that you took that struggle and turned it into a way to help others. The picture's great. I think we're going to talk a little bit more about that. That's from the Framework. Great. So we'll be talking about the Databricks AI Security Framework in a bit.

But you also, as we were getting ready for this, you had a great analogy about medicine and the field of medicine and how that can help, you know, newer CISOs to this space or security professionals. I wonder if you could share that with everybody.

Omar Khawaja 05:58

As I'd gone through this journey and it didn't feel like I did it methodically, it just felt like I just stumbled through and it felt a little painful.

I said, well, if I'm going to have others go through this and my goal is to make it easier for others so they don't have to go through the struggle. Instead of just looking at my own experience, what if I looked at the experience of others that are learning a discipline that's very complicated, because AI and AI security certainly isn't the first new discipline we've had to learn as humans.

And so I thought one discipline that probably feels like there's a long history of pedagogy around than many other disciplines is the discipline of medicine, because we've been trying to figure out how to stay healthy since the beginning of humankind. And so I looked at how medical students go through the process of learning all the things that they need to in order to become doctors that we entrust with our health and our family's health. And they start with anatomy. 

And anatomy is nothing but talking about what are the components that make up the human body. Subsequently, they move on to physiology, which is how do these parts of the human body work together? Are they part of some kind of system that work together, that have some kind of an overarching function? And then they talk about what's the epidemiology? What are the disease states that can cause the human body and particular components of the human body cannot operate the way that you want them to operate, and they can cause you some kind of harm or discomfort or prevent you from being your best self.

And it's only after that one of the last things that medical students learn is pharmacology, which is, what are the interventions? What are the drugs that you can use in order to either proactively stay healthy or react to a state of lesser health and return to that state of health. So when we looked at building out a framework for helping security leaders and in particular the CISO figure this out, we thought, let's make this as simple as possible.

And the goal was in like three hours we should be able to take an individual and as a result of that learning, they should be in the top 1% in terms of their understanding of AI and how to secure it. 

Diana Kelley 08:22

I love that. And looking at the body as a system and root cause analysis instead of just treating a symptom, understanding what's going wrong in the system.

So taking that analogy, how would we help CISOs understand risk management of the AI system? 

Omar Khawaja 08:39

Yeah, and you know, it's kind of like the human body where, yes, you typically go to a primary care physician, but if it's not something that is standard and that's basic. So if you have flu, or I guess now COVID is a pretty standard thing to have, or if you have a cough, or if you have a cut, like those things are pretty standard. Your PCP (Primary Care Physician) can address those ailments. 

The other part of the PCP’s function, which in many ways is the much more important and impactful one, is your PCP’s job is to help you figure out– if they can't help you, it's their job to help you figure out who you need to go to and to sort of be your care navigator, your concierge, your sherpa through that journey.

And they may say, We think you need to go to a rheumatologist. Oh, that didn't work. Maybe you need a hematologist. That didn't work. Okay, well, it looks like we pinpointed it to your kidney, so you really need a nephrologist. And so the same thing here, there's some things that are going to be basic and they're going to address the entirety of the AI system. The fundamentals, and those are what we call components 11 and 12. 

But the others are going to be specific to some parts of the AI system. So this is about something happening in the data ops space, or this is happening in the ModelOps space, or this is where you're actually getting ready to serve the model and make it available so an individual or another computer can actually interact with it. And so that becomes really important. 

Diana Kelley 10:07

Yeah, I love that. And you know, talking about the system, but the other components, I was wondering, what are your thoughts on supply chain security within AI and ML? Some companies are just starting to wrap their heads around that.

And what does that look like in practice? 

Omar Khawaja 10:22

Yeah, you know, over the time, as I've spent more and more time with it, I've realized that in the world of AI, we have many of the same types of risks and concerns that we worried about in the deterministic world with standard static applications. It's just the words we use are totally different.

So supply chain is a really good example. When we think about supply chain security in the traditional spaces, we think of it as I'm getting code from someone else. So I'm buying, or I'm accessing maybe an open source library, an open source API, or I'm getting some kind of an application. I'm using it. Can I trust that application or not?

So Log4j, SolarWinds, those are examples of I'm getting some code from someone else, I'm putting it into my environment and it may have the unfortunate opportunity to cause harm. And while yes, we can take that and apply that to the world of ML in a very simple way, we can say I'm getting my models from somewhere else.

That's true. But the other part that's also just as much about supply chain – but even for me, this was like an “Aha!” over the last few weeks – is the raw material that makes up applications is code. The raw material that you use to build a model is data. And so supply chain security in AI, it's yes, you should care where you get the model and provenance and all that, but you also care about the data.

Where are you getting the data from internal and external sources that you're using to now train the model, which essentially is to build the model. The model is literally built based on the data that you feed into it, so that data becomes of paramount importance. 

Diana Kelley 12:06

Yeah, and interestingly that's something that, you know, as we have assets and inventories within organizations, we're used to inventorying a piece of software. Okay, maybe a model, but it's going a next step to data inventory and starting to track and audit the provenance of the data and who trained with the data. Right? 

Omar Khawaja 12:24

Yeah. You're exactly right. So, you know, Diana, what we think of as supply chain security in the world of AI, many of us likely have heard the term training data poisoning.

Training data poisoning is a supply chain security attack. In hindsight, it feels obvious, but I was hearing those terms and keeping it separate from supply chain until I finally, finally connected the two together. You were talking about lineage and being able to track the data. That becomes important. You know, in many ways it feels like the core concepts that we knew were important in our traditional world are ones that are coming up over and over again.

And we're saying this is important. So just like you mentioned, you know, having the lineage and having versioning and knowing the ownership of our applications has always been something that has been sort of the ultimate vision of asset management. And oftentimes those types of activities were relegated to the long tail of controls that we never got to because we were prioritizing patching and authentication and logging and detection and hunting and all of those things.

And asset management felt like, you know, it's a nice to have. We know the regulators and auditors would really be happy with that. And yeah, there's some use cases where our threat team, and others, and our vuln team is saying it would be nice to be able to connect the dots, but it almost felt like those were a nice to have.

And sometimes that was out of sort of sheer practical reality, which is: I only have this much time and these controls just don't fit within the time that I have. With AI, these controls around lineage and asset management and tracking and treating data as an asset, those no longer are nice to have – they become must haves. So for instance, if I want to know why my model all of a sudden started to misbehave, I need to know, what version of the data was it trained with?

And if I just say I've got this data pipeline coming from these sources, well that's not good enough? Well, how do I know what data came from those sources? Did it intentionally poison the training data set and cause the model to behave in ways that it wasn't supposed to? Was it unintentional? Was it a quality issue? Or maybe our suspicion is wrong, and the reason the model misbehaved had nothing to do with the new training data that it got.

But if we don't know what training data it got when that caused that misbehavior to happen, we don't have that lineage, that tracking, that provenance, we’re in trouble. 

Diana Kelley 15:04

Yeah, that's such a great point. And you know, another thing in supply chain, what about the solutions that are running the AI and ML? You know, we still have a sort of standard software supply chain running that, right?

Or how is the industry doing at securing that? 

Omar Khawaja 15:23

Yeah, I think it still feels in many, many cases that it's, it's aspirational. We have a vision for where we would want to be from a software supply chain perspective. We want to have SBOM’s – a software bill of materials that's provided by every single software supplier that we have implemented within our environment and perhaps even some of the SaaS vendors that we're interacting with beyond our own internal environment.

However, how many organizations are able to take all of that information, that valuable information, that's in those software building materials, and to be able to digest it and to be able to instrument their detective controls and preventive and responsive controls in the security environment, to say, I now have this information and this should serve as input in my decision making around how I secure the environment.

That feels like that's the vision, that's nirvana. And we're still on a path to get there, and I don't think we're quite there yet in this case. 

Diana Kelley 16:27

Yeah, yeah, it's a process. And speaking of process, over your career as a CISO, I'm sure you've had to adopt and change. As you know, we've been through like wireless and the cloud and all these things that change everything, but ML And AI is truly different.

What kind of a cultural shift needs to go on with the CISOs in order for them to be able to embrace and really understand the risks that they're now assessing with AI and ML?

Omar Khawaja 16:57

I'll double click on what you said, Diana. The amount of change that AI and ML is bringing. In the Databricks AI Security Framework when we define AI, we define it as a mashing together – that's a very technical term – of three different subsystems. It's the world of data operations, it's the world of model operations, and it's the world of DevSecOps. And when I say model operations, what I really meant to have said is MLSecOps, because there's got to be security baked in, baked into that piece as well.

As you can tell, this is still a journey. Very much. But you know, the security teams have been really focused on this idea of DevSecOps. We were very comfortable with waterfall. I will confess as a CISO I really liked waterfall except when my applications had to use waterfall, but waterfall was easy to secure. It was very slow.

There were gates, there was bureaucracy, and the slower things move, the easier they are to secure, and the less likely they're going to introduce risk because they're just not moving as fast. And so as we're still trying to figure out this DevSecOps, then we go from waterfall, we go to DevSecOps, we go to CICD, and then we move all of this stuff to the cloud. And then this DataOps world kind of has been operating on its own, and the Chief Data Officers typically have their own shadow IT. They have their own infrastructure teams and you know, everyone knows they're doing this and they're told you're very important, you've got to get this done fast. And so the CISO, the CIO, others are saying, you know, we'll let you do this, we're not quite sure we understand your world, but just make sure you kind of follow these policies and these guidelines.

And yes, we'll run some tests and we'll make sure that the infrastructure is updated. Whatever is happening on top of that, kind of is a little bit of a black box. But we're going to hope everything is okay. That world is not a world that the average person in the security organization really understands. You know, there may be people in the security organization that came from the infrastructure team, maybe some that came from the application team.

I don't know of any that have come from the data team. Right? Maybe there are some out there, but that's got to change. And as that changes, we're going to bring more of that data expertise into typical security teams. That's going to be important. So the DevSecOps, data, MLSecOps, and the move to the cloud.

So you've got four different sort of movements happening and they're happening at the same time. That, I think, is what exacerbates it. If the move to AI just required one of those, we'd say, hey, this is going to be as big as the cloud, but it's requiring all for this. I routinely talk to organizations that are moving to the cloud.

Their first big use case for the cloud is machine learning and AI. That's the reason they're moving, and the fact that they're moving to the cloud, which is a really, really big change from being on premise and trying to figure out this whole AI space at the same time. That's not an easy thing to do. The metaphor that I often use is, if we think about the risk, we think about the concerns, we think about the controls.

Many of those are still the same. The nuances with AI are going to be one that you have to know where to deploy them, because the components of AI have words that are unfamiliar to many people in security, including me. I did not know what most of these words meant even six months ago. But the other is, the speed at which those risks are going to manifest themselves and at which the threats are going to come is going to be unlike what we've seen anywhere else.

And part of the reason is these four shifts. But the other part is the demand for the business to want to implement AI-based use cases is way greater than the demand from the business for cloud, or DevSecOps, or many of those other things, because those felt like they were technology and the technology team should figure it out.

Some sort of technically astute business leaders were pushing for this, but when it comes to AI, this is something that the average human being on the planet is interacting with. You don't have to be on the team to have a leg up and understand this. We're all sort of starting democratically at the same place. And so the demand to get to this is way higher.

But the metaphor I use for this is, imagine going from using a handsaw to cut a piece of two by four to using a chainsaw, both quote unquote, do exactly the same thing. I'm using a sharp object. I'm cutting the same piece of wood. However, the riskiness of the chainsaw is significantly greater. You know, my 12 year old, I will gladly tell her where my handsaw is, but I'm probably not going to tell her where I keep the circular saw.

And the reason is if she cuts herself with the handsaw, it'll be a learning event. She'll get a nick. That's it. The moment she gets a nick, she'll hurt and she'll stop. But with the chainsaw or with the circular saw, by the time she realizes she is, she did not place it in the right place, she's probably already lost a finger or two. 

That's not how I want her to learn and figure it out. You need a lot more controls to securely use a power tool, which is what AI is, than you need to use a tool that doesn't have that kind of power. And then to be clear, the power is both for good and the power, if not handled properly, can misfire.

Diana Kelley 22:44

And that is such a great point because the good can advance rapidly, but the bad can also – the malicious actor can also advance rapidly. And you made a point about bringing different groups to the table, the data team, data scientists, ML engineers, the DevSecOps, the security teams. My colleague at Protect AI, Charlie McCarthy, calls this “The MLSecOps Dream Team,” and we can see it, but getting people to come to the table and actually be that dream team can be difficult.

I'm wondering, do you have any advice in that regard? 

Omar Khawaja 23:18

Yeah, I think Charlie is– She's spot on. If you can get the data team, the IT team, and the security team to be on the same page, and aligned, and have each other's backs, it’s huge. You know, so much of this, Diana, is we learn from the science of change management.

And so if you're trying to drive change, it starts and you're trying to drive significant change, which is what AI is really going to require for us to do this successfully, you can't do it by yourself. There's the old Chinese adage that Confucius said, “if you want to go fast, go alone. If you want to go far, go together.”

And I think the stat around AI is that something like 75% of AI projects fail. So they're going fast. They're just not going far. The ones that do go far are the ones that are going together. They're figuring out how to take each other along. Some ideas on how to do that. One is creating alignment on the destination.

Where are we actually going? So if security thinks I want secure AI, and the data science team thinks I want ROI for this business process that is inefficient, and the IT team says I've got to do this while reducing my cloud costs. Well, then all three of you are going in three different directions. So to create the definition of success that all three can say, that is amazing. I am excited about that. 

And so then you can decide we're pointing in the same direction. We can all be sitting in different places in this boat, but we're all going to be rowing in the same direction and we're going to get there. And if I see along the way that cloud costs are going to be going up, I'm going to go to my friend in IT and say, hey, this feels like a concern.

How do we address this? Because I no longer think that's an IT objective. I think of it as this is my objective. This is our objective. So if we can replace me and I and my department and security with this is what the enterprise expects of us. This is what the customer expects of us. This is the imperative. This is tying it to some higher mission, some stakeholder that is common. 

A friend of mine would say, how do you make sure you're in the same foxhole together? Because when you're in the same foxhole together, you get alignment. And that is a beautiful thing. And so one is alignment. And the second, which was hidden in there is this idea of I am going to care about your success.

And if I show up and see data, I know you care about something that has nothing to do with me, then maybe the next time you will show up and say, Omar, I care about this. Not because of my department, but because of your department. That's how you build connections and relationships and bonds and start to smash silos that hold us back.

Diana Kelley 26:18

That is just so powerful. Yeah. See, it is our shared vision. Our shared goal. Yeah. Instead of everybody going in their own direction. So you've been running AI security workshops for CISOs for a while now, which is amazing. I was wondering if you could talk a little bit about what maybe has surprised you one or two things that have really surprised you as you've been running these workshops?

Omar Khawaja 26:43

You know, I ran one just yesterday in the DC area and I was very pleasantly surprised by the openness. And, you know, when you've got a senior, a director, a VP, or a chief next to your title, it's very hard outside of a one on one conversation to say, I don't know this, I don't understand this, I can't figure this out.

And so that gives me a lot of hope and confidence when I see senior leaders doing that, because the senior leaders are modeling the great behavior for everyone else in the room. And that's probably the most pleasantly surprising experience that I've had as I've been running these CISO workshops. 

Diana Kelley 27:31

Oh, that's great to know.

And yeah, it is. It's true. So so many of us are in learning mode and it's, you know, coming together and saying, yeah, we got to learn, and thank you for going out and helping to educate and staying in that space. You've been working on something pretty exciting. The Databricks AI Security Framework, and I was wondering if you could talk a little bit.

You showed the picture earlier a little bit about why at Databricks and you led this initiative and what you hope that others will get out of it? 

Omar Khawaja 28:01

Yeah, you know, Diana, it goes back to what you're saying it's about. It's about learning. And Microsoft had this pretty significant turnaround with Satya taking over the helm, and a lot of the analysts watching Microsoft and figuring out how sort of a legacy technology company could turn it around and now become a $3 trillion company.

They say one of Nadella’s secret sauces was he came in and changed the Microsoft culture from being a know-it-all culture to a learn-it-all culture. And that sort of is the ethos that we're trying to promote is don't focus on what you don't know. And if you do that, you will have insecurity.

You'll have confidence issues, and you will be tempted to pretend to know things that you don't. And that's probably the single biggest threat of whether you move to the cloud, you move to AI. Anytime you're going to new technology, there’s this threat of I think I know what I'm doing when I really don't. So you're better off saying, “I don't know,” and I don't know is a sign of vulnerability and management Science tells us showing vulnerability is the single best way to actually build partnerships and relationships in an enterprise.

And so when you say, “I don't know,” you open yourself up for learning, you leave that fixed mindset behind. You have that growth mindset. More people come in and say, Let us work on this together and help figure this out. And so the Databricks AI Security Framework is intended to be a tool to do just that. It's to accelerate people's ability to learn AI, the risks associated with AI, the controls, to mitigate those risks and to be able to do it together.

Diana Kelley 29:51

And who's the audience for this? Who should read this framework? 

Omar Khawaja 29:55

The audience is intended to be organizations that have a fair amount of complexity. They've got their high stakes organization that have motivation to do AI responsibly, to do it securely, and they don't really want to make every single mistake and learn from it.

They'd rather skip several steps ahead, learn from others' mistakes, and figure out how to do it together. So the idea is like this is built so that if you're on the data team, you're on the IT team, you're on the security team, you'll have a good sense of everyone is looking at the same picture. So we're helping create alignment just by giving everyone the same picture of the world that they're working in.

And everyone can add their acronyms to each of the 12 components that are specific to their organization. But the idea is it's one picture and it's not built by a silo, but it's actually built for the enterprise. 

Diana Kelley 31:01

Yeah, for the full coordination of the dream team. And where can our listeners find the framework? 

Omar Khawaja 31:09

Yeah.

So for now there's a blog post on Databricks.com with a sneak peek into the framework so you can see this picture in there. There's 54 risks that make up the framework, and the full framework with details on the 54 risks and the 52 controls that map to each of those risks and exactly how they map and then subsequently mapping to all the different frameworks out there that are regulatory or from standards organizations. That more detailed white paper, that will probably be a 60 plus page document, that will come out in the next couple of weeks.

I think I did a LinkedIn post and maybe I could send you a URL for people to sign up. Or you know what, let's keep it really simple. Send a message to cybersecurity@databricks.com and let us know you're interested in the Databricks AI Security Framework. 

We will make sure that you are going to be on our list and will be one of the first ones to get to see it.

Diana Kelley 32:09

That's great. That's really wonderful. And yeah, it's really great to hear about the cross walking within it because I think that there are so many regulations that are coming down the pike to have some idea where you're not going out and creating a new silo. You've crosswalked to existing frameworks, which is wonderful. 

Omar Khawaja 32:26

Yeah, and just to make sure that we're doing the same thing that we want everyone else to do, which is to have that learning and growth mindset and to be able to say, “I don't know,” as we've built the framework, so much of it is really the genesis of that has been the workshops we've done with the CISOs, the engagement we've had with who I think of as real AI security expert. Diana, that can say they've been doing this for more than seven, eight, nine months that I've been doing it. So getting people like Diana and others that have been doing this for many years to review the framework, give us feedback and us iterating through it, the graciousness and the generosity of those people giving up their time and sharing their insights has been awesome.

And without that, without all of you, I don't think this framework would be half as good or as useful. 

Diana Kelley 33:21

Oh, thank you. It was really, it was a great, great opportunity to be able to get an early sneak peek at the document and yeah, to anybody, what is it? cybersecurity@databricks.com – I strongly recommend, reach out to that email so you can get a copy of this framework when it is published. 

With that, Omar, I was wondering if we could just wrap with one question that based on all that you've learned, you've been on a really rapid learning journey. You've been talking now to a lot of CISOs about this. If you could distill out one thing that our listeners take away so that they can improve AI security at their organizations, what would be that thing that you'd ask them to do? 

Omar Khawaja 34:00

Yeah, Diana, the question that I get more often than any other is, is securing AI the same as securing systems that have come before it? And when I started, or even before I started the journey, my answer was yes.

And the reason it was yes is not because that answer came from a position of knowledge. It actually came from a position of ignorance, or better yet, it came from a position of convenience. Because if my plate is full, I've spent 25 years learning stuff. Do I really want to go learn a whole bunch of new stuff? I'd rather have the luxury and comfort of confirmation bias to say I don't need to learn new stuff.

Everything I know applies to AI and ML security. And so therefore, yeah, I'm going to find answers that meet that because who doesn't want convenience and comfort. And so over the several months that I started looking at it and paying attention, my initial reaction was, oh my god, this is so different. If it was the same as securing the traditional applications and systems, then I should know how to secure AI, but I don't know how to secure AI. So there has to be something wrong with my assumption. 

And so I moved to the other side of the spectrum. Where I am at today, and I'll preface this with, in six months I might change my mind again. But where I am today is as we look at the list of risks and we look at the list of controls, and particularly the controls, most of the controls are not ones that would be unfamiliar to a security professional that's been doing cyber for a few years. Authentication, authorization, logging, segmentation, access control.

There's nothing all that new and novel about it. Where I think the novelty element comes into, when it comes to securing AI, is where do you deploy the control? So, do I deploy this control on the training data? Do I deploy the validation data, on the test data, on the monitoring data? You lost me because I don't even know what the difference is between those data sets.

And do I really care? Isn't data just data? No, it turns out it isn't. In the world of AI, those data sets have very, very specific functions and meaning and value for a particular use case. So the short answer to your question is pay attention, not necessarily just to the controls that need to be deployed to AI, but pay attention to where in the AI system, which component the control needs to be deployed on.

And once you figure that out, securing AI is not that hard. 

Diana Kelley 36:57

Yes, that's great advice. And to everybody, to all of our listeners, thank you so much. Keep learning. Keep growing within MLSecOps. So, thank you very much to the continued support of the MLSecOps Community. Huge thank you to Omar for being here, for all you're doing to help advance the wisdom and the knowledge around AI security and for being a little vulnerable with us and sharing that, you know, it was an uphill climb for you. It was definitely an uphill climb for me. So anybody out there, if you're feeling a little bit overwhelmed by what you need to learn, it's okay. You will get through it. And now there are some really great resources like the Databricks AI Security Framework.

So thank you again, Omar, for being with us. And thank you so much to the MLSecOps Community for listening. 

We'll see you next time. 

Omar Khawaja 37:51

Absolutely. Thank you, Diana. Thank you for having me. Thank you for all that you do for the community.

[Closing] 37:57

Additional tools and resources to check out:

Protect AI Radar

Protect AI’s ML Security-Focused Open Source Tools

LLM Guard - The Security Toolkit for LLM Interactions

Huntr - The World's First AI/Machine Learning Bug Bounty Platform

Thanks for listening! Find more episodes and transcripts at https://mlsecops.com/podcast.