MITRE ATLAS - Defining the ML System Attack Chain

This week The MLSecOps Podcast talks with Dr. Christina Liaghati, AI Strategy Execution & Operations Manager of the AI & Autonomy Innovation Center at MITRE.

Episode Summary: 

Chris King, Head of Product at Protect AI, guest-hosts with regular co-host D Dehghanpisheh this week. D and Chris discuss various AI and machine learning security topics with Dr. Liaghati, including the contrasts between the MITRE ATT&CK matrices focused on traditional cybersecurity, and the newer AI-focused MITRE ATLAS matrix. 

The group also dives into consideration of new classifications of ML attacks related to large language models, ATLAS case studies, security practices such as ML red teaming, and integrating security into MLOps.

YouTube:

Transcript

Introduction 0:08 

Welcome to The MLSecOps Podcast presented by Protect AI. Your hosts, D Dehghanpisheh, President and Co-Founder of Protect AI, and Charlie McCarthy, MLSecOps Community Leader, explore the world of machine learning security operations, aka, MLSecOps. 

From preventing attacks to navigating new AI regulations, we'll dive into the latest developments, strategies, and best practices with industry leaders and AI experts. This is MLSecOps.

D Dehghanpisheh 0:38 

Welcome back to The MLSecOps Podcast, everybody. With me today is Chris King, Director of Product for Protect AI, my colleague, and co-host for today's episode. 

Today we welcome Dr. Christina Liaghati from MITRE. Dr. Liaghati is AI Strategy Execution & Operations Manager for the AI & Autonomy Center at MITRE and one of the ATLAS leaders, which is what we're going to be talking a lot about today. Welcome to the show, Doctor. 

Dr. Christina Liaghati 1:06

Thanks so much, D. 

D Dehghanpisheh 1:08 

So I guess the place to start is, tell us a little bit about MITRE's mission and how it has made a difference for enterprises and researchers and technology vendors. Talk to us a little bit about that.

Dr. Christina Liaghati 1:21

So, MITRE is in a pretty unique place in that we're a not for profit that works across most of the government sectors, really supporting and advising our government sponsors, understanding their unique missions and needs and bringing our deep technical expertise and really the network of 10,000 MITRE engineers, scientists and really technically brilliant folks forward when there's a need to work on those most critical and impactful problems across the government. 

Because we're also kind of a not for profit, objective 3rd party, we also get to work across industry and bring industry and government together, like we do with ATLAS, to solve problems that we know will impact the entire community of both the public and the private sectors. 

D Dehghanpisheh 2:05 

Awesome. So you mentioned ATLAS, but that's a derivative, I think, if I'm correct, against a broader framework that you have in terms of security and information AppSec, security and cybersecurity called ATT&CK. Talk a little bit about ATT&CK, and then I have a quick follow up question for you.

Dr. Christina Liaghati 2:23 

Absolutely. So, ATT&CK, as you know, has been a really foundational tool for a lot of the cybersecurity community. So when we were thinking about what to put together for the AI security space, which is really bringing that AI community and the cyber community together, we wanted to put it in the most familiar context possible, which meant directly building ATLAS off of kind of that inspirational platform and format that is ATT&CK. 

Quite a few of the tactics and techniques in the ATLAS matrix that an adversary might use to attack our AI enabled systems actually come directly from ATT&CK. So the two are meant to be used in concert with each other. Very complementary systems that help enable both the cyber and the AI community.

D Dehghanpisheh 3:07 

ATT&CK has kind of become an industry standard nomenclature and a syntax, if you will, almost like a taxonomy for understanding security threats and processes. 

Before we get into that overlay, what has made that one, ATT&CK, so widely adopted? What do you think is the secret to its success? 

Dr. Christina Liaghati 3:25 

I think it's definitely evolved a little bit over time, but I think one of the major reasons why it was adopted so quickly was because it provided a coherent language, a consistent model that the entire cyber community could use to describe and talk about these different cyber threats as they were popping up and becoming more common across the cyber community. 

Kind of before that point, people were using all different terms to refer to the same thing and they weren't really speaking the same language. That was kind of the similar situation that we were facing in the AI security community. Right. 

Like, there was adversarial machine learning, but then as people were starting to think a little bit more about where the broader AI security topic was evolving in general, we wanted to try and provide a consistent language for how to describe these threats, and really in that clear, consistent attack chain that a adversary might use to take advantage of incorporating AI into our broader cyber systems. 

Chris King 4:21 

Makes total sense. And kind of building on that theme, you mentioned the need for specific attacks that are unique to machine learning and AI oriented workflows. 

What are some of the unique challenges that you did see of ML workloads that ATT&CK really couldn't account for traditionally? 

Dr. Christina Liaghati 4:37 

Not only were adversarial machine learning attacks like something that was totally in a different realm than what was already captured in the traditional cyber perspective, but really the combination; I think we actually have a case study on the PyTorch dependency chain that we put up from the end of last year that really kind of talks to this a little bit, but it's taking advantage of the combination of incorporating an AI enabled or an AI system inside of your system of systems. Right. 

The vulnerabilities really come from the combination of the two, which is why it's more than adversarial machine learning or just cyberattacks. There's a significant number of things that an adversary can do, even just from like a reconnaissance or resource development or initial access perspective that is more complex and significantly more vulnerable than it might be in a traditional cyber system because the cyber community is somewhat familiar with like, all right, you don't put details out there of exactly what's going on behind your security procedures. Right. 

But in especially the AI community and so much of the models that we're putting out there or the massive GPT kind of takeover of the world right now, a lot of that is inspired by things that have been open sourced and are available for the community to look at, make kind of comparable datasets with, and help enable adversaries to really do much more targeted attacks that take advantage of the vulnerabilities that come from incorporating machine learning into those broader system of systems that the cyber community probably isn't thinking about in the same way, which is why it's so different. 

Chris King 6:14 

Got it. And when we're thinking about things that are open, while the underlying technologies might be very much open source, like PyTorch or TensorFlow, the exact models, even if you can shift them around ground, they're obviously quite a bit more opaque. That's quite complicated in research in general. 

So how is ATLAS similar versus different compared to ATT&CK? When we're thinking about some things that are opaque and some things that are very much just as transparent?

Dr. Christina Liaghati 6:39 

Things like the initial access or reconnaissance that an adversary might be doing to both get direct connectivity to the machine learning system that they're trying to interact with or taking advantage of the vulnerability in, those are also very consistent between the cyber world and the AI world. But I don't think we think about them as much. Right. 

So much of the community has thought about the brittleness of these models specifically. But, say, when you're putting out a press release about the types of systems that you're using inside of your deployed products or things that have a front end to them, you're actually putting out a lot of information that helps an adversary craft specific attacks that are tailored to your systems. Because even just telling them we're using a GPT model helps a lot because you can target very specific types of attacks in that direction. 

The other piece is that there's a lot of things like the establishing accounts piece, right? Like, you wouldn't necessarily think about the chain of events where an adversary needs to just get past traditional cybersecurity measures in order to take advantage of some of these vulnerabilities. But they are vulnerabilities because they can find a way to bypass the systems. 

One of the case studies that we talk about quite a bit was actually a $77 million theft from the Shanghai Tax Authority. And that theft was the result of two individuals, like not even nation state actor level here. Those two individuals were able to take advantage of the vulnerability in the facial recognition system that was using - they were basically able to create accounts using very static headshots, right? Like just basic photos of people's faces, creating a really crude video that was enough to bypass the facial recognition system and present that using a cheap cell phone that could have a modified video feed, like right. Instead of holding up a regular front-facing cell phone camera, they were able to present a modified video feed to the facial recognition system that, of course, had the ML model inside of it verifying that a person was who they said they were. 

But that whole attack chain was kind of enabled because they were able to purchase people's identifying information off of the black market and those static headshots. And then create those established accounts that gave them that privileged access to the system, where over the course of two and a half years, they were able to submit invoices and fraudulently get away with what they did, stealing $77 million. Right. 

So it's more like that broader chain of events that is so important for the full community to be thinking about. And I think that's why we're trying to kind of put it in the context of the ATLAS framework, which is directly inspired by ATT&CK, to show you that full chain in very clear, concrete, defined terms. 

D Dehghanpisheh 9:33 

In the example you were just giving us about the tax authority and the $77 million heist, was that a type of adversarial machine learning attack? Is that a particular type of attack that was used? Like, is it an evasion attack? 

Dr. Christina Liaghati 9:47 

They definitely did evade the model. Right. They were misusing the system for external effects. But it wasn't even, like, I don't think when the adversaries were developing that attack, it's not like they were modifying the photos to make them classify differently. Right. 

It wasn't necessarily like an adversarial patch laid over the top of it to specifically target the computer vision system. They were familiar that the face ID recognition system was vulnerable, but they didn't, I think, target the system in such a way that it was really a hardcore adversarial machine learning type of attack. 

They ended up evading the model, but it was just by creating a really crude video of a face slightly turning the eyes, opening and closing the mouth opening and closing, right. Very, very simple. Not even near as artistic as the types of overlays you see on some of the recent images that are designed to evade computer vision systems. 

D Dehghanpisheh 10:47 

When you think about that, are there new classifications, if you will, or new categorizations of how to think about threats in machine learning and artificial intelligence systems? And if so, can you talk a little bit about that? 

Dr. Christina Liaghati 10:59 

Absolutely. And I think that's part of where we're going with ATLAS, right? And ATLAS is still constantly evolving, especially with the new world of these GPT attacks and folks being able to take advantage so much more via things like prompt injection than we had been expecting them to even just six months or a year ago.

It's opened up almost a parallel pathway for attacks to be executed just via prompt injection. Kind of circumnavigating the original set of attacks that we've seen kind of deployed in the many different case studies that we've captured already. 

I'd say the matrix in and of itself and what we're seeing is possible and confirmed adversary pathways, both between those realistic red teaming attacks and adversarial attacks like executed by an actual adversary. Those are evolving pretty quickly.

And actually, even within the ATLAS matrix, we actually denote the case studies that are directly inspired or pulled from ATT&CK with a little red ampersand. So you can kind of see like, all right, the combination of a traditional cyber tactic here, in combination with the vulnerabilities that are coming from purely incorporating a machine learning system into your broader system of systems. That's where we're starting to see this evolving. 

I've talked about establishing accounts a little bit there, acquiring information about the infrastructure and information about the Say model that you're using, developing the different techniques to really take advantage of the physical environment, even that you're putting that model into, that all has a big impact on how these systems are being used and how they're vulnerable. And that's really where we're starting to see a lot of this evolve in an exciting way, because it means that we as a community can do a little bit more. 

Like, while we're thinking about being proactive here, right. We don't have a ton of regulation in place yet. We can really try to capture and describe what we're seeing here to help enable our blue teams to better prepare for it. 

Chris King 12:59 

You mentioned the GPT models, and we've seen, kind of, a rise of somewhat smaller but kind of community and somewhat open source large language models [LLM] that are being disseminated. You can kind of grab those, use those as a starting point similar to previous models, where you might have some bit of transfer learning to go further, and then comparing that along with just your own organic model that you might be building. 

So when we're thinking about machine learning operations as this large umbrella that encompasses all of that, specifically on the operational side, how are you thinking about specific threats to just that ML ops workflow itself? 

Dr. Christina Liaghati 13:36 

I mentioned the GPT world has kind of been taking us by storm. We just put up our first or most recent case study that incorporates the GPT element that I think really was directly enabled because of the explosion of the usage and adoption of these models. It's a MathGPT attack, and we've actually got several more GPT or large language model focused case studies coming soon because so much of the community has pivoted to really capturing and focusing on the ways that these systems can be evaded or eroded. And it's quite interesting to see how many of our industry partners are actually trying to kind of use and secure these new LLM based systems as we're working to really expand that part of the ATLAS matrix to better capture those new attack pathways. 

As quickly as the large language model field is changing, the attack pathways are also emerging right along with it. And it's really exciting to help people familiarize themselves with more of the ways that it can be misused or broken in this kind of rapidly evolving space. 

The MathGPT attack specifically was executed by a German master student, an external actor that really significantly impacted their system, but he was actually doing it in more of an exploratory way right so, actually, in that MathGPT study, we describe it as an exercise, not an incident, because he was not a malicious actor trying to take advantage of their system, but he was able to get access to their GPT-3 API key and could have executed a lot of charges on their behalf if he'd been really malicious about it. 

Even in some of the kind of collaborative or more exploratory work and discussions that were happening around that, he was also able to bring down their servers for a couple of days. So there's like a denial of service attack in there too. But again, because in a lot of cases, we're seeing these folks as not being malicious, more they're just trying to live- explore how these systems can be broken. And he was actively trying to tell the MathGPT folks as he was finding some of these vulnerabilities in their systems too. Right, so a lot of folks have good intentions, but we're also starting to see a little bit of the malicious stuff as well. 

D Dehghanpisheh 15:50 

Christina, ATLAS has, I think, somewhere between 16 or 18, maybe around 20 ish case studies in only about 27 months, which is pretty impressive. And you're mentioning this case study that's either out or coming out shortly. 

I'm curious, can you walk us through the process of selecting a case study? How would someone become a case study and become part of the ATLAS framework? And what are some of the key factors you consider during the process of whether that fits into the ATLAS type of model or not?

Dr. Christina Liaghati 16:23 

Absolutely. And the MathGPT case study is already out. That's our most recent one at the bottom of the list of case studies. We're actually just about to reverse the order so the new ones pop up [at the top of the web page], because a lot of folks are like, oh, I didn't even see that one down there. But yes, absolutely. 

The process to incorporate a new case study into the matrix is largely more about tying in something that's unique. As I mentioned, all of the tactics and techniques in the ATLAS framework are directly pulled from those public facing case studies. 

So rather than having a million GPT studies or attacks where they're all able to get GPT to write them a play on how to build a bomb or something, right, instead of having all of the case studies be more like based on how often different things are happening, it's more about capturing the unique tactics and techniques that the community needs to understand and know a little bit more about. 

So a little bit more about quality and less about quantity, if you will.

D Dehghanpisheh 17:22 

With that, are you seeing actually - as you said, ChatGPT or GPT just generally, is a model taking over the frontal lobe of a lot of people in terms of interest - are you seeing an increase in case studies due to the rise of these large foundational training models and coming up and other foundational models beyond large language models? Are you seeing an increase in those case studies into your group? 

Dr. Christina Liaghati 17:46 

Absolutely. That's part of what we're working on right now. We've actually stood up an entire working group, if you will, underneath ATLAS, mostly in our Slack space there, and in a lot more of our community conversations that are happening because folks are seeing these vulnerabilities crop up especially if it's different AI security organizations that are trying to kind of better prepare for or better prepare their customers for how to deal with these different GPT or large language model based attacks as those are kind of coming down the pike, or they're just interested because they're seeing it as a rapidly evolving space. 

The number of attacks that are being reported to us that are unique and will be incorporated into the matrix are significantly on the rise at the moment. And the amount that our ATLAS community is kind of shifting to focus on it has been significant in the last couple of months as well, because we're actually thinking about adding new tactics and techniques to specifically address these new attack pathways that are, like I said, kind of evolving really rapidly. 

D Dehghanpisheh 18:44 

Let's talk about that a little bit. One of the things you just mentioned was like this concept of tactics, techniques, procedures and things like that. 

How does MITRE and the ATLAS team, how do you distinguish between incident and exercise type of case studies? And could you provide some examples of each and help the audience understand how they are different in terms of their value to, say, a blue team in charge of protecting the machine learning systems?

Dr. Christina Liaghati 19:15 

It really has to do with the intent of the person executing the attack. If they have malicious intent or have a negative impact on purpose, we call that an attack. But if instead the person or people attacking the system are doing so to discover the vulnerabilities and they don't have a malicious intent to do harm for some other reason, we treat that as a red team exercise. 

But really the core element of when and how things get incorporated into ATLAS is them being executed on a deployed real world or realistically deployable like a mirrored system, like in some cases an internal organization. Like, you know, if MITRE or Microsoft is red teaming a mirrored system or a representative system, then that's another way to kind of safely explore a way to attack these systems without necessarily taking down our own organization's kind of front end. 

I can definitely see groups being hesitant to do that, which is why there are different ways that you can safely red team your realistic real world systems. It's definitely much more to do with the intent. 

Which is why, like I mentioned, that German master student who was able to get access to the GPT-3 API key and take down the server for a couple of days in the MathGPT case study, because he did not do anything malicious with it and was actively reporting it to MathGPT while he was doing it, we consider that an exercise. 

Chris King 20:45 

You're mentioning red teaming ML assets and ML systems, and we see that starting to pick up a bit and we're seeing more activity on it. But still, at least from our perspective, we haven't seen it as quite the same volume as we see for traditional systems. 

I was wondering if that's consistent with what you guys have observed as well and any thoughts on maybe why that is and why it's not quite as popular yet?

Dr. Christina Liaghati 21:07 

I think AI red teaming exercises actually happen much more often than we see, because red teaming cyber systems has been around for a long time, and we've seen quite a few organizations at the leading edge of deploying AI systems stand up their own AI red teams, though we may not hear a lot about what they're doing because it's such an internal-facing activity. 

For example, here at MITRE, we have an AI red team that works across several national security departments and agencies to provide Red Teaming support and help their government sponsors see what types of attacks could realistically impact their AI enabled systems.

But I think to your point, why isn't it happening as much as we see or why don't we hear a little bit more about it? I think it is largely because so much of the community is kind of figuring out their risk level here, right? Like how much of an investment should we be making into the AI red teaming space? Or how much should we adapt or modify our existing cyber red teams or cybersecurity group to focus a little bit more on these threats? 

Because they are significantly different and it requires a significant shift in thought and posture and activities to focus more on red teaming systems that have AI or AI enabled than it is just to red team existing cybersecurity systems. 

Chris King 22:23 

That makes sense. And you talk about this as an investment and better understanding your risk tolerance. So how should companies be thinking about increasing budgets or different allocations to probe their ML systems? 

Dr. Christina Liaghati 22:34 

If you're going to deploy a machine learning system in any kind of consequential setting where anyone might have a motive to misuse or attack your system, you should probably be thinking about a healthy budget for red teaming.

It can be hard to put a number on how much money the value of red teaming has saved or protected even just in the cyber space over the last 15 years, but I think it's safe to say that this kind of exploding AI market comes with a lot of risks and that it's - if an organization wants to be or remain a leader in AI, they have to be pretty serious about preventing their AI enabled systems from being misused or attacked because of the risk to their reputation or level of trust or even adoption of their systems. 

Chris King 23:18 

Understood. And certainly there's a tax agency that had a 70 plus million dollar penalty where red teaming could have helped. If we're now thinking about just, again, large language models or other foundational models and the [ML supply chain] around them, what should red, purple, and blue teams be thinking about in terms of penetration techniques for these kinds of ML systems? 

Dr. Christina Liaghati 23:44 

I think one of the biggest ones that we've obviously seen explode recently here are these prompt injection attacks, right? I mean, that opens up a totally new pathway. 

In the Math GPT example that I gave earlier, that entire pathway of attacks was enabled because they connected GPT with a python front end, and you were able to directly ask the system to execute things ad infinitem, right? Like, it was able to say, like, all right, continue calculating this forever. And that's what brought the server down for a couple of days, right.

So there's things that I think organizations are not realizing they're exposing or making vulnerable through prompt injection attacks or to prompt injection attacks as they're kind of standing up these different systems. And I think that's that's going to be a really interesting one for our community to continue to flesh out and define the new tactic pathways that an adversary might be able to use around these new LLM models very quickly here, or even just in generative AI as the whole field continues to kind of explode. 

D Dehghanpisheh 24:41 

Given MITRE's touch points into enterprises, government agencies, critical national security components, if you think about just the private market, one of the things you said was, hey, the amount of money that Red team has saved is basically almost incalculable. You kind of need to be doing it. And then we've talked a lot about unique techniques, procedures, things like just adding prompt injection and fuzzing elements into your natural AppSec, infosec processes - I’m curious.

I have a two part question. The first question I have is, are you seeing more companies actually do that, or are they just kind of skipping it and assuming that security is an afterthought? That's my first question. 

Dr. Christina Liaghati 25:28 

I think, fortunately, we're starting from a bit of a leg up compared to what the cyber community started with 15-20 years ago, right? 

Most folks, once we explain to them, like, what the risks look like here compared to their existing cyber risks, they're much more willing, I think, to think about it and prioritize it than the cyber community probably did as a whole 15 or 20 years ago. Which is really exciting to me because it means we might be able to actually take advantage of this proactive window before I think the majority of the malicious actors in this space have fully caught up even with the vulnerabilities in these systems. 

Because I think in a lot of ways it's easier to just take advantage of the existing cyber pathways right now. Or like even that in the $77 million case, right? They were mostly using those photos and taking advantage of the vulnerability of incorporating ML into that broader system of systems to traditionally get access and create privileged accounts, right.

We haven’t seen as many, yet, attacks that require such a significant level of sophistication on the adversaries’ part, where they're really understanding and specifically targeting the machine learning models in a very exquisite way. The level of sophistication of these attacks can vary widely, and we've seen so many low sophistication attacks be very successful. 

I think that's where we haven't seen the adversary have to evolve super quickly yet to take advantage of these vulnerabilities. But I think we're going to, and that's where I'm excited to see how many off the industry partners and how much of the community has pivoted to focus on this. 

I mean, if you just think about the number of startups in this space now versus how many were here a year ago, it's been kind of wild to see how much of the community has shifted to focus on this, even just getting ready for some of the different summit and conversation conferences coming up. 

We've even been having conversations with Johnson and Johnson, the Campbell Soup company, right? Like household brand name organizations that are starting to realize and think about how they can take advantage of AI within their broader enterprise context and at the same time thinking about how they can do that in a safe, secure, assured way. Which is why we're so excited to see so much of the community prioritizing and pivoting to focus on this right now. 

D Dehghanpisheh 27:49 

Alongside that, you just now talked about how some of these household brand names and what other people might call legacy brands are thinking in a very forward way about how to protect their AI and ML systems and their investments. 

We've talked about prompt injection attacks. Are there other things that you would guide blue teams and blue team budget owners to think about adding into their existing processes and frameworks to stay ahead of where we think these attackers are? 

I mean, that's a pretty rare comment that you gave, which is like, hey, private industry might actually be ahead of the curve at this moment in time, but they could catch up very quickly. 

So what are some of the other things that you think blue teams should be doing to improve a company's AI and ML security posture? 

Dr. Christina Liaghati 28:41 

I think it's exciting, like I said, to see these kind of household names start to pivot and focus on this a little bit more. But it's also kind of an indicator of how quickly AI is going to be adopted and deployed in so much more than just the big tech companies, right? 

Like Microsoft, Amazon, Google, Meta, they all have really cool AI security teams that are focusing a lot on this because obviously they've poured millions and billions of dollars into developing the AI applications that they're using as tech giants at the forefront of AI. But AI is becoming so much more accessible, it can be applied to so many different fields and in different ways across the community that I think it is very exciting to see so much more of the community focus on this. 

And like I said, we're in that little bit of a proactive window to where not much of the regulation has been applied yet, at least inside of the U.S. environment. And I think as the red teams and blue teams - because I mean, a lot of these bigger organizations that are thinking about incorporating AI already have cybersecurity teams that are thinking about some of those easy ways that their systems could be taken advantage of, but they may not have AI security teams yet. And I think a lot of them are going through the thought process right now. Of okay, do we bring in help to support this? Do we stand up our own AI red team? What does an appropriate investment look like? 

And that's where I think it's pretty important to bring the community together and have those conversations about what realistic risk your organization is facing. And that's where a lot of the ATLAS value and continued kind of collaboration is what we're so excited about, because not only does it help organizations, the pathway of an attack that could be very similar. Right? 

Like the point of grounding it in those real world case studies is to show organizations that have similar types of systems deployed. This is the way that this could impact your system. You can see the similarities in the ways that you're deploying and exercising AI, which means that it's helpful, I think, for the broader community both to see those case studies and participate in the conversations to better understand what risk they might actually be facing. 

Because if you're just trying to take on every single adversarial machine learning attack that all of academia is putting out every minute, you're not going to have a budget left for any cybersecurity work anymore. Which is not the solution, I think. 

D Dehghanpisheh 31:10 

You mentioned regulation at the top of that commentary and kind of like the U.S. lagging from a regulatory and public policy perspective, but one area that has been coming out are the multiple Executive Orders, particularly Executive Order 14208, which was really about kind of custody and bill of material chains. There's more coming in the realm of kind of forcing entities to understand their security postures across technology stacks. 

Do you see these types of quasi regulations or regulation and policy frameworks upstream that have not made its way into legislation or even other Executive Orders as sufficient for ML systems? And kind of why or why not? 

In other words, if you were to take Executive Order 14208 and apply that to an ML system, is it sufficient in MITRE's view, or do other things need to be added to it?

Dr. Christina Liaghati 32:07 

They’re a step in the right direction, but I expect to see much more before we're as mature as the cyber community. One of my biggest hopes for our collaborative ATLAS community is that we help to inform the future regulation to make it as effective and as efficient as possible. I think government is being appropriately measured in how they adopt and deploy and think about regulating these technologies because the space is evolving so rapidly. 

I'm actually really happy with how much of our even international community - we just stood up a couple of NATO exploratory teams to focus on this, one of which is centered around ATLAS. To think about how the broader community can inform each other on the different threats that we're facing, as well as tools and capabilities that we're finding effective to start combatting the different concerns that everyone is raising. 

Because really, we've talked a lot about security, but I feel like the broader assurance problem is what a lot of the community is starting to think about here. Because security is one element, but equitability, interpretability, robustness, resilience. I think a lot of that is going to come hand in hand and be much more of the kind of holistic assurance problem that I think a lot of the government regulation or even just standards and different expectations, regulations and requirements even, like the acquisition requirements, they're going to come down the pike, are going to be a lot more focused on.

So it's going to be more than just security, which is why the bigger community is, I think, rightfully focused on more than just what regulation has already been put out there, but it's what is coming and how can we be proactive in informing and making that as effective as possible? 

Which is honestly a lot of the exciting element of what the government partners that we've been collaborating with have raised here because they want to put out the most effective regulation that they can. 

I don't think they want to hinder or hamper any of the amazing exploration and developments that a lot of the industry community is really focusing on right now. But I think it's got to be a collaborative partnership to come up with exactly what needs to be laid down as regulation to keep us all in check, but without having to kind of hamper or hurt the exciting developments that we're seeing crop up across the broader industry space right now. 

Chris King 34:28

You mentioned assurance and that kind of goes in lockstep with things like attestation about assets and how, I guess, the provenance of how an ML model or system comes to be.

And certainly on the traditional software stack and traditional applications, we've got a couple of decades of experience building that and starting to understand how to quantify it. 

Are you seeing ML teams being able to step in and kind of adopt a lot of that initial kind of legwork and they can be really effective? Or are you seeing more challenges at porting some of those approaches to ML and maybe that's hindering some of this adoption?

Dr. Christina Liaghati 35:04 

It's definitely a mix. I think it's exciting that we're able to leverage so much of what the cyber community has already put together, right? Like having built ATLAS directly on top of the existing ATT&CK framework that I think is very helpful because it enables the community to adopt it fairly quickly. 

But I think we're also, especially in the broader assurance space, like you said, that is going to be much more of a challenge, right? Because trying to build a framework for equitability; that doesn't necessarily work in a tactics and techniques or adversarial perspective because that's much more about misuse or, you know, causing harm in an unintentional way. 

That's part of where our ATLAS collaboration is actually kind of expanding beyond security right now to kind of think about what we're doing in the broader assurance space. And that's where I think a lot of this gets much more tricky than what the existing cyber community has successfully implemented. So, yeah, we're leveraging everything that we can, but especially the broader assurance problem itself is going to be much more complicated and rapidly expand beyond what the cyber community has already helpfully laid out for us to build off of.  

D Dehghanpisheh 36:12 

Christina, before we go, what is your call to action for those listening both in the community and the vendor community and others who are supporting MLSecOps like Protect AI and others? What's your call to action for them, and then what is your call to action for enterprises and governments as those who are listening to this podcast?

Dr. Christina Liaghati 36:36 

Yeah, thank you. I think absolutely all of this comes back to getting involved in the community. Ultimately, I don't think any of us are going to solve this in a silo. None of us are going to be able to see the full landscape of what these threats or concerns look like on our own. 

That's exactly why we stood up ATLAS as a collaborative partnership both across industry and government. So I would highly recommend that, at minimum, if you're coming up to speed on this, you're at a very early stage of thinking about how to deploy AI; at minimum just be aware. 

Keep an eye on what's going on in the AI security community. Keep an eye on the ATLAS matrix. We've got a Slack space that - just keeping an ear to the ground on how this is evolving before you deploy your systems and end up putting yourself in a risky situation that you could have potentially avoided. 

And on the other side of folks that are, like us, much more involved in this community, that are really developing tools and capabilities to really arm the community to do something about it, I think that's where it's so exciting to have us actually involved in driving the conversation, right? Like MITRE doesn't develop all of ATLAS on its own, right? 

This is very much a collaborative development that we need folks involved in, like I said, the working group that's really focused on generative and GPT threats and concerns and expanding the tactics and techniques to focus on that. There's a lot of incident sharing work that we're standing up right now, some formalized mechanisms to actually share protected incidents and even anonymized incidents across our industry partners. 

Because I think that's the other piece. We don't have enough ground truth on where we're going with this yet. So if you are much more of an expert in this space, you're trying to figure out how to protect your own large, important, especially consequentially, applied AI systems, I think it's very important that you quickly get involved in our community and think about being part of the incident sharing community specifically. 

I think that's where we're going to get a lot more ground truth for what is coming, where it's going, what we can realistically do to our systems. And I think standing that up in a protected way is going to be really critical as the community starts to get that picture outside of our individual organizational silos. 

D Dehghanpisheh 38:54 

Well, on behalf of the MLSecOps community, we want to thank you for your time, Dr. Christina Liaghati. What a fascinating conversation. And for those listening, be sure to check out the ATLAS framework and all of the case studies and stay in touch, stay connected. And with that, we hope everybody stays safe and secure and go make sure you are testing your AI systems. Thanks again, Dr. Liaghati. Thank you for joining us and until next time, everyone be safe. 

Dr. Christina Liaghati 39:25 

Thanks, D. Thanks Chris. It was good to see you all.

Closing 39:27

Thanks for listening to The MLSecOps Podcast brought to you by Protect AI. Be sure to subscribe to get the latest episodes and visit MLSecOps.com to join the conversation, ask questions, or suggest future topics. We're excited to bring you more in depth MLSecOps discussions. Until next time, thanks for joining.

Additional tools and resources to check out:

Protect AI Radar

Protect AI’s ML Security-Focused Open Source Tools

LLM Guard - The Security Toolkit for LLM Interactions

Huntr - The World's First AI/Machine Learning Bug Bounty Platform

Thanks for listening! Find more episodes and transcripts at https://mlsecops.com/podcast.