AI Security: Vulnerability Detection and Hidden Model File Risks

Dec 9, 2024 By Guest

Audio-only version also available on Apple Podcasts, Spotify, iHeart Podcasts, and many more.

Episode Summary:

In this episode of the MLSecOps Podcast, the team dives into the transformative potential of Vulnhuntr: zero shot vulnerability discovery using LLMs. Madison Vorbrich hosts Dan McInerney and Marcello Salvati to discuss Vulnhuntr’s ability to autonomously identify vulnerabilities, including zero-days, using large language models (LLMs) like Claude. They explore the evolution of AI tools for security, the gap between traditional and AI-based static code analysis, and how Vulnhuntr enables both developers and security teams to proactively safeguard their projects. The conversation also highlights Protect AI’s bug bounty platform, huntr.com, and its expansion into model file vulnerabilities (MFVs), emphasizing the critical need to secure AI supply chains and systems.

Transcript:

Intro 00:00

Madison Vorbrich 00:07

Welcome back to the MLSecOps Podcast, brought to you by Protect AI. I'm Madi [Vorbrich], your host for this episode; the Community Manager for huntr.com, Which is the world's first AI ML bug bounty platform powered by Protect AI. And today is going be an interesting episode because I'm actually with my two favorite coworkers here at Protect AI, Dan McInerney and Marcello Salvati. So, do you guys want to just tell our audience what you do here at Protect AI?

Marcello Salvati 00:36

What do we do here?

Dan McInerney 00:36

Please, you first.

Marcello Salvati 00:38

Okay. So no, I'm Marcello, I'm a threat researcher here at Protect AI. So that involves triaging reports, understanding the vulnerabilities, also automating a lot of the backend stuff with the bug bounty platform that we run here at called huntr. And this is Dan McInerney. Dan?

Dan McInerney 00:55

I do exactly the same thing. We're in the same role, so there's not much need for double introductions, but yeah, we just kind of handle the cutting edge research in the AI sphere, both with AI and security of AI.

Marcello Salvati 01:11

Yeah.

Madison Vorbrich 01:11

Awesome. So I'm just gonna dive right in and open up the conversation to Vulnhuntr, which is the Python static code analyzer that leverages LLMs like Claude to essentially find and explain vulnerabilities; which you two actually created, along with Ethan [Silvas], who's another threat researcher at Protect AI. So, can you kind of explain what inspired you to start it? Was there a problem that you saw and a need?

Dan McInerney 01:36

Yeah, so this actually started right when ChatGPT dropped. He was actually the one that told me about ChatGPT. He texted a group chat that we have and was like, "hey, you guys seen this thing?" And I check it out - I've been working at Protect AI for six months or something, I don't know, don't quote me on that - but I check it out. And it was like, whoa, this thing is insane. Because just literally the day before, we didn't have a chat bot like this and so as soon as I saw it, obviously being hackers, the first thought is -

Marcello Salvati 02:07

Can we use this to find vulnerabilities?

Dan McInerney 02:10

Can we destroy things with this?

Marcello Salvati 02:12

Yeah, yeah, yeah.

Madison Vorbrich 02:12

Very natural. Very natural.

Marcello Salvati 02:13

Yeah, exactly. So, we, like I, everything, everybody in the security community just randomly started copy/pasting code into it to see like, you know, if it can find some interesting vulnerabilities. And at the time of GPT2 - well, was GPT2 the first release, like public release, I forget. Well, in any case yeah, GPT2 or something - it wasn't that like good at it per se. It's kind of, ChatGPT is actually still not kind of good at it, honestly, to be perfectly honest, Claude is a lot better. But the, but basically the further along the GPT releases came out, the better it kept getting. And currently it's at a point where we're able to like, actually where we were able to actually build a tool that actually is able to find vulnerabilities just using LLMs.

Dan McInerney 02:59

Right. And so there was a <laugh>, there was a lot of reasons why Vulnhuntr wasn't released earlier by anybody else. It was actually Claude 3.5 that was the release that made it really possible to do autonomous discovery of vulnerabilities because we, the title we're claiming with Vulnhuntr is that it's the world's first autonomously found AI powered zero days. Meaning you just give it a project, you just dump the project, point the tool at that code, and it goes and will just tell you if there's any zero days. You don't have to just copy and paste snippets. You don't have to say, "Hey, this code looks suspicious. Let's, let's drop that in." The reason for that was context windows. So ChatGPT 3.5 came out and it was only a context window of 1,024 tokens, I'm not mistaken.

Marcello Salvati 03:47

Yeah, that sounds all right.

Dan McInerney 03:47

Which is just not enough, like, space to fit a complex vulnerability chain.

Marcello Salvati 03:54

So, you had to do like complicated things with like RAG systems and bunch of other stuff to actually get the thing to work, which it makes it a lot more complicated. And also the results are dramatically less good.

Dan McInerney 04:06

Less good, yep. Less

Marcello Salvati 04:07

Less good, yes. <Laugh>

Dan McInerney 04:08

Definitely less good.

Marcello Salvati 04:09

Yes. Less good. Dramatically less good than than if you were just like to paste the, like the actual vulnerable snippet directly into the context window.

Dan McInerney 04:19

And so what's interesting is right after Vulnhuntr dropped, Google dropped a Project Zero zero-day, where they found -

Madison Vorbrich 04:27

Oh, that's right.

Dan McInerney 04:27

- amemory corruption bug in SQLight.

Marcello Salvati 04:30

Oh, yes.

Dan McInerney 04:30

Which they then said was, you know, the first AI found memory corruption bug in popular software.

Marcello Salvati 04:39

Yeah. "In the wild" quote unquote.

Dan McInerney 04:41

Yeah. Which, I mean, Vulnhuntr found 14 "in the wild" in the previous like three or four months.

Marcello Salvati 04:46

But just to make a distinction though, like Vulnhuntr doesn't find memory corruption bugs.

Dan McInerney 04:49

Right.

Marcello Salvati 04:50

It only finds like web application, like your standard OWASP Top 10 kind of stuff.

Dan McInerney 04:54

Yeah. So what we're seeing right now is I think we're gonna see a big explosion of these tools that can automatically find vulnerabilities using LLMs. And right now, obviously all of them are kind of specific, specific to <laugh> something. Like ours [Vulnhuntr] is specific to Python. Google's was specific to memory corruption bugs. I think in the very near future there's gonna be a big tool, I suspect maybe by GitHub is probably in the best position to do it.

Marcello Salvati 05:21

For sure.

Dan McInerney 05:21

To have a generalized tool that you can just feed a project and an LLM goes through and finds vulnerabilities. The reason that this, that, like we did this to begin with is because LLMs are really, really good at taking input and then reasoning about it, and then giving you some kind of structured output. So before Vulnhuntr, there was static code analyzers, obviously. There's things like Snyk, there's things like -

Marcello Salvati 05:42

Semgrep, Veracode.

Dan McInerney 05:45

Veracode, yeah. The problem is they just kind of read like lines of code at a time. They don't have the context of the whole project.

Marcello Salvati 05:52

Yeah. I mean, and mostly it's just Regex (Regular Expressions) too at the end of the day. It's like literally just like trying to find specific code patterns that are hard coded into the actual software. And there's no reasoning behind it. Like, unless you're writing it in a sandbox, which sometimes these products have but are generally not that good. So like, this is the first time where, like you said, like they can reason about, like, it's not just like the code. It's like, "Hey, where does the input come from?" And like, the LLM can make these determinations by giving it more and more of the project's code, which is what Vulnhuntr does basically, which was an entire process of actually getting that down. I don't know how far we wanna go down that road. <Laugh>

Dan McInerney 06:27

I'm happy to go into the technical details later, but this was supposed to be the intro and now we're like seven minutes in. <Laugh>

Marcello Salvati 06:31

Yeah. Months of No, no.

Madison Vorbrich 06:33

This is, this is great. Were there any like "aha!" moments that you had when you were developing it where it kind of just clicked and you were like, oh, this is a game changer?

Dan McInerney 06:41

Yeah, definitely.

Marcello Salvati 06:43

Yeah. I think the first time I found the vulnerability; that to us was just like, whoa, okay.

Dan McInerney 06:46

That was pretty cool.

Marcello Salvati 06:47

This actually works. First off, that was the big thing. And then once Ethan [Silvas] started running it on a bunch of stuff, like, he started finding, like, he found like 12 of them, like back to back in like a span of like a few hours, I think, right? Or at least like a day or two. Like it was very, very fast.

Dan McInerney 07:04

Yeah, he just started pointing it at things and zero-days started falling out. There was some other stuff that we did that like kind of let us know that it was even technically feasible, one of which was - well, it was actually, I said this earlier - it was the fact that Claude 3.5 came out. And then I put a bunch of code into Claude 3.5, and it was actually really, really good at analyzing the code. And that was when I was like, okay, wait, I can make a weekend project into this. Let's see if this can actually find vulnerabilities. And in that weekend project, I pointed it at a project; I just whipped it together on something really simple, and it found an 0day in a GitHub project with like 200 stars. So obviously, I'm not gonna go report that to MITRE. You know, we're not gonna get a CVE for a 200 star project. But it like was an "aha" moment that this was possible.

Marcello Salvati 07:49

Working. Yeah. Yeah. And then I found something in like a 10,000 star repository.

Dan McInerney 07:54

Yep, and that was like, okay, well now this is, now we're talking. This might be now a long project.

Marcello Salvati 07:58

Now we're cooking, you know, now we're going. And so it just kept going from there. And, and I think like the biggest, what is it, the biggest, it was like 60,000 stars was like the -

Dan McInerney 08:05

65,000.

Marcello Salvati 08:07

65,000 stars. Right. Like the biggest project that I found vulnerability in. So yeah, it's definitely, definitely produced some results. And we actually have code to back this up as opposed to everybody else's approach up until this point, where it's a whole other rabbithole.

Dan McInerney 08:23

Oh yeah, we should talk about this.

Marcello Salvati 08:23

Yeah. Where the academic papers tend to - because this is not a new idea. Like this has been been researched and an active area research since ChatGPT came out, like the LLMs started, you know, hit the scene. - But the problem is, like all of the research so far was in academic papers who like mostly gave long-winded explanations of like, architectures of how would I build this if I were to actually build it? And they never released any code or proof of concept or anything along those lines. And a lot of it was sort of like misleading as well. A lot of the way they titled the papers and stuff like that. So it really bothered us and Dan specifically, like it really bothered him <laugh>. And then, so, and because of that, like, we were adamant that not only are we gonna build this thing, but we're actually gonna release it to make sure that hey, this is actual code that you can run and actually like find vulnerabilities.

Dan McInerney 09:13

Right. And to be fair, so when I was reading all the academic papers, to be fair, I don't necessarily think the titles were misleading, but they are attention grabbing, if you know what I mean.

Marcello Salvati 09:24

Yeah. Yeah. It's like click baiting.

Madison Vorbrich 09:25

Give us a sense: is there an example of one that you might remember off the top of your head?

Dan McInerney 09:29

Yeah, there was one about AI agents. So technically the titles that were most misleading were actually the news articles about the academic paper.

Marcello Salvati 09:40

About the paper. Yeah.

Madison Vorbrich 09:41

Oh, okay.

Dan McInerney 09:42

So the journalists would kind of extrapolate, and I remember one of the titles from the journalists was "AI Can Find and Exploit Zero-Day Vulnerabilities." And so when we released Vulnhuntr, that actually got brought up a lot by people saying, oh, this wasn't the first time that AI found autonomous zero-days, because look at this news article, but that news article was referencing a paper that just found that AI could find one-day vulnerabilities, like that it wasn't trained on. So they gave AI a whole bunch of [vulnerable code] that they knew was vulnerable - and everyone knows was vulnerable because it had a CVE - and told the AI to find vulnerabilities in it, and the AI found it, and the AI was not trained on that code. Meaning that's brand new information to the AI and it proves that AI can technically find zero days, but it didn't actually get any CVEs or find zero-days.

Marcello Salvati 10:30

Yeah.

Madison Vorbrich 10:31

Mm-Hmm.

Dan McInerney 10:31

So, to be fair, to the academic authors, that's not necessarily their fault that the news articles were headlined like that.

Marcello Salvati 10:37

They were very clickbaity, yeah.

Dan McInerney 10:39

They were definitely clickbaity.

Marcello Salvati 10:40

Yeah. That's, that's part of the problem.

Madison Vorbrich 10:42

So would you say that Vulnhuntr also teaches developers in the process? It's kind of like a, I don't know, like a two-way conversation, or it benefits both sides?

Dan McInerney 10:53

Yeah, for sure.

Marcello Salvati 10:55

Mm-Hmm. <Affirmative>. Yeah. And from a like, defensive standpoint. Yeah, absolutely. I mean, I would say actually that this is a great tool for blue teamers to actually just run on their code bases. Because like, if Vulnhuntr can find it, that's I mean, it's not like - I mean, Vulnhuntr is good, but it's not finding like super complex, you know, stuff. It does find like multi-step exploits, but it's, you know, the complexity level, I'd say that the vulnerabilities that it finds is from like low to medium, I'd say. Right?

Dan McInerney 11:23

Yeah. I would say like an average decent bug hunter or pen tester would find mostly the same vulnerabilities given "X" amount of time.

Marcello Salvati 11:31

Yeah. Exactly.

Madison Vorbrich 11:33

Okay, so it just speeds up the process. It makes it more efficient.

Dan McInerney 11:35

Yeah. You hit enter and it's just -

Marcello Salvati 11:37

It's just done.

Dan McInerney 11:38

Here are all my bugs.

Marcello Salvati 11:39

So it's a good tool for blue teamers to actually run because it's like a good first pass at things, right, without too much effort because you just point and click it. And like, if Vulnhuntr finds it, odds are that you would've - like the pen tester on your next pentesting round would've found that.

Dan McInerney 11:56

Right.

Marcello Salvati 11:57

So it's a good hack.

Madison Vorbrich 11:58

Got you, okay.

Dan McInerney 11:58

And so it also has an analysis paragraph where it tells you logically step-by-step why it thinks there might be a vulnerability. But the real value too is if you write really clean code, you might not know what the hot spots of your code are and what is the most important security sections of your code.

Marcello Salvati 12:18

Mm-Hmm. <Affirmative>.

Dan McInerney 12:18

And so Vulnhuntr may not find a vulnerability, but it will tell you, "Hey, I mapped out this chain of user input, and right here seems to be like, there might be some potential issues you should go investigate." But it's a great way of highlighting the hotspots of your code for developers just to make sure that you go back to that one little section of your code and say, "you know what, okay, I did this, I did this right." It's all, what, sanitized and all that kind of stuff.

Marcello Salvati 12:41

Yeah.

Madison Vorbrich 12:42

So, going back to how Vulnhuntr has found zero-days, can you give us an example of some of the zero-days that it's found and exactly why it was so impactful or significant that really made Vulnhuntr such a powerful tool?

Marcello Salvati 12:59

I think the most, what, like, what would be one of the like most complicated [vulnerabilties]? I think like, it was able to refind, if I recall correctly, one of the vulnerabilities in MLflow that was like a multi-step exploit. And that to us was kind of really another moment where we're like, oh, like this is actually - like that's impressive. Like, at that point, like that's actually -

Dan McInerney 13:20

Yeah, that was the original MLflow LFI that I found, that was the first, I think it was my first CVE ever actually.

Marcello Salvati 13:28

Yeah, yeah, yeah.

Dan McInerney 13:29

That was two years ago.

Marcello Salvati 13:31

But in terms of new ones, I don't recall. I'd have to think, I think like, [there's a lot of IDORs.] Like, there's definitely, like, I found a lot of IDORS.

Dan McInerney 13:47

The RCEs were pretty cool.

Marcello Salvati 13:48

Oh, yeah. Yeah. It did find an RCE -

Dan McInerney 13:49

It found multiple RCEs, and RCS is like the gold standard for a hacker.

Marcello Salvati 13:52

Yeah. That's true.

Dan McInerney 13:53

Because you can take over the entire system when you have an RCE.

Marcello Salvati 13:55

Yeah. It was actually very interesting because it was in like a, it was in like a request that seemed very innocuous. So the fact that it found it was like a very impressive, I think.

Dan McInerney 14:03

Yeah. One that I remember was the RAGFlow one where -

Marcello Salvati 14:06

Yeah. That was the first CVE one, right?

Dan McInerney 14:08

RAGFlow was like calling a - it was using a dictionary, and the dictionary -

Marcello Salvati 14:15

Was calling dynamically.

Dan McInerney 14:17

It was dynamically calling a function suite.

Marcello Salvati 14:18

A class. Or functions. Yeah.

Dan McInerney 14:21

Yeah, classes, which is, it's kind of complicated to read. So if you were just reading over the code, you might -

Marcello Salvati 14:28

You might just walk over it, yeah, yeah, it's like easy.

Dan McInerney 14:29

Vulnhuntr's eyes were like, "Nope, this is definitely an RCE." And credit to the RAGFlow team. I hit them up on Discord and they're like, okay, fixing right now. And within like, literally minutes -

Marcello Salvati 14:40

Minutes, they fixed it.

Dan McInerney 14:40

They fixed it. It was awesome.

Marcello Salvati 14:42

It was nice. Yeah. That's great.

Madison Vorbrich 14:43

So would you say that these are pretty widespread or common in AI/ML systems across the board, RCEs and these kinds of vulnerabilities that you've been finding?

Marcello Salvati 14:52

Right now, yes. Because we're at a, we're in a weird transition period, well, not transition period, but it's mostly just like the beginnings of like an entire ecosystem of tooling. So it's like, you know, the way I like comparing it is like the 1990s with like, everybody trying to figure out, just networking and all the server side vulnerabilities where you could just like, I don't know, just pop into somebody's FTP server randomly, because like, security was never a concern. So we're sort of in that phase now where people are just like creating stuff without thinking of the security implications. I think like that is gonna, we're gonna get to a point where like, we're gonna start seeing a lot more security consciousness.

Dan McInerney 15:34

I think so.

Marcello Salvati 15:34

Oh, wow. I did good <laugh>. And in like the next two, three years probably, I would say probably two, three. I feel like that's a, maybe, maybe a little bit more than that.

Dan McInerney 15:43

Agree.

Marcello Salvati 15:44

But, but like for now, it's, it's that's sort of like the state of things.

Dan McInerney 15:48

And it's interesting that all the zero-days that Vulnhuntr has found have been in AI tooling. And that's not on purpose. That was just because we went to Python most starred projects and hit, you know, sort by popularity, and they're all AI tools, like everything on GitHub right now that's really popular is an AI tool, and it's pretty much all written in Python, which is great for us. But I think that does also highlight the fact that the ecosphere of AI tooling in the open source world is broadening very quickly, and it's often being done by machine learning developers rather than professional security professionals, web app engineers. Yeah. But I think you can make a very similar metaphor back to mobile apps when they first came out.

Marcello Salvati 16:35

Yeah.

Dan McInerney 16:35

Mobile apps when they first came out were ridiculous. Like the Pizza Hut app, you could literally just cancel one request or response that was coming to the request and get free pizzas. Like it was literally security from 1993.

Marcello Salvati 16:48

Yeah. I'd say, but this is, it is a sort of a general trend in like anything new technology.

Dan McInerney 16:53

Absolutely.

Marcello Salvati 16:53

Like security is sort of like always trails a little bit behind for like the first two, three years.

Dan McInerney 16:59

Yeah, but then it picks up.

Marcello Salvati 17:00

And then it picks up. Yeah. And they rediscover their previous, all the previous mistakes basically.

Dan McInerney 17:03

Yeah. And this is why we have Vulnhuntr to begin with, is to speed this process along. This is why we have huntr, this is why we have all our [AI/ML] bug bounty programs. We're trying to make that gap between like, you know, 1999 security and modern security as short as humanly possible. And I think that we've, you know, done a pretty good job with that concerning, we have 300 zero-days we found on the huntr platform. Again, don't quote me on that. I'm not actually sure what the number [inaudible].

Madison Vorbrich 17:28

<jokes> We'll leave it at -

Dan McInerney 17:31

It's 600. <laugh>

Madison Vorbrich 17:32

Yeah. Yeah. Just keep the number up.

Marcello Salvati 17:33

<jokes> 1 billion. A hundred percent.

Madison Vorbrich 17:36

So, if we were to take a step back and think about Vulnhuntr in the context of MLSecOps, right? Where does that kind of fit in and how does it kind of align with the MLSecOps framework?

Marcello Salvati 17:48

I mean, it, I mean, as part of like your security process, like when you're evaluating, say you're using dependencies for a specific machine learning project or some AI internal AI tool or something. Like, you could technically use Vulnhuntr to scan dependencies to see if there's any low hanging fruit in any of the dependencies that your project relies on, I guess.

Madison Vorbrich 18:15

Mm-Hmm. <affirmative>.

Marcello Salvati 18:17

I mean that's, that's one use case for it.

Dan McInerney 18:19

I would see it as very, very useful into your CI/CD pipeline.

Marcello Salvati 18:23

Yeah, for sure.

Dan McInerney 18:24

When your developers are creating Python tools, they automatically get scanned. So DevSecOps, a lot of them are using tools like Semgrep and Snyk and Veracode - all these static code analyzers - as a first pass to find vulnerabilities. They really should be using LLMs as a first pass as well because it's not expensive. I mean, Vulnhuntr takes tokens, but it only costs what,

Marcello Salvati 18:44

Probably a couple of cents. Yeah.

Dan McInerney 18:46

It's, it's pennies to -

Marcello Salvati 18:47

I mean, the max is probably like a couple dollars, like, which is nothing. I mean it's definitely, yeah, that's a good point. Like you can use a combination of both legacy sort of like static code analysis and also Vulnhuntr to scan all of like, you have a pipeline and just put that right into that pipeline.

Dan McInerney 19:02

Yeah. I think we're in this like interesting stage where the transition between traditional machine learning and LLMs is not complete. I think eventually LLMs will take over 90% of like the AI world. Yeah. Mhm. But it's actually still quite useful to use traditional machine learning in things like log analysis and then give the output of the traditional machine learning saying, "Hey, here's a bunch of logs that we think might be vulnerable." And then giving that to the LLM to then further refine and give you some logic about whether that log file or that log entry is vulnerable. Mhm. But, all of this is to say, you can just inject all these tools right into your developer pipeline. Yeah. That way these zero days never come and see the light of day. That's the whole point of MLSecOps. Mhm. You want to make sure the security is done before you release the product. Yeah. Right. And this speeds that process along.

Marcello Salvati 19:47

You don't, you don't wanna be reactive. You wanna be proactive. Exactly. Yeah. No, that's the big thing.

Madison Vorbrich 19:51

So what kind of teams would benefit mostly from, from something like Vulnhuntr? If it were to be integrated or used in certain workflows or within that ecosystem?

Dan McInerney 20:02

Honestly, probably Python app developers. Yeah. If you're building out Python apps that stand up a web server like Fast API.

Marcello Salvati 20:08

Yeah, if it's network connected.

Dan McInerney 20:09

If it's network connected. Yeah. Or something. Yeah. They're gonna benefit the most. Yeah. From running Vulnhuntr on all of their code whenever they make a commit. Yeah. Cause It's not like it's an expense over time consuming. It's just a really easy way of seeing if you made any, any semi complex mistakes.

Marcello Salvati 20:24

Mhm. Yeah, I agree.

Dan McInerney 20:24

But then also pen testers. I mean, obviously.

Marcello Salvati 20:27

Yeah, I mean, a lot of, a lot of like, yeah. Especially if you have internal pen testing teams. Yeah. They'll benefit from this tremendously. Like it's, it's, yeah. It's definitely like a dual use tool. Yeah. Both sides can definitely benefit from it.

Dan McInerney 20:38

Yeah. But to be clear, for the pen testers, you do require the source code of this.

Marcello Salvati 20:42

That's true. Yeah, it's more of like a white box thing, yeah. Where you have

Dan McInerney 20:45

So you're not able to point Vulnhuntr at a web server.

Marcello Salvati 20:48

Yeah. This isn't like a, like a, I guess now they're called hack bots, right? I guess, right? Like, this isn't like a black box tool that you can just point at like a, just a random server. Like you need access to a source code.

Dan McInerney 20:59

Right. So I guess the defensive side would actually benefit more and get more utility out of this than the offensive side.

Marcello Salvati 21:05

Yeah, yeah for sure.

Madison Vorbrich 21:05

So, do you think with tools like this that come out, that more teams would kind of adopt more robust MLSecOps security practices overall? This would kind of shift the way that they're trained across the board?

Dan McInerney 21:18

Yeah. I think it would change the process for sure. Yeah. I mean, it's just an evolution of the process that was already there. So, like MLSecOps is coming off of DevSecOps. Yeah. There's some small changes and there's some major changes between DevSecOps and MLSecOps, but the fundamentals of DevSecOps is gonna be find the vulnerabilities before you release the code and before you do that kind of thing. And so Vulnhuntr fits right perfectly into that baseline ability. Like the differences between DevSecOps and MLSecOps is more or less just,

Marcello Salvati 21:47

I mean, it's more of just like a mental, it's more of like an adjustment. Like it's more of like a, just a, a like, instead of just like development, like as a whole, you're mostly focused on like machine learning.

Dan McInerney 21:59

Right, so you have some things like

Marcello Salvati 22:01

Specialized, I guess. Yeah, yeah. Yeah.

Dan McInerney 22:02

Model files don't exist in traditional DevSecOps.

Marcello Salvati 22:04

Yeah, exactly. Yeah.

Dan McInerney 22:05

So there's like small minor differences, but the core process is still the same. You wanna remove the vulnerabilities before they hit production, and Vulnhuntr fits like a puzzle piece into that niche.

Marcello Salvati 22:14

Yeah. It's all part of the onion. They're all part of the security onion. All part of the onion. Yeah. It's all part of the layers. It's all layers. You gotta really take the Shrek approach to the security just general.

Madison Vorbrich 22:25

I mean, I think that's a great quote for this episode.

Marcello Salvati 22:27

Yeah. Yeah. That's, there you go. You're welcome.

Madison Vorbrich 22:32

So I wanna transition into something equally as exciting. And it's part of huntr, which is model format vulnerabilities. Yeah. So, just to give people context or background, and I'm gonna do a very high level overview of it, but typically on huntr we've done OSS vulnerabilities, and now we're kind of shifting the focus into focusing on model format, model file vulnerabilities that first off, were also double the amount of our typical bounties that we have. Which is awesome.

Marcello Salvati 23:00

Yeah. At least, I mean, a minimum double. Yeah.

Madison Vorbrich 23:02

But also it, it's, it's relatively new, right? I mean, it's not something that I think maybe everyday hackers might know how to find. I mean, what do you guys think? And for listeners that might not know and that are very beginner friendly. How would you just go into those vulnerabilities? How would you explain it and why are they important?

Dan McInerney 23:22

So, baseline first is, when you think of MLSecOps, I mean there's really three categories of security in MLSecOps, and this is why MLSecOps is slightly different than DevSecOps. You've got the supply chain, which was huntr, that was all the software that goes into building and storing and training models.

Marcello Salvati 23:37

Yeah, the models themselves.

Dan McInerney 23:38

Yeah and then you have the model file, which is the model itself. The security of the model file is interesting because a lot of people don't realize the model file can execute code in many, many instances. A lot of times it's just a serialized object that you can just inject your own code into. And the model still works, it still makes predictions and does stuff, but then it also calls back to a -

Marcello Salvati 23:55

Yeah, it compromises your -

Dan McInerney 23:57

And then the last part is LLM security, which is like, you know, a firewall for your LLM, making sure that it's not jailbreak and stuff like that. So, huntr is now taking, huntr already takes over that first supply chain section. All the tools they use to build, train, and store models. And now we're moving into that middle section, which is the model file itself. It's a really interesting field because traditionally machine learning engineers have stored their models in pickle format and pickle. Is well known to have a decentralization problem. When you load the model,

Marcello Salvati 24:28

It's not a secure file format. Yeah. So when you load the model, like there's a, like if it's, if it's malicious, or like if it's compromised in some way, an attacker can like inject basically a malicious payload in the model file so that when you load it, it executes code on your machine that can do basically anything. So there's, there's like, there's been some like pivot, I guess for lack of a better word, to like safer model file formats. Mhm. But those two come with their own issues sometimes. Like, we had a bunch of reports for example, even previously to huntr making this transition on like memory corruption bugs affecting some of these libraries that load these safer model file formats. Right. So they come with their own set of challenges and, and security issues as well. Yeah. So it's not it's not like it, it just because it's considered safe doesn't necessarily mean like, there, it's completely devoid of attack service.

Dan McInerney 25:20

And so, like, there's many, many different kinds of model formats. Pickle was the one that was used for the last 10 years. Because it's convenient. It's just a - pickle essentially just takes a Python object from memory and stores it to disk.

Marcello Salvati 25:31

And it's built into Python, so you don't have to download anything out. Super easy. Yeah. Super easy.

Dan McInerney 25:35

But now there's a whole bunch of different ones like GGUF and... Right, yeah. PD, Params, and JSON, just a million different kinds of file formats. And what we're finding is that there are some issues in everything from Zip Slip where if like a library allows you to upload a TAR file, then sometimes that leads to a Zip Slip vulnerability where you can now overwrite files as soon as you load a model. There's issues with memory corruption, like out of bounds. All kinds of things. But if you wanna get started in this, I would say go back and look at the original Pickle deserialization issues. That there's a lot of resources online to look at how that works. Get a good feeling for that. Then go check out the huntr model file vulnerabilities page to see what kind of other formats that we support for these bug bounties. And then go Google what other kinds of attacks you can do to those. So I would start with Pickle, then I'd probably move into Keras. Keras is another model file format that has had remote code execution issues. It's a little more complex. It's actually a lot more complex than the pickle one. So start with pickle, go check out Keras, and then start doing some research on the other model files. From there you'll have a pretty good idea of what we're looking for and what would actually impact the security of AI as a whole.

Marcello Salvati 26:53

Also, if you're familiar with like, already like, like your reverse engineer and you're familiar with file format corruption vulnerabilities, like that's it generally, like that applies here as well. So I mean, it's, at the end of the day, it is a file. So I mean, if you're familiar with those kind of classes of vulnerabilities, like you'll, you'll feel at home immediately. Like the, it's a little bit like the, the file itself is structured always a little bit differently, but in itself, it's the same deal.

Dan McInerney 27:14

Yeah. We're, we're paying more for these because they're harder to find.

Marcello Salvati 27:18

Yeah.

Dan McInerney 27:18

They're much harder to find. It's a little more gray area about like what exactly is a file format vulnerability, but.

Madison Vorbrich 27:23

And this just builds off of typical web hacking skills that you would traditionally have? Is there any other special things that you might need to learn or know to find these?

Marcello Salvati 27:32

Well, I mean, it's, it's not necessarily like web hacking in this particular case. Like the skill set is somewhat different, I would say. I would agree. But it's, it's like if you're familiar enough, like if you can, if you're like already like a huntr for example, and like you've, you're familiar with web hacking. Like the, the transition between like web hacking and like this kind of stuff, it's not that difficult. So I mean, you can you can transition pretty easily.

Dan McInerney 28:02

Definitely different though. I mean...

Marcello Salvati 28:04

It is definitely different. Yeah.

Dan McInerney 28:05

Yeah. Like you're not finding local file includes or cross site scripting in model files. You're looking for strange things that happen when you load a file in a library.

Marcello Salvati 28:13

It's like basically strange things like what happens if you like flip a bit in this part of the file.

Dan McInerney 28:18

Yeah, what if you change a parameter that's...but if you compress the model file differently before, like what if you payload the model file with an injection, like a Pickle attack and then compress it? Like does that bypass scanners like modelscan? So if you ever have a question about whether something's a vulnerability, it's a good idea to start with our open source tool called ModelScan. And see if you can payload a model to execute code and have ModelScan not detect it. If you can do that, then you are definitely on the right road, you probably have something valuable to submit.

Marcello Salvati 28:52

Yeah, agreed.

Madison Vorbrich 28:53

So would you say there's a lot of ways for exploitation with these types of vulnerabilities? And also are they becoming more common, at least that you've seen in your field? Is it, it's not as niche as it may seem?

Dan McInerney 29:05

Yeah. Like, it's actually insane to me. When this all started, I thought it was really just Pickle deserialization bugs. Like, I thought that was it, you know? And as soon as we move into a safe file format, like, you know, Keras or just a different file format than Pickle, everything would be solved. That has not been the case. Which is, it's interesting. The weird part is we have safe model file formats that have not had any bugs. Safe tensors I think is one of 'em that I don't think we've ever seen a vulnerability in those. So why everyone doesn't just start moving into those...

Marcello Salvati 29:38

Safe tensors. Yeah. I think it's mostly, I'm not sure why. It's probably, it's probably like just tech debt. It's a lot, a lot of it is probably, I think so I'd assume like a lot of it is tech debt sometimes. Maybe there's something about that file format that maybe doesn't work with other systems that they already have set up. Like there's, there's probably a lot of stuff. But yeah, there's, although like, I don't know if, do they just be, I'm assuming you can still have memory corruption bugs with safe, like safe tensor five.

Dan McInerney 30:03

It's certainly not foolproof.

Marcello Salvati 30:04

Yeah, exactly.

Dan McInerney 30:06

But as of today, we haven't seen any issue. Which means it's probably a lot safer than some of the other formats like Keras with many issues of code injection.

Marcello Salvati 30:13

Makes sense.

Dan McInerney 30:14

But the issues that we're seeing now are very interesting. They're, we're getting beyond Pickle, and now we're starting to see things like brand new attacks in by injecting code that's formatted in a very specific way, so that now when you load the model from Hugging Face it's not just a pickle deserialization, it's like code is being executed based on let's say a file that is included in the model file folder. So some libraries are loading like config.JSON when you download from Hugging Face. So the model is fine, it's totally safe, but the config.JSON actually points to something malicious somewhere else, and the library automatically executes that. That would be an, that is right on the gray area of whether that's a model file format vulnerability or not. Because it's not in the model itself, but it's an example of how this attack chain as base is, I wanna go download a model. Is it safe to download this and open it? That is the amount of vulnerabilities we're seeing in that chain is expanding. Much beyond what we saw before where it was just pickle deserializations.

Madison Vorbrich 31:17

So it's pretty new then, right? I mean, this is all we're kinda learning in real time.

Dan McInerney 31:22

Oh, this is absolutely brand new. This is the cutting edge I think, of AI security right now.

Marcello Salvati 31:26

Yeah, for sure.

Madison Vorbrich 31:27

So if we were to zoom out and look at the bigger picture, right, in terms of model format vulnerabilities, can you give almost like a real attack scenario? I know that there was a model serialization attack or, or vulnerability. I don't know if you guys heard about it or you've talked about it in other content, but can you just kind of walk through, I guess, that attack process and its potential impact?

Marcello Salvati 31:53

Yeah, I mean, I think the most straightforward impact is that like you can potentially take an existing pickle model and Hugging Face, for example. And inject it with some malicious code. And then did what, like I think Dan, you, you did this actually where you could just send the link to the pickle file via email to like a bunch of engineers and like in a phishing email and whoever runs it you can basically be able to compromise their, their computer. I mean, that's one aspect of it, because it's, it's sort of like a tr I mean, it's not, not, I don't think a lot of, and machine learning and AI engineers are security aware sometimes. So when they see pickle files, they're not necessarily thinking, oh, this could be, you know, this could be used mostly. So that's part of it. And I think there's also the fact that a lot of like traditional antivirus scanners don't really scan these files, or at least they're not good at detecting payloaded, like malicious pickle files. I'm assuming that we're gonna catch, they're gonna catch up to that at, at a certain point. But as of like, when last time we tried, like they weren't really good at it.

Dan McInerney 32:54

Yeah, it was just a few months ago. I don't know six months ago or something, or at least that blog post showing that none of the antivirus scanners detected a very basic basic 60 form coded Python pay code in a model file. Yeah. And models are viruses. I mean, they can execute code. Yeah. So a phishing email, they're like, this is the golden age for phishing AI departments, because machine learning engineers have massive amounts of permissions usually in their networks. Yeah. Because they have to, they have to be able to reach data from all over the place, put it all together, and then have to have access to massive amounts of computing to train models and things. So, and they usually come from academic backgrounds, not professional developer backgrounds. Yeah. And again, these are generalizations. This is just what I've seen. I could be completely wrong, but it, it's, it's what I've seen when I've gone and talked to, to many of the machine learning engineers around the country.

Dan McInerney 33:44

Now you send 'em a phishing email with a pickle file and you say, Hey, I was getting an error with this model. Can you just load this and tell me what the, you know, why is the learning curve all screwy like this? They're just not gonna think of this as suspiciously as a word document. That has macros enabled. Like, that's suspicious. Why don't you say, oh, hey, can you just enable macros on this Word document so that I can execute code, but not a model file. It may not even be thinking it can execute code. Golden age for phishing machine learning engineers.

Marcello Salvati 34:09

And also this is on the supply chain side of things. Because like, you know, in a traditional ML or AI stack, sometimes you have like, I don't know, one part of the code that's doing OCR on something that needs a specific model from hugging face. Like what happens if like, all of a sudden that model gets compromised, right? So, from the supply chain side of things, it's also an important aspect of it because it like, you have a lot of like on a, in like a, like well engineered service of some kind that uses machine learning, you're gonna have different components of that service using different kind of models. So from the supply chain side of things, like you're, you're not always thinking, oh, damn like, you know, this thing is actually using this basically an executable to perform this task.

Dan McInerney 34:49

Downloading from the internet.

Marcello Salvati 34:50

Yeah, downloading from the internet. So, you know, you really should basically think that these models are basically like exe files on windows. They're basically, it's basically the same thing.

Madison Vorbrich 34:59

Yeah, so there's a lot of real world risk. I mean...

Dan McInerney 35:02

Very, very significant real world risk. Yeah. I'm not trying to be hyperbolic, this is like - from a hacker's perspective of 15 years, I would've loved to be doing red teams against companies with AI departments because I could fish them with model files.

Marcello Salvati 35:15

Yeah, a hundred percent.

Madison Vorbrich 35:15

Wow. That's it's really crazy when you think about that and all the possibilities and just how it's not, not that it's not being talked about enough, you know, but it's still, again, we've said it's very new. It's kind of, we're living in real time where this is unfolding and how it works.

Dan McInerney 35:30

As we watch these model file format vulnerabilities come in through huntr, my mind is being blown 'cause I really didn't think there was gonna be a lot of creativity. But it's like, you know, you guys are defining the rules of the model file format vulnerabilities as you submit your reports. Because a lot of this is just stuff I never would've thought about.

Marcello Salvati 35:50

Yeah, that's true.

Dan McInerney 35:51

Like the whole config.JSON actually pointing to a malicious thing. It's not even a, the model file itself in that case, but it still -

Marcello Salvati 35:58

It's, it's just the way the library loads the model basically.

Dan McInerney 36:00

Right, and it completes that attack chain of -

Marcello Salvati 36:03

Yeah. We're still, yeah. Yeah. Basically like the, the vulnerabilities that we already got are informing the guidelines that we're gonna be putting on the website. So I mean, that's, that in itself is, is yeah, it's a lot. That's a lot of work too. It's gonna be interesting to root out all the gray areas when it comes to that.

Madison Vorbrich 36:18

Yeah. I'm sure. So tying this back into Protect AI, how does MFVs tie into Protect AI's overall mission? If you were to put a nice little bow on it?

Dan McInerney 36:29

Well, we've got those three sections of AI security. Supply chain, model file, LLMs. This is squarely giving us the ability to discover zero-days before they're exploited in the model file format section. And nobody else has this capability and nobody else is really doing this.

Marcello Salvati 36:44

Yeah, a hundred percent.

Madison Vorbrich 36:45

Wow. Very interesting you guys. Well, thank you. To all those hackers out there that are really interested in MFVs and wanna take home $3,000 per vulnerability that you find, I implore you to please go on huntr.com. We'll be coming out with more educational content to kind of guide you through the journey and finding more of these vulnerabilities. Once again, I'm your host, Madi. Thank you to our listeners for your continued support of the MLSecOps Community and our mission to have AI Security content. And a huge thank you for Dan and Marcello for joining us today. This episode was awesome. Be sure to check out within the description of this episode, all of the resources that we kind of spoke about in this episode, as well as Dan and Marcello's contact information. If you wanna hang out with them, you guys are available, right?

Dan McInerney 37:40

Sure. Yeah. Just go open our house.

Marcello Salvati 37:41

Yeah, just text me honestly.

Madison Vorbrich 37:43

We're all available.

Marcello Salvati 37:44

Just put my number there, you know. I'll give you my digits.

Madison Vorbrich 37:48

Of course. Slide into your dms.

Marcello Salvati 37:50

Yeah, yeah. Slide into, yeah. We love unsolicited dms. Yeah, we love that. Yes.