Autonomous Agents Beyond the Hype

Episode Summary:

In Part 2 of this two-part MLSecOps Podcast episode, Principal Security Consultant Gavin Klondike joins Protect AI threat researchers Dan McInerney and Marcello Salvati to separate agent hype from hands-on reality. They explore threat-modeling for agent pipelines, securing API and data access, realistic deployment timelines, and everyday workflows that already save security teams hours. A must-listen for anyone building—or defending—the next generation of AI applications.

Transcript:

[Intro]

Marcello Salvati (00:07): 

So I was going to ask, like in terms of agents…

Dan McInerney (00:11): 

A hot topic now! 

Marcello Salvati (00:11): 

Yeah, switching topics for a bit. Like what do you see in terms of like emerging security issues or like emerging vulnerability types with agents specifically? Because I think a lot of people are trying to predict the next frontier when it comes to agents specifically. Like what's your opinion on  that? 

Gavin Klondike (00:34): 

Yeah, actually it's kind of funny. I was actually at CypherCon this last weekend doing a whole workshop on how to build practical AI agents and what that looks like, because I feel like a lot of people have been talking about agents at a really high level. And I'm like, no, show me how to make this. And so I just put a workshop together for that. Before I get into it, I want to see between the two of you, what are your definitions of AI agents? I know that that's been changing depending on who you ask. 

Marcello Salvati (00:58): 

Yeah. I mean, honestly, from just having built agents, I would say I'd classify agents as like LLM applications that autonomously make decisions or autonomously take actions. I think that's, or at least a quorum of agents that autonomously are able to take actions between each other. 

Dan McInerney (01:20): 

I would mostly agree, I would say an agent in LLM that has access to some kind of tool or function that it can then perform an action without user input. 

Marcello Salvati (01:27): 

Without user input. I think that would be my key distinction. But again, yeah, this is such a new thing. Like I think everybody has their own terminology or definition. 

Gavin Klondike (01:41): 

Yeah. I definitely agree with that definition. You essentially have the LLM as your reasoning agent, and  then you give it access to tools and functions. So you get long-term memory, short-term memory, backend API functions. You get things like RAG. You can even have certain design patterns like reasoning and planning. I'm actually a huge fan of workflows, building specific workflows. 

So the things that I talk about in my workshop is ETL pipelines. They stand for extract, transform, and load. A lot of this is stuff that we've solved in software development and software design, and it's just the data science community is finally catching up to some of it. And so they're trying to say, oh, this is brand new. And it's like, yeah, cool. We've had it for 20 years. So let's integrate some of these things. Registry patterns, right? 

Gavin Klondike (02:24): 

Just have somewhere where you can write stuff down so that agent five can see what agent one said. So yeah, that's kind of how I would describe agents as well. So given that, as far as new and unique vulnerabilities, instead of playing golf where you hit the ball directly, you're playing pool where you have to hit a ball to hit a ball. A lot of the vulnerabilities are still kind of the same. 

But again, I still point to the bug bounty community. I know Joseph Thacker, he actually just released a massive blog post on his research and things that he's learned. Hacking AI agent systems, both some that he's created and some that  he's found in bug bounty platforms. That's where I think you can get the most information about some of  these new technologies is find the nerds who spend all day, every day trying to break these systems and  see how they got in. Cause they're motivated by money, right? They get paid if they find a vulnerability, they don't get paid if they don't find a vulnerability. 

So if they're able to share and collect a lot of that and curate that information for us that's where I would start looking. Because if your competitor has been exploited by these vulnerabilities, it's very likely that you're going to get exploited as well. So try and be a little bit more proactive. 

Dan McInerney (03:44): 

Yeah. Where do you see the biggest issues from the [OWASP] LLM Top 10? Because I feel like the LLM Top 10 list changes a little bit with the agents. Personally I think the direct prompt injection now suddenly becomes way more impactful because now you're not just the attacker and the victim, you can affect other systems. What was your opinion on where the threat surface is or the threat modeling should be in an agent, the hotspot? 

Gavin Klondike (04:09): 

That's kind, that's kind of hard for me to answer. So when it comes to the Top 10, I was a huge contributor in version one of the Top 10. Version two came out with some new stuff that I don't particularly agree with. But I didn't have a heavy hand in contributing to version two, so that's kind of hard to say. I do remember that I've told myself I have to stop looking at it because there are some rants that I go on that I don't like who I become when I see 'em. I'm trying to look it up right now just to see what was on version two. 

Dan McInerney (04:46): 

And you don't even have to tie this back to the OWASP. I'm just curious like what your threat modeling would be for an agent. Where, like, what's the hot button issues? That should be the main focus for defenders? 

Gavin Klondike (04:56): 

I would still say, look at your APIs and look at your data access. Make sure you follow principle with least privilege, right? So do out-of-band authentication and authorization, make sure that your APIs that the LLMs talk to are also locked down. So you can do pen testing and typical QA on those. Same thing for any sort of database access. Vector databases are still databases, so follow best practices for data handling on those things. And then once you shrink down the size to one agent, it's a lot easier to separate it out to multiple agents. You still need to manage your trust boundaries. So think of the environment holistically. Think of each of the components that an LLM interacts with individually. And that is where I would start with trying to properly threat model an architect, an AI agent system. 

Gavin Klondike (05:48): 

Right now though, it's really interesting because last year at RSA, there were a lot of companies that either had AI in their name or they had AI on their sales pages, but they didn't really showcase a lot  of AI. I found out later, and I don't remember if it was like an RSA enforced mandate or if there was like a legal mandate, but if they weren't doing AI, they weren't allowed to talk about it. And so a lot of these companies that had AI in their name didn't talk about it. So that tells me that they're probably not using AI  as much as they say they're using AI. And, and I'm not trying to speculate too much, it's just when it  comes to these AI agents, I don't think a lot of people are building them.

Marcello Salvati (06:26): 

Yeah. 

Dan McInerney (06:27): 

A hundred percent. 

Marcello Salvati (06:28): 

No. That's exactly what... So we've had this discussion to ourselves, between ourselves basically. But like, from what I've seen, I think agent adoption is relatively small in organizations right now. Like, I'm not seeing a whole lot of at least completely autonomous agents, right? I'm not seeing a whole lot of completely autonomous agents being used internally at organizations right now, or at least I haven't heard  of a lot of them using it. 

I think there is a, the current argument from people is that this is going to be the year where organizations build those out, and then next year is when things will blow up in terms of actual like huge amounts of autonomous agent deployments and organizations and stuff. I'm a little bit skeptical, honestly. But I'm not exactly sure what the landscape, I think anybody would be hard  pressed to accurately predict the landscape of... 

Dan McInerney (07:33): 

I would disagree. I would say I think you're absolutely right when you say this year is the year that they're trying to invest in it. Like you're giving these presentations, I'm sure, to quite large audiences on how to build AI agents. Cause everyone wants to know right now, they'll build that knowledge up over the course of the year. And I think that you're right, that 2026 is the day that it comes. And I'm curious about your opinions here on your timelines of when are autonomous agents going to be deployed in production at production level scale? 

Marcello Salvati (07:56): 

Scale, yeah. 

Gavin Klondike (07:57): 

I, this is where we get into semantics again, because people keep changing the definition of agent to be something different. It's just like AGI, right? They're talking about artificial general intelligence, AGI, for a little bit. And they're like, oh, it's just around the corner. It's just around the corner. And we had an idea,  like there was no canon definition of AGI, but we had an idea of it's, an AI... It's like WALL-E or the movie AI, right? It's the Jetson’s car that can drive and park, and it has its own little personality. 

And so they shifted from that concept and they're like, oh, no, no, no, it's, it's a thing that can do stuff on its own. And then, oh, no, no, it's this. And then eventually it's, oh, maybe AGI was a fruit basket the whole time. 

Gavin Klondike (08:42): 

But don't worry. Now we're looking at ASI, artificial super intelligence. It's a, we create a new definition for the exact same thing that we had in mind like five years ago at this point and we've been saying that AGI was only a few years around the corner for the last 70 years. So for, again, people who found their 10 years of AI experience, once ChatGPT came out, they're having a very different conversation than those of us who've been studying AI for many, many years. And so it's really interesting. 

I wish more people understood the Chinese room thought experiment. I think that's a really good way of understanding how these AI systems actually work. I also wish more people understood the history of AI winters and what caused them and what really came about. So as far as AI agents, I think your definition, Dan and my definition of AI agents, there are companies who have been using 'em.

Gavin Klondike (09:35): 

They just link LLMs together and give them access to APIs. There are companies that are doing that. They're still, they are finding some use out of it. But it's not as crazy as we thought it was going to be. There are other companies, and this is the thing that really worries me, that are just throwing spaghetti at the wall to see what sticks. And so they're building out these massive, AI agent amalgamations that are doing something a lot less efficient. 

So a really good example and I'm not, I'm not trying to, to name and shame, I just need to call them out so that people understand what the landscape looks like. Devin was one that was supposed to be a software developer in a box, right? It was this whole agentic system to build software. And so they released a trailer, and the trailer looked fantastic, right? 

Gavin Klondike (10:26): 

And they said, Hey, here's Devin. I want a piece of software that does this. And it would go through and it build out the entire software by itself. Awesome. Then a software developer took a look at that trailer, and they looked at the timestamps and they said, Hey, the software that you're trying to get this thing to build would take me just a few hours. This took like 12 hours. So why would I use an autonomous system to do something less efficiently? 

And then you look at, it was going through and debugging code. It was debugging errors that it created. So it would add something, it would make a push, it would add  something to the code that would break the code, and then it would remove the code that it added that  broke that code, and it debugged and fixed the same problem. 

Gavin Klondike (11:08): 

So like, this isn't to name and shame Devin, right? Because if I tried to build something like that, I'm sure I would fall into the exact same problem. It's just, there's a difference between the hype and what people are actually getting use out of. There was an article that Google put out that said, oh, 25% of our code is created by AI. It's auto complete. My smartphone does auto complete. My phone, like from the early 2000s did auto complete. 

So I can't take that in full faith saying that AI is really going to be the next big thing. I think there are fantastic use cases, do not get me wrong. And I think that there are some who are really in the know that are building these systems that are really interesting. I have a couple companies that I'm keeping an eye on because I'm really excited to see what their research does. But as far as, you know, are all of these companies really going to blow out of the water with AI systems next year? I have no idea. 

Marcello Salvati (12:02): 

Yeah I think, I think anybody like, I don't think anybody can predict what exactly is going to happen. I think anybody who like says, ah, yeah, I know I'm a hundred percent the expert, I'll definitely be able to  predict this. I think... 

Dan McInerney (12:16): 

I'm a hundred percent the expert, and I'll never be able to predict this. 

Gavin Klondike (12:18): 

Buy my stock! 

Dan McInerney (12:24): 

Like there's going to be a tipping point that is a stark tipping point. Yeah. It's when the next model comes out that literally performs human tasks at least minimum human level, which I think is, you know, one of the million definitions of AGI. Which I suspect based on extrapolating the intelligence performances, it's probably in the next two years or three years, three, four.

Gavin Klondike (12:47): 

So how do you, how do you measure that? Exactly. Like doing something that a human can do is so subjective. Like, we have systems in airplanes that can land a plane, like, does that count? And we've had that for a decade at least. 

Marcello Salvati (12:59): 

And technically, like I think OpenAI did release, or at least was planning on releasing a model that performed that like PhD level human intelligence, right? There was like the $20,000 a year thing. 

Dan McInerney (13:11): 

Oh, that was o3-mini high. They let it just think about the task for that for a really long time. 

Marcello Salvati (13:15): 

Yeah. So, I mean, but like, by your definition, is that human level like? 

Dan McInerney (13:19): 

I would argue yes. And this is just my opinion but I think benchmarks are a good measure of generally  performing tasks across a broad range of features that, where the model is not trained on any one of those features. And if you look at all the benchmark requirements from the ARC-AGI, the original one, up to ARC-AGI-2 now. 

The curve is, it's really pointing somewhere between the next two to five years where it hits this asymptote and just takes off. Because it's interesting that the intelligence of these models is increasing exponentially and the cost is decreasing exponentially too. Which makes me feel like we, you know, five years conservatively is about when, but again, to your point, speculation. 

Marcello Salvati (13:57): 

Yeah, yeah, yeah. 

Gavin Klondike (13:58): 

And you even need to be careful with those benchmarks because for example, the software development  benchmarks, it's mostly in Python. So because it does really well on a software engineering benchmark, can it write in rust? Can it write in C can it write in C++? I don't know. Some of the mathematic benchmarks are so esoteric. 

And then even once you get the results of those benchmarks, for example, when GPT-4 came out, not 4o, 4, then it could pass the LSAT, right? Which is the, it was a test that you had to do to get into law school and so a lawyer thought that, Hey, it is good enough. I'll use it as a paralegal and I'll have it write my legal brief. And it wrote a beautiful legal brief. The problem is that everything that it wrote was hallucinated. It was completely fictitious. 

Dan McInerney (14:41): 

Yeah. 

Gavin Klondike (14:43): 

So even though it could pass the LSAT, which is amazing what you can do, once you've memorized all  the answers. How much does that translate into real world impact and actual usability? So you gotta take a lot of this with a grain of salt, and that's why I'm very hesitant on really like, pointing to the  benchmarks. I don't even look at benchmarks personally, just to let you know, because GPT-4 was amazing. 

ChatGPT was amazing, came out, going to change the world. GPT-4 came out. ChatGPT-3.5 is ChatGPT. GPT-3.5 trash, nobody ever use it. GPT-4 is the new thing. GPT-4o comes out awesome. GPT-4 is trash. Don't use that. 4o1 comes out awesome. 4o1 has reasoning. This is two weeks away from, or two years away from AGI. And then now we have GPT o3. o1 is trash. Don't use that. Anthropic. I'm a huge fan of Claude 3.5. 3.5 in my opinion, was actually a lot better than 4o1, but they also kind of filled different tasks, trash everything else. 

Like, and so we're, we're following this hedonistic treadmill onto the  next best thing, the next best thing, the next best thing. I'm sure in two years maybe we'll have some  amazing AI systems and we're going to throw that away the year after. So we'll see what happens. 

Dan McInerney (16:03): 

Yeah, to your point with benchmarkings too, is those benchmarks it is not impossible to confirm that these models haven't trained on the benchmarks themselves. Or the benchmark answers, which you had mentioned earlier, which then just makes the benchmark completely pointless. Like literally worthless. 

Marcello Salvati (16:20): 

Yeah. And there's absolutely, to your point, like there's absolutely a disconnect between like real world applications of LLMs versus the actual benchmarks. 

Dan McInerney (16:30): 

Which is, this is actually why I like the software engineering benchmark that Google had where they took a whole bunch of actual software engineering jobs and said, how many of these jobs can our LLM complete and get paid for? 

Marcello Salvati (16:43): 

Oh, you mean like the Fiverr benchmark thing? Yeah. Gotcha. 

Dan McInerney (16:47): 

That's a good benchmark. Although... 

Gavin Klondike (16:48): 

That's a great benchmark. 

Dan McInerney (16:50): 

Yeah, at the same time, they also kind of fudged this a little bit because they didn't, I think they used  problems that were already solved. Which again, just comes back to the point where like, well, maybe this thing just scraped the website and found issues that were already solved and then it tried to solve them again, but had already trained on that knowledge. It wasn't an actual live, I solved these things and made this much money. It was, I could have solved these and I could have made this much money. Which is like, okay, but let's, let's actually make this a live benchmark, you know? Real world work, how much money can you make? 

But we're slowly running outta time here. I want to make sure we got to everything you wanted to talk about. Was there anything that has particularly sparked your interest or anything that  you wanted to talk about that we didn't get to at this point? 

Gavin Klondike (17:33): 

Oh, good question 'cause we covered so much. This is something I could talk about for like hours and hours and hours. 

Dan McInerney (17:38): 

Oh, join the club.

Gavin Klondike (17:39): 

So limiting it to one hour is really difficult. 

Marcello Salvati (17:40): 

Yeah. 

Gavin Klondike (17:42): 

I think one of the big things that I'll say is for these kinds of conversations, something I'm realizing is that there's a separation between people who are really plugged into AI systems. Cause we're standing on the shoulders of giants, standing on giants, standing on giants. 

I did a podcast not too long ago where somebody was asking about DeepSeek R-1 and some of the history and where a lot of that came from. And I had to go all the way back to 2016, right? That first paper, Attention Is All You Need, and say, Hey, this is where we started getting Transformers, OpenAI built GPT 1, 2, 3. Nothing really interesting came of it. Three was kind of cool, but 3.5 came out and that was ChatGPT. And then everything kind of blew up from there. And then just a few months later, Meta had their own large language model. 

Gavin Klondike (18:24): 

And then a couple months later, right, GPT-4 comes out and then Orca from Microsoft comes out, which is their LLM, Google had Bard. And so now we have nothing or very close to nothing, and then all of a sudden all these companies pop up within the same few months. And then that wasn't enough. And so now we have an open source version, and now we have people building on the open source version, and now we have GPT-4o, and then 4o1 where we start getting into reasoning models. And then we change up our architecture of the reasoning models. And so when we got to DeepSeek R-1, and I'm only bringing it up because it's brand new, it was reinforcement learning on top of all the lessons that we learned from Transformers, which again, goes all the way back to 2016. 

Gavin Klondike (19:07): 

So we're looking at about a decade of research coming into where we are today. But to properly understand it, you have to kind of be familiar with a lot of it. So, one of the things that I try to do, and this is something that I really want your listeners to kind of focus on, is like, I want to educate people and bring them into this conversation instead of, you know, staying at a high level of this is AI, this is all the latest and greatest of AI. 

If you're interested in this stuff, and, and I do encourage you to be interested in this stuff, play with an LLM. It doesn't matter what LLM you use. Just start playing with one of them. Try to see what kind of tasks you can get it to do. Try to generate code, try to do some knowledge synthesis. 

Gavin Klondike (19:46): 

I do a lot of research. I'll throw research papers into Claude or into OpenAI and I'll have it, you know, summarize a lot of that research for me. 

Dan McInerney (19:54): 

Super useful. 

Gavin Klondike (19:56): 

One of the agents that I built was actually a pipeline of the first agent I would ask, I would send a conversation. The first agent extracts a YouTube link. The next couple steps is it uses some code and yt-dlp to download the transcript of that YouTube video. And then I have another agent that summarizes the information out of that transcript and I wrote a whole prompt on how to do that. 

You can look at Daniel Miessler's Fabric, if you want to see some good examples of prompt. I used the extract wisdom prompt there, and then I had another agent take that massive analysis of that YouTube video and turn it into essentially a short form, like 2,000-3,000 word blog post. 

Gavin Klondike (20:35): 

And so now when DEF CON, for example, or any security conference releases hundreds of videos, I can't watch hundreds of videos, but I can put them through this AI system, it'll synthesize a lot of that knowledge and it'll give me a prioritized, curated list based on my interests that I tell it, of which ones I should watch first and which ones really aren't worth my time. 

So now I'm not automating my entire life, I'm just automating a small piece of it, and it's saving hours and hours and hours and hours. And I'm using this same methodology and this same process to shave hours off of my day. 

Dan McInerney (21:12): 

Right. So your message is, it's not that hard to be at the cutting edge of AI. You just kinda have to know  what's the cutting edge application to an average person's life today? 

Dan McInerney (21:23): 

Yeah. I completely, completely agree. I think it gets a lot of people get overwhelmed with the enormous amount of AI information that comes out. DeepSeek R-1 just came out now, the new Llama model just came out, well, how do I use these things? It's good to get these kind of educational quick videos online or Twitter threads that just lay it out exactly. Here's how you can book a flight automatically with an agent and now you're using AI in your everyday life and it's not as complicated. It doesn't really matter what the latest and greatest model is. Here's the workflow. 

Gavin Klondike (21:50): 

Yeah, use it as a travel agent, right? I was out in Milwaukee, Wisconsin. I knew nothing about Milwaukee. So I'm asking ChatGPT, I'm using the voice feature. I really like that. And I'm like, Hey, what are some, I'm walking around downtown right now. I don't have access to a car. I do have Uber, but  what are some places I should check out? And it just created a small list. And so I did a walking tour, a self-guided walking tour around the city of Milwaukee. 

Dan McInerney (22:11): 

Oh, that's a good idea. You can get the history of the whole place too. You can be like, Hey, I'm at this point, like corner of Boston, what crazy events happened within a one mile block. Yeah. These are all realistic scenarios that everyone could use that turns down the hype and turns up the practical  applicability. 

Alright. With that, I think we are now officially out of time. It was a pleasure chatting with you. Yeah. actually for the second time, because Gavin and I had actually met at BlueHat last year, the Microsoft AI Conference. And it was a pleasure to chat with you then too, so I'm happy to talk with you  again. And I hope our paths cross more often. 

Marcello Salvati (22:45): 

Yeah. And a pleasure meeting you. I think this is the first time we talked. So pleasure meeting you. 

Gavin Klondike (22:49): 

Yeah, nice meeting you. Nice seeing you again, Dan again for the listeners, just because I really enjoyed this, Dan had a fantastic presentation at BlueHat and one of the things that I took away that I still use in every single one of my conversations with customers is the DFIR hierarchy of needs. Like build out that measuring stick and then show people where they land on that measuring stick and try to bump them up a couple points. That is invaluable. So thank you. 

Dan McInerney (23:13): 

Great. That's great. Yeah, we learned a lot today. This is Dan McInerney, this is Marcello Salvati, and this is Gavin Klondike. You can plug your Twitter, your website if you'd like right now, so that people can loop in. 

Gavin Klondike (23:26): 

If people would like to continue the conversation I'm on LinkedIn, Gavin Klondike. I'm also on X and Bluesky at GTKlondike. And then my YouTube channel is NetSec Explained. So I'm going to be releasing that video in the next couple of weeks talking about practical AI agents. So keep an eye out for that. 

Dan McInerney (23:45): 

Awesome. All right. Great chatting and we'll talk to you next time. Have a good one. Thank you!

[Closing]

Additional tools and resources to check out:

Protect AI Guardian: Zero Trust for ML Model

Recon: Automated Red Teaming for GenAI

Protect AI’s ML Security-Focused Open Source Tools

LLM Guard Open Source Security Toolkit for LLM Interactions

Huntr - The World's First AI/Machine Learning Bug Bounty Platform

Thanks for checking out the MLSecOps Podcast! Get involved with the MLSecOps Community and find more resources at https://community.mlsecops.com.