AI Beyond the Hype: Lessons from Cloud on Risk and Security

 

Audio-only version also available on Apple Podcasts, Spotify, iHeart Podcasts, and many more.

Episode Summary:

On this episode of the MLSecOps Podcast, we’re bringing together two cybersecurity legends. Our guest is the inimitable Caleb Sima, who joins us to discuss security considerations for building and using AI, drawing on his 25+ years of cybersecurity experience. Caleb's impressive journey includes co-founding two security startups acquired by HP and Lookout, serving as Chief Security Officer at Robinhood, and currently leading cybersecurity venture studio WhiteRabbit & chairing the Cloud Security Alliance AI Safety Initiative.

Hosting this episode is Diana Kelley (CISO, Protect AI) an industry powerhouse with a long career dedicated to cybersecurity, and a longtime host on this show. Together, Caleb and Diana share a thoughtful discussion full of unique insights for the MLSecOps Community of learners.

Diana Kelley 00:08

Hello, and welcome to the MLSecOps Podcast. I'm going to be hosting today; I'm Diana Kelly and I'm the CISO here at Protect AI. And joining us is a just fantastic person - who, if you haven't heard of him I don't know where you've been - Caleb Sima, who is currently a builder and founder of White Rabbit, which is a venture, oh, I'm forgetting what it is. It's a venture studio.

Caleb Sima 00:31

Cybersecurity venture studio, yeah. Which is a mouthful, but basically it's a startup that tries to create cybersecurity startups. So, that's a whole different ballgame.

Diana Kelley 00:45

Yeah. It's amazing. And you certainly know about startups. I think I met you back in 2005, and you had founded SPI Dynamics at that time. Is that right?

Caleb Sima 00:54

Yeah, that's right. It was - those were the good old days, right?

Diana Kelley 01:00

The early days of DAST, yeah.

Caleb Sima 01:02

Early days. That's right.

Diana Kelley 01:03

Yeah. So you've been at this a long time, extraordinarily successful. So we're really thankful and grateful that you're here to join us on the show today. Our audience knows a lot about MLSecOps and AI security, but they may not know how you intersect with it. And specifically, you know, what have you seen about that in your career and what led you to the work you're doing today with the Cloud Security Alliance and AI Safety?

Caleb Sima 01:28

Yeah, I mean, that's a great question. I mean I guess I'll start from the beginning and tell the story, which; I think my intersection with really, I would say machine learning, not even AI, was when I started at Databricks. So I joined Databricks to become sort of their head of security, or really Security CTO, I believe was my official first title in joining the company. Where I started, it was a very, you know, very small - Databricks now is a fairly well-known space or company in machine learning, but at the time it was 300 employees or so - but how I got there, it's funny. I had my first child, my first, my daughter Ava actually. And I took a year off to kind of like, you know, figure out what it's like to be a dad and I, and at the time, I really wanted to dig into more hands-on tech stuff.

Caleb Sima 02:27

So there's two things I really wanted to learn about. At the time, I wanted to learn about crypto, right? Blockchain. And I wanted to learn about machine learning. These were like the two things I wanted to learn as I went through that year, really diving deep into that machine learning was just so amazing to me, and with so many adjacencies that I just got obsessed. And that led me over to Databricks, where I you know, I'm, I'm proud and honored to say I worked with probably some of the smartest people in machine learning that you could possibly be around. And I spent a good amount of time in Databricks doing that. And then I went to Robinhood, to be the Chief Security Officer over there. And then during that timeframe, I thought I was up to date with machine learning.

Caleb Sima 03:13

Like my time in Databricks I learned machine learning, I learned a lot of this stuff. And then during my time at Robinhood ChatGPT broke out, LLMs broke out, and all of a sudden I started, like, it's an entirely different world. And everything was moving so fast at the time. I'd spent about two and a half years at Robinhood, and I was trying to learn about LLMs and learn about AI, and I just realized, you know what? Like, I cannot do this with the family, with the, I gotta go take the time to go do this. So I left Robinhood, I said, it's time. I am going to focus on learning AI and catch up to it. And so I spent four to six months, literally as my job, nine to five reading, building, understanding, and I just started posting on LinkedIn, all the references and sources I was learning from. And all of a sudden, I guess people started noticing and they started following. And then somehow I became an "AI expert." I just, you know, and - which by the way, I am not. I am not an AI expert - I just happened to really want to catch up and learn as much as I could. And really, I wanted to learn the fundamentals, right? The real basic building blocks. And so that's sort of how I think, you know, the intersection of AI came into my life.

Diana Kelley 04:34

Yeah. It's amazing. That does, you know, we start reading, we start learning, and it's what - it was wonderful. You were sharing with others, and then that's it. You are an AI expert now.

Caleb Sima 04:44

Well, they say I am and I'm telling you; I'm not <laugh>. It's just, it's just, I happened to be able to take the three months or so to study and learn the basics. And that's allowed me to I think take a good first principles view of everything that comes into AI, which I don't think a lot of people have the capability, but will over time, to be able to.

Diana Kelley 05:07

They will, yeah. You were at the beginning, you were at the forefront of DevSecOps and building security into the development lifecycle. As you look at MLSecOps or AISecOps, how do you frame building security in and securing machine learning?

Caleb Sima 05:22

Yeah, so that's a great question. And that's, I think where my intersection with CSA or the Cloud Security Alliance really came into play because at the time I was learning a lot of the basics and fundamentals of how does an LLM work, and how does it behave, and what does it do. And of course, as security people you know, when AI started coming around, everyone puts the brakes. And there's a lot of like, fear, uncertainty, and doubt being thrown around around AI. And what you need to be scared of and what you don't need to be scared of, and well, let me be clear. There's no one saying, you need to not be scared of something. Everyone was, "you need to be scared of everything in AI. It's a threat to humanity, threat to nation, threat to everybody," right?

Caleb Sima 06:07

And so, and you know, when I was going through this, I was like, hey, like there are legitimate things to be afraid of. Like, we need to understand how to manage risk here, and there are easy ways to go and implement sort of controls around this. But what I was missing is no one was really talking about practical things a security team can do to help mitigate risk if you are building a model, if you are using a model, like, what does this look like? And so Jim [Reavis], - who was the founder of CSA at the time - and I were talking, and he is like, this is exactly what happened with cloud, right? When cloud came out, everyone had the fear, everything was there, and there's no practical guidelines around what matters. What can an enterprise security team do in order to help mitigate the risk. So he is like, AI is bigger than cloud. We need to do the exact same thing. So Jim and I sort of joined forces and, you know CSA is sort of a nonprofit, so I donate my time.

Caleb Sima 07:10

And I helped create the AI Safety Alliance with CSA. And we went and we basically, the, the motto was very simple. My mission was, I wanna build practical guidelines around how to secure AI. No fear, no craziness, just, hey, by the way, did you know 98% of the things you do to secure an AIML pipeline is all the same stuff you know today, <laugh>, like, you know, it's nothing different. It's like access control, patching, system management. Like, you know, it's just on a different flow in a different path. And so, building that together, so we were able to get all of the great CISOs together from all of the great model providers, from Anthropic, you know, OpenAI, Google, Microsoft, AWS, like a lot of these people joined and helped because they also wanted to go in. So we ended up getting 2000 or so volunteers in this area to help kind of build practical guidelines. So that's what we're in the middle of. In fact, I just got back [from] where we had an, you know, sort of an in-person building of the new AI control matrix around what does that look like and how do we look at an MLSecOps pipeline, and what is, what's practical? What are the things that are model specific versus what are just generic controls and how to do them.

Diana Kelley 08:39

Yeah. And as you're, as you're building that out, and as you talk to CISOs both that are volunteering and working with you and to CISOs that are just getting up to speed, what are some of the things that you recommend they do to integrate ML security into their existing security program?

Caleb Sima 08:57

You know, I think a lot of, first, you know, I'll, I'll take this from a broad spectrum and then I'll kind of dive down specific. You know, there's a lot of things I think enterprises and CISOs specifically are trying to understand when it comes to an AI program, AI governance, what does all this mean? I won't go into that. I'll go down to the very, very specific technical, like, okay, what are you doing? Right? Like, for example, most organizations today are using a foundational model. They are using, when I define that foundational frontier model, this is [for example] "I am using AI. I am using Anthropic, I am using Gemini." Right? Like, these are the things I think most organizations are doing today, and they're just trying to figure out what your use case is. And so I think a lot of this has to do with your use case.

Caleb Sima 09:46

You are not building models. You are probably not fine tuning models. But what you are probably doing is maybe using things like RAG [retrieval-augmented generation] where you are adding data to your model to be parsed. And so you just have to think about your data pipeline. That I think becomes the most important. When you think about what you're doing is, okay, if I do have RAG and I'm adding data, where does that data come from? How am I cleaning that data? How am I formulating that? And also more practically, what sort of access permissions am I applying on that data? Or really a great question to ask is, well, if I'm using a vector database for this, do I have the ability to put access permissions on this? And so, you know, there's, these are, I think are the basics at what enterprises are dealing with, and they're going from what I would say the most obvious answers for, which is most people are building chat bots or other kinds of search functionality, wiki functionality.

Caleb Sima 10:47

And they're just building it in very, very simple single tenant like models, which is the finance group has a RAG database. The engineering group has a RAG database. The HR people crew has a RAG database. And then the model itself will pull in its own, in its own form. It has its own access to finance, engineering, and people. And then there's no intersection, right? Like, that's the way that it works. Or some are trying to just figure out, well, I have a single RAG database and a single LLM pipeline: how do I manage permissions now between engineering, HR, finance? And that becomes a little bit more difficult. Then you start thinking about, well, what about prompt injection? Which by the way, is out of all of the attacks, I think today is the most enterprise applicable problem and is not a solved problem. And it is a deep problem. However, what's, I will tell you some of the mistakes that I see people make, which is, oh, well, they believe, well, hey, actually the, where I need to put something like an AI guardrail or a prompt, or an AI firewall, LLM firewall - I dunno, Diana, what do you call your - what's the, what's the accepted term that you guys -

Diana Kelley 11:59

<Laugh> I keep hearing "AI firewall."

Caleb Sima 12:03

AI firewall. Okay. Yeah. You know, it's interesting because most, the definitions, when you say firewall, everybody thinks on the external, from the external public to the inbound to the API. So people will say, oh, I'll put sort of this AI firewall and add a network layer at the AI layer where my employee or my customer is writing their text prompt - by the way, not multimodal. Text prompt. - and then it will analyze it, look for prompt injection, deny it, et cetera. However, what they're missing out on, and I think this is very critical, and most people have not thought about this, is that prompt injection comes from everywhere, not just an employee writing something in a chat bot. It comes from a document. It can come from metadata, from an image, it can come from an email content, it comes from everywhere.

Diana Kelley 12:53

Another LLM.

Caleb Sima 12:54

Yeah. Another LLM. That's a great point. And so, like, this is like cross-site scripting (XSS) and SQL injection back in the app days. It's the exact same thing. It can come from anywhere. And so you really need to think about putting LLM firewalls in your orchestration layer where your messaging passaging is going on. Like, I am taking things in an orchestration layer from an agent, from content, from the data, passing it to LLMs, taking the output of LLM passing it to another LLM, you know, your orchestration's really your router. And so you need to inspect for prompt injections at your router, quote unquote, in your AI, not just this external viewpoint of where things are. And so I think that that is a, another sort of, I don't think the market or the industry has gotten to the point of recognizing these kinds of things yet.

Diana Kelley 13:47

Yeah. And I love that you're pointing out the input validation and sanitization, because it, I've been thinking about that a lot. You know, we looked at something like SQL injection, it's a pretty easy problem. Now I realize it's been pernicious and it's still out there on the, you know, we still see it, it's still exploited, it's still not, you know, we still see developers not doing it, but it's an easy problem. We know the fix. I start thinking about how we sanitize inputs to LLMs, and it's a - I mean, the complexity of that, because we have to be able to accept what humans saying, humans are not -

Caleb Sima 14:22

You don't. Yeah. And, and also, like, look at cross-site scripting as another great example, as a relevant, which is people gave up on trying to do input for cross-site scripting. Yeah. They just decided to go on output <laugh>. Right. Which is, oh, okay, I can, I'll just, anything that's outputted, I'll just encode, which has been a sort of acceptable method on refusing this. But the problem with LLMs is there's, you know, I've always broken this up; and it's a control plane/data plane problem, right? Which is, SQL injection and cross-site scripting have very similar methodologies with very different impacts, right? But the methodology is effective, which is I'm taking untrusted data and treating it as trusted data. Which is, I'm taking data, making it into control plane data, and it executes in a browser or it gets executed by a database.

Caleb Sima 15:17

With an LLM it's the exact same thing. I am taking untrusted data from a person or a thing, or wherever it is, and it's being passed to an LLM like a command. Because an LLM does not understand the difference between a command and data, right? There's no difference between control plane and data plane. And as far as I can tell, I don't know if there's a way at which that does get created. At least, you know, there are different ways, but I have not seen the solution. And when that happens, maybe, maybe then you can, you know, solve a prompt injection problem. But like that is the real issue. And so you can't input this, you can't output this except to use another model in order to analyze the input and the output <laugh>. But yeah.

Diana Kelley 16:03

Right. We're, we're all just gonna be just, it's just going to be model on model on model, and we're just -

Caleb Sima 16:08

I mean, you say that in jest, but that is probably the reality.

Diana Kelley 16:13

It is. Yeah, yeah. And one thing that you've worked on actually to help people understand the risks is the AI Model Risk Management Framework for CSA. I was wondering if you could walk a little bit through, you know, what it is and how organizations can benefit from using it.

Caleb Sima 16:31

Yeah. It's a much more higher level view of, hey, what are the things that I need to think about when it comes to models, a governance program, a process. So like, you know, we're still at our infancy in trying to understand AI, and so we need to first understand, well, okay, where are models being used in our organization, right? Like, I think today it's easy. Most people are calling these foundational models, but if you wanna fast forward, or even some enterprise people are using Llama or Mistral or, you know, open weight models at which can be contained on their infra, but as we fast forward I think there'll be more smaller models. More and more people are gonna be downloading these from [open source model repositories like] Hugging Face, deploying them in enterprise environments. You, you have to start thinking about, okay, where are my models?

Caleb Sima 17:26

Where are they being used? What is the, what's the right, the right word to say this - what is sort of the supply chain, right, of the model? What is the S-BOM (bill of materials) of the model? Where does it come from? What's the reputation of the model? So you wanna create model cards to start saying, okay, one, where are my models? What are my models doing? What versions are they at? What reputation are they in? What data did it get created from? What is it acting on, right? Like, there's almost a behavioral profile that says [as examples] my model is making decisions about inbound candidates, right? My model is making decisions on customer, you know, purchase items and predictions. You know, like my model is rewriting emails. You know, like there needs to be these profiles, these things that gets defined so that you understand where in your organizations these things are being used, how they're being used, the reputation of the places that're having - all of those kinds of things.

Caleb Sima 18:24

You also need to understand where the models are being deployed. What are the sets of protections that you have around that model? Is the, if the model is open weight, clearly, is it just sitting on an open S3 bucket with no privileges and everybody can access it and change the model, and then therefore, and entirely disrupt what that model's behavior could be? Is it running an inference at places in containers that have the wrong access? You have to think about these things as production servers. You don't want engineers SSHing or accessing prod, and especially if it's running in a model, right? If data's being fed into that model and inverse, how are you protecting that? What's the privacy around that? Can you put them in enclaves and have the data go into the enclave in the model? And then we as an enterprise have no visibility of the data going into the model, right? Like, there's a lot of these sort of thought processes. I think we kind of drive down into that to kind of look at both at a, both at a high level of your model, the governance around your model, the organizational impact. Who in your organization should be involved in those models, how they get deployed, what are the kinds of things you want to think about? So that really is sort of the driver behind that.

Diana Kelley 19:36

Yeah. And I completely agree about AI Bill of Materials. We need different information. We need to understand model provenance, exactly. All the things you were talking about. And if we don't have that, we're not gonna be able to properly govern the deployments.

Caleb Sima 19:50

Yeah. And it's, and it's, you know, there's a lot of, we've already seen the vulnerabilities come from, you know, it's the basic ones, right? Oh, you can actually, you know, when you download models, they have this way of executing code. No one really looked at that <laugh>. So, you know, it's just like, there's all these basics, but you're right. Like it's really interesting because the problem with models, there's no visibility and you don't know how it will be behave. And there's no, so like if I download a model from Hugging Face, that is specifically around classification, right, how do I determine that it is classifying the right things, or the model that I'm downloading is the model that was truly intended to do the purpose that I think it's supposed to do. Like, how do I know about the deviations if someone got malicious with it, [maybe] they swapped the model out? Like there's just a lot of these problem spaces that is gonna be super fascinating.

Diana Kelley 20:44

Yeah. And, and very few organizations are scanning those models before they use them.

Caleb Sima 20:48

Yeah. I agree.

Diana Kelley 20:49

Which, you know, you scan a PDF that comes in your email, but you're not scanning, the models you're downloading?

Caleb Sima 20:54

And you know, largely, I think it's because as enterprise, we're so early, again, going back to this point, most people are gonna be using OpenAI and Anthropic and, you know, Google. So you're not - only really large enterprises today are really starting to go down in this, oh, I'm really gonna start like, downloading random models off Hugging Face and deploying them in prod. Like, you don't, there's not a lot yet, at least that I can tell. But that is coming, right?

Diana Kelley 21:24

Yeah. Yeah. Do you have any tips to talk to ML engineers, data scientists, help them? You know, we, we talk a lot about - in the MLSecOps Community - about education and helping to educate security teams on how AI and ML work, but also to educate the AI and ML folks on how risk management works. Have you had any opportunity to do that? And any guidance on how we can really make those conversations beneficial?

Caleb Sima 21:54

No, you know, I haven't, and you know, you bring up a really good question, and actually I think a pretty big gap. You know, because like Diana, I don't know what I would say. You know?

Diana Kelley 22:06

<laugh>

Caleb Sima 22:07

There's a, you know, there's obviously the data cleaning PII part. I feel like these are obvious things. There's a permissions part around like, hey, do all data engineers or scientists need to have access to the model in prod, or like, you know, or does it, do they need access? Do they need access to the raw data? Just the structured data, the filter data? Like, you know, there's a lot of this sort of permissioning kind of thing. But like, what would I tell a data scientist about how to think about security is a good question. I don't know the answer. I really don't. <Laugh>

Diana Kelley 22:49

There's something I read last year, and it simultaneously made me so happy and really, really sad. Which was some, a data scientist wrote a Medium post about Jupyter notebooks, which, you know, data scientists use a lot. And it said, hey, you know what, it's really not a good idea to store secrets - things like passwords and API keys - in the notebooks, because they're not encrypted. And this is actually a risk. So to all my other data scientists, folks, hey, we should really stop doing this. And <laugh>, you know?

Caleb Sima 23:21

Yes. You know what, I am also a very proud; I agree with you, Diana. That is a very proud, you know - what's also fascinating to me is most people are under the impression, or do not understand, that Jupyter notebooks are code. They don't treat Jupyter notebooks the same way they treat code.

Diana Kelley 23:44

Yeah.

Caleb Sima 23:44

And they do not go under the same level of scrutiny. They, a lot of them don't even get put into GitHub, either, <laugh>. Like, there's, there's like a lot of these things that when you talk to some of these enterprises, they, that switch has not flipped, right? That this is code, needs to be scanned and analyzed the same way that we do code today, needs to have the same permissions and thoughts around code today, because actually notebooks as code in production is a thing, right? And so, like, this is something I think most security people are just not aware of yet.

Diana Kelley 24:24

Yeah. I think more communication, more more collaboration, communication between the teams, I think that'll really help us in the long run.

Caleb Sima 24:31

Yeah. Yeah.

Diana Kelley 24:33

So you spent a lot of time studying AI and ML, helped CSA move forward and really start to establish frameworks. What are you looking at in the next six to 12 months? Where do you think the real issues, where should we be focusing our attention as security professionals?

Caleb Sima 24:53

Man. You know, this is an interesting one. I've always sort of been waiting because I have, by and large, I've felt that quote unquote "AI security" has not really hit mainstream yet, because most enterprises have just not gotten to the point of just saying, what is my use case for this thing? <Laugh> and deploying it. However the things that I think are gonna become super interesting and challenging security problems in the future is obviously agents. Right? Like we have been using - I actually have a presentation where I've got this model of sort of how we use AI today, how we use AI tomorrow, how we use AI in two years. And I'll sort of represent that verbally here. You know, we have largely interacted with it with text, right?

Caleb Sima 25:54

We have largely used public information. Our use cases have been more about search and translation and creation in terms of our models, right? And what we do with it. I think you're seeing the movement now, like I would call, I call that yesterday. I call, maybe the call is today. The way we interact with it is now text and voice, right? And you're seeing that start to kind of come into play. Images, obviously you're, you're passing in images for this. A lot of the sort of ways at which we're starting to use this in some instances is more friends, dating, you know, there's virtual girlfriends, AI girlfriends. There's mentoring, there's a lot of these sort of, actually, who was it that just - was it Mark Benioff that says he uses AI as a therapist right now? I think he just published that a week ago or two weeks.

Diana Kelley 26:55

Oh, I hadn't seen that. Alright.

Caleb Sima 26:55

Yeah, you know, it's, now you're starting to see it take off a more, more of a advising, human interaction kind of capability. You're starting to see it in agents start to make some decisions, and it's starting to get some actions, right? So, hey, hey, AI, I really need to solve my - I'm just gonna make this up - calendaring problem. Can you figure out how to help me in calendaring? Yes. Here is your calendaring today. I see someone has invited you to calendar tomorrow, I will make the changes on your calendar to adopt. Right? Okay. Now it's both thinking, reasoning, and deciding and acting. Right? Then now things start getting really, really interesting. <Laugh>, right?

Diana Kelley 27:44

Oh, yeah.

Caleb Sima 27:46

And then you start thinking about permissioning. As you start thinking about intent, you start thinking about this. And then if you go to tomorrow, where I think, and also by the way, the data is starting to become enterprise data, right? Like, that's where all this is. And then tomorrow you may interact with it at, in video, you may interact to it with robotics, right? Machines and physical things and how you may interact with it. I think the data that it consumes will be personal data, enterprise data, public data, all that. And maybe real time data, right? Like when - no one's really thought about an LLM streaming and analyzing logs in real time, right? Because the context windows just aren't there. The capability is just not, performance is just not there, but it will get there. And so it starts analyzing real time data. Then when you think about the things it does, it will then start reasoning, deciding and acting at scale.

Caleb Sima 28:37

So now, instead of making a decision on my calendar, I might manage the entire enterprise's employee's calendaring, right? At scale. Or manage production infrastructure monitoring at scale, and making decisions on when to reboot something, or not reboot something or patch something, or not patch something. I think security, when we start getting into these modes, becomes super fascinating. Like the example [that] always comes to mind is if you have an AI assistant, you have to give it permissions to access your email, and you probably will give it permissions to access your social network. How do you prevent it from posting your email to your social network? Like, it has to have permissions from both. It doesn't have the knowledge to know not to do that. It's a great question. If someone prompt injected it to do that, would it not know how to do that or not?

Caleb Sima 29:37

Is a great question. And if it did do that, what repercussions does it really have? How do you punish an AI? How do you hold an AI accountable? You know, <laugh> is like soup. So I think those are really, really fun problems, really fun security issues to think about. I think about when LLMs models-to-model communication would be super fascinating and super security interesting, because, hey, like right now, they all communicate via English and text, but that is not the most efficient way to communicate. So when an LLM model starts talking to another LLM model, it'll be like, how will it, if you say, just make up the most efficient way to communicate to another model, what would it do? You know, like all, like, when that starts happening, like how do you even get visibility? Like how do you, like, it's just very mind boggling. So these are futuristic, funky things I like to think about.

Diana Kelley 30:33

No, it is, it's really fascinating. Somebody actually has already done a worm that they called [Morris II] that was on indirect prompt. And there's an interesting chaining attack where if you're using LangChain, there's MathChain, and what it does is it translates the equation into Python. So it can be parsed down the line by the next LLM. And the input attack here is rather than giving it an equation, giving it Python commands, which then -

Caleb Sima 31:03

Yes.

Diana Kelley 31:04

Yeah. I mean, so I know. How are we going to - anything else that you [can] think of, you know, governance and how we're going to govern these systems as we continue?

Caleb Sima 31:15

No, you know, I wish I had great answers to what does governance look like? But the most simplistic way I think about governance is I need to treat AI the same as I would treat an individual. So what are the permissions at which I give it? What's the restrictions? What's the time? What's the guardrails? Like, these are all like, what's the job? What's the focus? These are all the things you try to restrict and say, you know, what is the same governance and how can I do that? But, you know, I think everybody and their brother is trying to figure out what is governance for AI to some extent. I honestly, I think we are kind of shooting ourselves in the foot by, in some instance, by not - like, we don't even know what's coming in the next year. And so we kind of like make this stuff up and it just becomes irrelevant in six months. Right?

Diana Kelley 32:12

Yeah. So as you're looking though out into the, and you must hear so many and see so many great ideas, you know with White Rabbit, is what are you looking at? What are you excited about in terms of AI security in the future?

Caleb Sima 32:24

I am excited. You know, AI is both the most over-hyped thing in the world, but also not without some real relevance there, right? Like, it is going to change things, but I think very similar to cloud, it will take time, right? Like cloud, everyone got super hyped about cloud. Everybody was lifting and shifting their systems into cloud, calling themselves a cloud company. And then there was like this whole like dip of, oh, the crash of the cloud height. This is not - it's actually more expensive lifting and shifting my stuff into cloud than it is me running it on prem. And then you started seeing the startups who started truly building their companies around cloud. Like their entire company was cloud first.They molded themselves around the technology, not try to mold the technology around their business, right?

Caleb Sima 33:31

And then that started really, I think creating, oh, wow. It's not lift and shift, it's shared services, it's scalability, it's, you know, all of these things that, oh, I can now truly take three people and do what 15 did. Right? And, and I think that's when you started seeing what was really capable, that the shift of capabilities happened. And I think AI is the same, right? We're seeing a bunch of enterprises. Chatbot is, you know, <laugh>, rewrite <laugh>, some simple, you know, [actually I'm in the middle] of writing a document (a blog post) right now about "no one's defined what an agent is." So everyone who has some automation is calling it an AI agent now. You know, it's just like, it's, everyone's just doing the same thing we saw in cloud and, and the hype is dying as it should. Because I think there's way too high expectations for it. And I also think people are implementing it wrong. But what I'm getting excited about what you're seeing today is people who are truly building AI core companies and what that looks like. Similar, what is cloud core company? That's what changed everything. And AI core companies is the thing that I'm saying. They're wrapping their business around AI because that's what they're starting with. And so that is fascinating to watch.

Diana Kelley 34:55

So the AI native companies.

Caleb Sima 34:56

The AI native, yes. And, you know, I think it's gonna take us a year or two to really start seeing the real value being created by AI native, right?

Diana Kelley 35:07

Yeah. Fascinating.

Caleb Sima 35:08

What do you actually - is it just about me, but Diana, I'd love, I'd love to hear - I wanna hear your thoughts on some of the futuristic and these things.

Diana Kelley 35:18

Well, I completely agree with you that I think we have to get through the hype cycle and the trough of despair and the whole sense of, oh, magically the sparkles are gonna do everything we want them to do. That's not the reality. I like that people are starting to actually discover the real use cases and understand also the time and effort it takes to train up the AI, because some folks decided they were gonna do a lot of training and use cases that you really don't need AI for. So I think that people are getting a lot more grounded. And I do like some of these, these like approaches that are kind of leveraging the AI out into a way that a human isn't gonna be as good at.

Diana Kelley 36:06

So one of the things I got really excited about recently is there's a disinformation bot that, yes, that some researchers have developed. And this thing has all the time in the world to be very patient. It doesn't get upset. It doesn't start saying, you know, you first, and it's actually been really effective with people that have gone down really deep conspiracy holes, and it's hard to talk them out of it, but the bot has the patience and the time to continue having that conversation, and it's been fairly successful.

Caleb Sima 36:39

Is this thing out, can I go use this now?

Diana Kelley 36:42

I read about it. I don't know if it's usable, so it may just be within research, but it's that approach, it's starting to, to turn this around. It's not just magic that's gonna fix things, but how does AI do the thing that it's best at doing? Also, obviously we see this with machine learning, but massive amounts of data in security. There's no way that we can parse it all. And machine learning AI are so powerful there. So those are some of the things that I get excited about. The stuff that humans don't do well, it's nice when it's a chat bot that we interact with and it gives us an outline for a blog that we're writing. But that to me, isn't the real power of the power is that part that humans really could need in system. Like, who has the patience to help somebody out of a conspiracy hole? Not a lot of people.

Caleb Sima 37:32

I would, I actually would love to have a troll bot, speaking of disinformation bot, that will analyze the social network posts and then will automatically keep track of a troll score so that you can immediately filter them out. <Laugh>

Diana Kelley 37:51

I think you just created a new service.

Caleb Sima 37:53

That is something AI would be good at. You can look at this thing and be like, oh yeah, over a history of this, this guy's a troll. And like every post, yeah, this is a troll post. Like, I'm gonna build a score. You can get a filter. It says anything in my stream that is like a below 80 troll score, completely ignore.

Diana Kelley 38:12

Right. They're just out, they're no longer. Yeah, I know. I mean, I think there are a lot of, as, as we move past the, it's magical gonna solve all our problems and into the actual problems we can solve, I think it's gonna be pretty fascinating.

Caleb Sima 38:24

Okay. Awesome. Awesome. Yeah, it's good. There's a lot of cool stuff I think that we're really gonna see come out. So I'm, I am excited about that.

Diana Kelley 38:34

But we do have to keep it safe. And thank you for your work at the CSA AI Safety Initiative. How can people find out more information about that?

Caleb Sima 38:44

Just head to the webpage and you can go there, find all the information. We've published, you know, all the documents, guidelines, help join one of the groups. Would be happy to have you and help sort of define these, these guidelines, which I think are very much needed.

Diana Kelley 39:03

Yeah. It's fascinating. And again, to underscore yes, join. Be a part of the work. If you're listening to this and you got wowed by what Caleb was saying and said, I wanna help, you can help. So please go over to CSA and join the safety initiative.

Caleb Sima 39:17

I'm also going to be publishing, actually maybe today, maybe tomorrow, my - I'm gonna post a blog that I've been working on, on building a little bit. I wanted to build a architecture of what does the AIOps look like. So I have one and I've been working on it for a little while. I'm gonna publish that probably today or tomorrow. You can, so if you want on my LinkedIn I've got a newsletter. I'm gonna post it there.

Diana Kelley 39:47

All right. So everybody be on the lookout because by the time that this airs, you'll have published it I think. So check out Caleb's AIOps. And security's gonna be in there, right?

Caleb Sima 39:57

There's a little bit. It's mostly like, hey, here's how, and then I do put in my my little, there's a little small plugs around. And if you're security, these are the things that are more relevant for you. Yeah.

Diana Kelley 40:07

All right. Thank you so much. It was so good to speak with you. And, and thank you so much for sharing your, all your insights and everything you're working on. It's just amazing.

Caleb Sima 40:17

Happy to be here.

Diana Kelley 40:19

And thank you everybody who stayed here and listened to this episode of MLSecOps. Once again, I'm your host, Diana Kelly, and we are so grateful for your support of this community and all the work that our leader Charlie McCarthy does to help educate people on AI security. We'd also like to thank our sponsor Protect AI, and of course, a huge thank you to you, Caleb Sima, for being here. We're so grateful. If you want to check out the show notes for links to the resources that we mentioned, please do that and we'll see you next time on the MLSecOps Podcast. Thanks.

[Closing] 

Additional tools and resources to check out:

Protect AI Radar: End-to-End AI Risk Management

Protect AI’s ML Security-Focused Open Source Tools

LLM Guard - The Security Toolkit for LLM Interactions

Huntr - The World's First AI/Machine Learning Bug Bounty Platform

Thanks for listening! Find more episodes and transcripts at https://mlsecops.com/podcast.