Audio-only version also available on Apple Podcasts, Spotify, iHeart Podcasts, and many more.
Episode Summary:
In this episode, Dr. Gina Guillaume-Joseph shares her journey from predicting software failures to pioneering secure agentic AI at Camio, emphasizing data integrity, zero trust, bias audits, and continuous monitoring to build safe, lifesaving real-time analytics solutions.
Transcript:
[Intro]
Diana Kelley (00:08):
Hello, and welcome to this edition of the MLSecOps Podcast. My name is Diana Kelly and I'm the CISO here at Protect AI, and I'm absolutely thrilled to be speaking with Dr. Gina Guillaume-Joseph today, who is currently serving as the CTO at Camio. So Gina, you've had an absolutely incredible career and also an incredible journey specifically in AI. And I was wondering around 2012, what was it that inspired your shift into AI and how did your early work shape your perspective here in this field?
Dr. Gina (00:43):
Well, in 2012, I was a systems engineer at the MITRE Corporation. MITRE operates federally funded research and development centers for our US government. And at that time you know, AI was just starting to come around, particularly deep learning. It was starting to reach that inflection point. And so I saw an opportunity to bridge the gap between traditional systems engineering and the next wave of intelligence systems. And so I then decided to apply to George Washington University. They were leading a program with a PhD in systems engineering with a focus on AI, and I was accepted. And the beautiful thing about that is MITRE does pay for your educational attainment. And they were able to reimburse me the tuition fees for the program. I delved in, decided that I wanted to build a model to predict software project failure based on where I was at MITRE.
Dr. Gina (01:50):
MITRE comes in when there is a technology transformation journey that needs to take place within the government, or, you know, something bad happens, right? Or projects fail and we come in to assess and understand the why and provide solutions to counter it. And so I decided to build a model around that. And so that began my early you know, deep dive into AI. I used a logistic regression model for predicting project failures. Also used successful projects that, you know, that did well in order to understand, you know, my data, the assumptions, and what really was the underlying challenges with project failure.
Diana Kelley (02:44):
Yeah, I mean, what an interesting... I wanna deep dive on the project failure, but I know we're gonna talk about AI, but sometime offline, I'd love to hear more about what your findings were there. But, you know, working on the logistics regression models for those failures, I was wondering, you know, how did your understanding of clean and reliable data evolve? What's the importance of it, and what should organizations know about how they prepare data for their AI systems?
Dr. Gina (03:11):
Absolutely. So data quality and model assumptions matter as much as the algorithms themselves. Even the best models fail if the data isn't clean, it isn't structured and it isn't representative of the environment. And so part of my research was to go out and capture the data. And that really took the bulk of the time for my four years in the program, is identifying where the data was coming from, making sure I had representative data of failed and successful projects, ensuring that it aligned with the goal of my thesis, and then being able to ensure that then I used a system to ensure it was clean, it was structured. I used R to build the model and identifying which model was best for this use case that I was also delving into. And so that was really important in terms of my understanding of what is the importance of clean structured representative data.
Diana Kelley (04:16):
Hmm. Yeah. That's it. And you actually sometimes describe this very succinctly as "garbage in garbage out." I think you've talked about that, but maybe add a little bit more to how you've seen that and how you've seen it be useful. And then also what have you learned about the best practices to ensure that you've got the data quality so that you're not gonna have bad outputs and you're gonna ensure safety and reliability of that system?
Dr. Gina (04:49):
Yeah. Oh my goodness, yes. Garbage in, garbage out. Yeah, I learned that because again, I tested my model. I failed a couple of times with garbage, trust me. You know, had to present the, you know, the data to my advisors and getting the feedback from them to know that, hey, you know, I've got bad data that could lead to unreliable model, which then could lead to me not, you know, not graduating. So that, that was
Dr. Gina (05:23):
The circumstance that I was in. And so this mindset kind of carried over into real world where we're implementing strict data validation, strict data governments, doing bias checks to ensure the AI models are fair, transparent, they're robust, as well as they're responsible. Part of that was also balancing innovation with security. Yes, my model, you know, I was building my own personal model to, you know, to prove a theory or a thesis, but you wanna ensure that your models had ethical, they have security implications. You wanna ensure that you look at all the lessons learned around ensuring accuracy and privacy of your models, especially if they're mission critical models. So it's not just about efficiency, but it's also about responsibility, fairness, transparent. And also too, your model should be augmentative, not just autonomous. So one of the biggest takeaways from my experience in, you know, doing work in AI is that AI augments human decision making, not replacing it. Whether it's, you know, HR professionals using AI for talent insights or security teams leveraging AI video analytics, the best outcomes come from human AI collaboration.
Diana Kelley (06:51):
Yeah. Couldn't, couldn't agree more. Yeah. It's the partnership between human and AI rather than just, we don't matter anymore. As a CTO, I was wondering, what do you find is the most common misconception about AI adoption at organizations? Especially when you look at it through a risk and safety lens?
Dr. Gina (07:15):
Yes, that is, so one of the biggest misconceptions I see is the belief that AI is a plug and play solution. You set it and you forget it.
Diana Kelley (07:25):
It's so smart.
Dr. Gina (07:26):
You deploy it and that's it. It'll automatically deliver its insights without oversight. Yeah, no. In reality, it is continuous. Continuous monitoring, continuous updating, governance, to remain effective and secure. Another misconception is that AI security is just about protecting the models from external threats. No. While adversarial attacks are a concern, the biggest risk comes from within. You know, just as I learned in my software development days, you know, adversarial attacks, you know, attacking software, that can happen with your models. The bias training data, misconfigured access controls, unintended consequences of automation. So it's important for organizations to understand the risk, you know, externally as well as internally. And that requires some cross function collaboration to ensure that you are protecting your models from in, from outside. One other thing, AI safety and security, people think that it slows down innovation, but it's not true. A well governed, well organized AI system built with trust enables faster adoption because you're ultimately driving better outcomes and you're not going back for rework. And so the organizations that succeed the most with AI are those that are taking a security first approach, rather than thinking of security as building in and after.
Diana Kelley (09:15):
Absolutely. I mean, way back to Barry Boehm’s Law, any system, if you can build the security in or the reliability in early, yeah, you're gonna get better outcomes. Yeah, completely agree. Well, Camio is doing a lot of work in real time video analytics and things like monitoring swimming pools for safety, if there were no lifeguards on duty, for example, and identifying if there's a problem or there's an emergency. I was wondering how, you know, a use case like that, how do you balance operational functionality with the security of the ML models?
Dr. Gina (09:51):
Well, with a use case like this, you're talking about life saving. So accuracy, efficiency, security, all of those are really critical. And so our challenge is then ensuring our models perform with low latency, high precision, and robust security, because we are dealing with real world variability, life saving. And so how do we balance that? You know, we have edge processing where we're sending some of the videos to the cloud, but also we're processing some at the edge, the ones that we need to make real time impactful decisions. Like someone has fallen into a swim pool after hours and we need to alert security to go save them. And so then, you know, you're balancing that. Then you have to understand the zero trust security model where again, we have to encrypt video feeds in transit and at rest, ensuring that access is restricted only to those users who have the proper role-based permissions.
Dr. Gina (10:59):
And then we also do some adversarial testing and some continuous validation because we wanna ensure that our models are achieving the desire that we built it for. Again, because some of them are being used in these life life threatening situations. So they're tested against attacks, bias drift, environmental changes, and other things that could occur. And then we're also ensuring that within these compliance driven environments that we would use our models in, we are leveraging on device processing to perhaps blur and obscure facial detection, so that we are, you know, not exposing those personally identifiable information. So ensuring that we are building in that security first principles into our machine learning pipeline is part of what we have to do, because, you know, we are building a lifesaving solution.
Diana Kelley (12:01):
Yeah. And I love that. So it's security by design and privacy by design too. Yeah. So important. So one of the first things we ever talked about was agentic AI, and I know you're doing some really fascinating things at Camio. I was wondering if, first maybe you could just briefly explain what agentic AI is for the audience and then talk about the opportunities and challenges that we're seeing in the space.
Dr. Gina (12:25):
Yeah, so agentic AI are systems that can operate independently not completely autonomous, augmentative, we talked about. So they can operate, make decisions using real world data and using, you know, existing data. And without the step-by-step instructions that, you know, traditional AI models are using. So particularly, you know, in a scenario with, you know, what we're using-Camio is using it for, is automated threat response. So using agentic AI to detect intrusion, to analyze the intent, to trigger an appropriate countermeasure or response, as well as alerting the humans, alerting the security system, to lock down. Oftentimes without human intervention. And so it automatically knows that we've got a threat, it understands the threat, it analyze the threat, it sends the trigger for the countermeasures, and then it takes action.
Diana Kelley (13:39):
Got it. All right. Thank you. So a lot of security we need to think about there and when we're having some decisions made, right, by the AI.
Dr. Gina (13:51):
The ethical concerns too, because if the system is, you know, operating autonomously, it could make some decisions that could counter or go against, you know, saving a human life.
Diana Kelley (14:08):
So specifically in agentic, is there anything, you know, we talked about AI security and robustness, but is there anything that autonomy, does that create new intersection or vulnerability points? And then what do we have to add into the process to make sure that we're not just securing AI, we're securing agentic AI.
Dr. Gina (14:31):
Exactly. So the biggest challenge is autonomy versus oversight. So how do we ensure AI makes the correct decisions without introducing excessive automation risk? So you gotta strike the balance between AI automation, human control, that is really critical. How do we do that? I don't quite know, but...
Diana Kelley (14:53):
Work in progress?
Dr. Gina (14:55):
It's a work in progress. Another thing that you gotta think about is explainability versus compliance. So, you know, many industries require clear decision rationales because of just the nature of the business, right? So black box AI, it's a challenge when accountability is required in a legal or regulatory environment. So you gotta make sure that you're documenting, you're tracking, where's your data coming from, how is it being used? Those are all critical because we know we have, we do have laws that we, you know, we have to provide that. Adversarial attacks and manipulation. So it's more susceptible to model spoofing to adversarial perturbations or media attacks that could deceive decision making processes because it's slowly injecting malicious data into the models or it's, you know, it's spoofing. There's just a number of challenges that we have to look out for with agentic AI because we're giving it the autonomy.
Diana Kelley (16:08):
So you are using and leveraging not just agentic AI, but also large multimodal AI models at Camio. That's a bit of a mouthful. What is a large multimodal AI model, and how are you leveraging them to accomplish the great work that you're doing?
Dr. Gina (16:29):
Yeah, large multimodal models, they combine multiple input types. Video, audio, text, sensor data, to create richer, more context aware AI systems. So instead of relying on just one form of input, they synthesize multiple data streams to improve accuracy in decision making. So for example, in video analytics, a traditional model might only look at and analyze images, but multimodal looks at the video feed to detect movement and any anomalies, audio signals to recognize distress calls or gunshot wounds or gunshots, text data like access to logs to identify suspicious activity, sensor data for temperature or motion control or sensors for awareness of your environment. And so the fusion of the different sources allows for more reliable context aware AI systems where you're also reducing false positives and improving the situational awareness of, you know, of the model.
Diana Kelley (17:44):
Fascinating. And that's a lot of work, it sounds like. When you're evaluating the security of open source, customer based models that you bring into the system, if you're ingesting them or using them, how do you make sure that they're going to be secure, that they're reliable? Do you use a process, people, technology approach, or is there something else?
Dr. Gina (18:10):
Yes. So we have to follow a different approach. I mean, people, process, technology is the foundation, right? You wanna make sure that you have dedicated security and AI ethics reviews, where every model has to undergo security, bias, privacy assessments, before we deploy. Testing, red team adversarial testing, to simulate some of those bias, some of the things, the biases, some of the data poisoning, invasion attacks, and any other vulnerabilities that could come in from inside and outside. From a process perspective, you wanna track your data for lineage, lineage tracking, to ensure that the models aren't trained on contaminated, on biased, or compromised data sets. That's one of the things that I was doing with my research. I had to ensure that data provenance tracking was in line. And then continuous model monitoring. You don't just create it and build it and set it and forget it.
Dr. Gina (19:17):
You gotta look at and monitor its progress and it's degradation over time and ensure that you're retraining the models accordingly with new data sets. And then the tools, so you gotta do some, you know, model scanning. You wanna look at, you know, security frameworks like MLSecOps pipelines to scan for backdoors, for attacks, for vulnerabilities. All of those things are really critical and important to AI, agentic AI, to ensure that the safety and use of the product. And then also, part of that is to ensure that it meets the highest standards of security, reliability, and you ensure that you're injecting all the ethical AI principles into the model.
Diana Kelley (20:06):
Well, we're definitely big fans of model scanning and red teaming models here. So, yeah, completely agree with that. So I'm just curious, what's next for your team at Camio? What are the big innovations or projects you have on the horizon?
Dr. Gina (20:22):
Agentic AI.
Diana Kelley (20:23):
Okay.
Dr. Gina (20:25):
Multi-model intelligence. Those are the things that we're looking at. So, those are some of the key innovations, you know, on the horizon. Autonomous AI driven security workflow, so moving from just alerting, but decision making of the AI systems that they can initiate those responses autonomously, augmentatively, with humans of course in the loop. Edge processing so that we can provide real time analytics with lower costs and with enhanced data privacy. And industry specific AI applications because we're using the models across healthcare, transportation, government, industry, industrial security. And so being able to build out these models to specifically address some of those unique sector needs. And so yeah, we're gonna continue to push the boundaries, and do great work to protect humans.
Diana Kelley (21:27):
You got a busy year coming up and a lot to do, as do a lot of us in the AI space. I was wondering, given everything you've seen, I know it's gonna be hard to do it, but if you could sort of distill down one nugget of wisdom that you wish everybody understood about building reliable and safe AI systems, what would that be?
Dr. Gina (21:49):
What would that be? AI systems are not going away. We all know that. Look at the news. There's a new model coming out every week.
Dr. Gina (22:00):
And so the data, the data's massive. So security shouldn't be an afterthought. It should be foundational design principle for AI development. And we don't wanna rush AI adoption without securing our data pipelines, without securing our models, without ensuring that our decision making frameworks are secure. We wanna build, you know, the zero trust architecture. We wanna build the explainability and transparency and also have ongoing security and bias audit. Embedding these day one from when we're identifying the requirements for the model. Embedding that into our AI models will ensure that it continues to be a force for good. It's a trustworthy force for good, and removing as much of the risks or unintended consequences as possible.
Diana Kelley (22:53):
Yeah. Thank you. Such sage advice. And thank you so much for your time. Taking time outta your busy, very busy life and job to come here and share your wisdom with us. We're really, really grateful that you joined us.
Dr. Gina (23:08):
Thank you.
[Closing]
Additional tools and resources to check out:
Protect AI Guardian: Zero Trust for ML Models
Recon: Automated Red Teaming for GenAI
Protect AI’s ML Security-Focused Open Source Tools
LLM Guard Open Source Security Toolkit for LLM Interactions
Huntr - The World's First AI/Machine Learning Bug Bounty Platform
Thanks for checking out the MLSecOps Podcast! Get involved with the MLSecOps Community and find more resources at https://community.mlsecops.com.