Your AI Is Lying to You. Very Convincingly.

Episode transcript

Transcript provided for accessibility and search. Minor transcription errors may remain.

Introduction

SPEAKER_00 0:04

This is AI Security Multipliers. AI, Cybersecurity, Governance with Steve. This week, your AI is lying to you. Very convincingly. Generative AI can produce extraordinary answers. Detailed, confident, professional, and sometimes completely wrong. But what happens when a hallucination isn't simply an embarrassing chatbot response? What happens when AI-generated information starts influencing business decisions, security incidents, financial analysis, or the actions of another AI agent? To discuss why the most dangerous AI answer may be the one that looks completely believable. Here's Steve.

SPEAKER_01 0:49

Imagine this. It's Monday morning. You've got a meeting in 20 minutes. You ask your AI assistant, give me a summary of the ACME account. Focus on the contract, the current issues, and anything I should know before the meeting. And it produces something brilliant. Three paragraphs.

It Sounds Right

SPEAKER_01 1:05

Beautifully structured. Customer history, contract details, recent issues, outstanding actions. It even tells you that the customer threatened to cancel during a meeting last Thursday. That's concerning. You walk into the meeting prepared to deal with it. There's just one small problem. The customer didn't threaten to cancel. There wasn't a meeting last Thursday. The AI made it up. But it didn't say, Steve, I'm about to invent something. It didn't put a little warning beside the paragraph saying, probably nonsense. It just told you, confidently. And that's the problem. The most dangerous AI answer isn't necessarily the obviously ridiculous one. It's the wrong answer that sounds exactly like the right one. I'm Steve, and this is AI Security Multipliers. Today we're talking about hallucinations. Although, as we'll discover, I don't particularly like that word. Let's clear something up first. AI isn't actually lying to you. Lying requires intent. If I tell you I completed my expense report when I haven't, that's lying. Hypothetically. Obviously. A language model doesn't necessarily know that it's giving you false information. It's generating a response based on patterns, context, and probabilities, but it's fundamentally trying to produce a plausible continuation. And plausible is not the same thing as true. That's why the term hallucination can be slightly misleading too. The model isn't sitting there experiencing something that isn't real. It's generating information that may not be grounded in reality. Sometimes

AI Isn't Really Lying

SPEAKER_01 2:30

that's obvious. Ask for a citation and it invents a paper. Ask about a court case, and it creates one. Ask about a product feature, and it confidently explains something that doesn't exist. Those are relatively easy to catch, but increasingly the errors are more subtle. Wrong number, wrong date, wrong person, wrong interpretation. A real document combined with an incorrect conclusion. 98% correct, and 2% completely fabricated. And unfortunately, the fabricated 2% may be the part you make a decision on. Humans have a problem here. We associate confidence with competence. Someone gives us an uncertain answer. Well, I think perhaps. Maybe. We naturally question it. Someone gives us a clear, detailed, beautifully written explanation. It feels authoritative. A eye is extraordinarily good at authoritative. It doesn't get nervous, it doesn't look away, it doesn't say um seventeen times, it doesn't shuffle through its notes, it produces the answer. Instantly, professionally, often with bullet points. Nothing says truth, quite like a well-formatted bullet point. And that creates something I think organizations need to take seriously. Automation bias. If a system is right often enough, people stop trusting it. Eventually they stop checking. Think about GPS. The first few times you used satellite navigation, you probably watched the road signs. After a while, you just followed it. Until one day it directed you through a field. AI can create the same behavior, except instead of arriving at a confused-looking

Confidence Isn't Accuracy

SPEAKER_01 4:02

cow, you might approve a financial decision based on information that isn't true. Now somebody is already shouting, RAG! Retrieval augmented generation. And yes, grounding AI in trusted enterprise information can dramatically improve accuracy. Instead of relying entirely on what the model learned during training, you retrieve relevant information, provide it to the model, and ask the model to answer using that context. Excellent. We've talked about RAG before, it's enormously useful. But RAG doesn't magically create truth. Because what happens if the source information is wrong? Or old, or contradictory? Imagine your company has three documents describing the same process. One from 2022, one from 2024, and one called Final Process. Version 7, really final updated, use this one, which is obviously the authoritative enterprise record. The AI retrieves all three. Now what? Rag can ground the answer in your data, it can't guarantee your data is correct. And this is where AI starts exposing another uncomfortable enterprise reality. Organizations have spent decades accumulating information, not necessarily managing it. Those are different things. Think about a large organization, ask a seemingly simple question. What is our official process for approving this? There may be an answer in SharePoint, another in the policy system, another in Teams, another in an email, another in somebody's OneDrive, and the process everyone actually follows may be completely different. Humans navigate this mess through, experience. You know who to ask.

RAG Doesn't Create Truth

SPEAKER_01 5:31

You know which document everyone ignores. You know Susan updated the process six months ago. You know the SharePoint version is wrong. The AI doesn't necessarily know any of that. It sees information. And increasingly, the organizations want AI to turn that information into answers, which means information governance becomes part of AI accuracy. Ownership matters. Authoritative sources matter. Retention matters. Metadata matters. Versioning matters. Data quality matters. Suddenly all those information governance projects people have successfully avoided for 15 years become AI projects. AI has an extraordinary ability to make neglected enterprise housekeeping strategically important. Let's make this more interesting. Cybersecurity. Imagine your AI security assistant analyzes an incident. Thousands of events? Identity logs, endpoint telemetry, cloud activity, threat intelligence, emails, network connections, it produces a summary. User account compromised through phishing. Attacker authenticated from this address? Privilege escalation occurred at 0214. Three systems were accessed. No evidence of data exfiltration? Fantastic. That might save an analyst hours. But what if one part is wrong? What if the privilege escalation never happened? What if there actually was evidence of exfiltration? What if the AI incorrectly attributed activity to the attacker? Now the hallucination isn't simply an incorrect answer. It's part of an incident response decision. Do we isolate the system? Do we notify customers? Do we invoke legal? Do we declare an incident? Do we notify regulators? Those decisions have consequences. AI can absolutely help cybersecurity teams investigate incidents. I think it will become extraordinarily good at it, for there's an important

The Enterprise Truth Problem

SPEAKER_01 7:12

distinction between AI-assisted analysis and AI-determined truth. We should be very careful not to confuse the two. And then we arrive at agents. Because hallucination is inconvenient when AI gives you an incorrect answer. It's considerably more interesting when AI can do something about it. Imagine an agent concludes, this account is compromised, and the agent can disable accounts, or this transaction is fraudulent, and it can stop payments, or this server is malicious, and it can isolate infrastructure. Or this customer qualifies for a refund, and it can issue one. Now an incorrect inference becomes an incorrect action. This is where the risk changes, the chain becomes information, interpretation, decision, action. If the first three contain uncertainty, the fourth needs controls. This is why human approval still matters for high impact actions. Why confidence thresholds matter? Why tour permissions matter? Why reversibility matters? Why audit trails matter, and why agents shouldn't necessarily be given every capability simply because technically they can use it. The question isn't can the AI do this? It's how wrong can the AI afford to be before we let it do this? That's a much better governance question. And this is where risk-based governance becomes important, because we could respond to hallucination by requiring human verification of absolutely everything AI produces, which would certainly reduce risk, it would also remove quite a lot of the benefit. If I ask AI, make this email slightly friendlier, I'm probably not assembling a review committee. If I ask it, summarize these meeting notes. Low

AI in Cybersecurity

SPEAKER_01 8:48

risk. If I ask, suggest five titles for my presentation. I'm willing to live dangerously but calculate the financial exposure of this contract. Different. Determine whether this employee violated policy? Different. Diagnose this security incident? Different. Approve this payment? Very different. The required assurance should increase with the consequence of being wrong. That's the principle. Not AI must always be verified. But the higher the impact, the higher the required confidence and oversight. That's governance. Risk determines control. Not fear, and definitely not enthusiasm. There's another control I think becomes increasingly important. Evidence. Don't simply give me the answer. Show me where it came from. If AI tells me our policy requires approval from the CISO, fine. Show me the policy, show me the paragraph. Show me the version, show me the date. If it summarizes a security incident, let me drill into the underlying events. If it gives me a financial number, show me the source data, if it tells me a customer made a complaint, show me the interaction. This doesn't eliminate hallucination, but it changes the relationship. We're no longer asking, do I trust the AI? We're asking, can I verify the answer? That's much healthier. Trust shouldn't come from how confident the AI sounds. Trust should come from evidence, provenance, controls, and the ability to independently validate important claims. In other

When AI Can Act

SPEAKER_01 10:15

words, trust but verify. Which sounds suspiciously like something cybersecurity has been saying for quite a while. I think this may be one of the biggest mindset changes organizations need to make. Traditional software creates an expectation of determinism. 2 plus 2 equals 4. Run the same calculation again. Still 4, hopefully. Generative AI is probabilistic. That means uncertainty isn't necessarily a defect we can completely engineer away. Models will improve, grounding will improve, reasoning will improve, verification will improve. But we should design systems assuming AI can sometimes be wrong. That means asking, what happens when it's wrong? Can we detect it? Can we reverse the action? Can we identify the evidence used? Can we reproduce what happened? Can a human intervene? What's the blast radius of that? Those are familiar cybersecurity questions. We design systems assuming credentials can be compromised. Networks can fail. Software contains

Designing for Uncertainty

SPEAKER_01 11:10

vulnerabilities. People make mistakes. AI should be no different. Mature AI governance doesn't assume perfect AI. It assumes imperfect AI and designs accordingly. Of course, there's an uncomfortable part of this conversation. Humans are wrong too, quite frequently. We've just had considerably longer to get used to it. Humans misremember, misinterpret, make assumptions, ignore evidence, confuse correlation with causation. Read the wrong document, send the wrong spreadsheet. Click, reply all. AI doesn't have a monopoly on questionable decision making. So the goal isn't to demand perfection from AI while accepting mediocrity from humans. The interesting question is, can the combination be better than either one alone? AI can process enormous amounts of information. Humans can bring context, judgment, experience, ethics, skepticism, at least on a good day. The strongest model may not be AI replaces human judgment. Or humans verify everything AI does. It may be something more collaborative. AI identifies, AI analyses, AI recommends. Humans challenge. Humans decide where consequences matter. And over time, we learn which decisions can safely become more autonomous. That's not as dramatic as replacing everyone with AI, but it's probably considerably more useful. So

The Multiplier

SPEAKER_01 12:34

here's this episode's multiplier. Take one important AI-generated output your organization already relies on. Not a hypothetical future system. Something being used today. A report, a summary, an analysis, a recommendation, a security finding. Then ask four questions. First, where did the answer come from? Can you identify the actual sources? Second, what happens if it's wrong? Not technically wrong, but business wrong. What's the consequence? Third, how would we know? Is there evidence? Verification. Monitoring? Human review? And fourth, can we recover? If the AI makes a bad recommendation or takes a bad action, can we reverse it? Those four questions tell you considerably more about AI risk than simply asking whether the model hallucinates. Because it will sometimes be wrong. The goal isn't pretending otherwise, the goal is making sure being wrong doesn't become catastrophic. That's the multiplier. AI is going to become more accurate, models will improve,

Confidence Isn't Truth

SPEAKER_01 13:34

grounding will improve, verification will improve, enterprise data will hopefully improve. Although perhaps let's not get carried away. But AI will continue producing answers that look authoritative. And sometimes those answers will be wrong. That's not a reason to stop using AI. It's a reason to use it intelligently. Understand where the information came from. Demand evidence when the consequence matters. Match oversight to risk. Limit autonomous actions where mistakes have serious consequences. Design for reversibility. And above all, don't confuse confidence with truth. Because your AI isn't really lying to you, but occasionally it can be extremely convincing while being completely wrong. And frankly, we've already got humans for that. But I'm Steve. This is AI Security Multipliers. And remember, AI should multiply your capability, not your mistakes. I'll see you next time.