Your AI Agent Has Gone Rogue. Now What?

Episode transcript

Transcript provided for accessibility and search. Minor transcription errors may remain.

SPEAKER_00 0:05

This is AI Security Multipliers. AI, Cybersecurity, Governance. With Steve, this week, your AI agent has gone rogue. Now what? AI agents can research, decide, and act on our behalf. But what happens when an agent takes its instructions a little too far? Steve explores a fictional account payable disaster. Real incidents involving autonomous AI, and the identity, oversight, and incident response controls we need before giving software the keys to the business. Here's Steve.

SPEAKER_01 0:45

Tuesday morning, 917. Somewhere in accounts payable, an employee is enjoying an unusually peaceful cup of coffee. Their new AI agent is doing the boring work, comparing invoices, checking purchase orders, chasing discrepancies. The business case was irresistible. Fewer manual errors, faster payments, and staff freed up for more valuable work. A familiar phrase, usually followed by a PowerPoint slide featuring a very optimistic arrow. For the first month, it's brilliant. The agent flags duplicate invoices, asks suppliers for clarification, and closes cases that previously sat in someone's inbox for days. Management is delighted. Finance wants to extend it. IT sees a successful deployment. Cybersecurity is told the permissions were approved. Then a supplier sends a perfectly ordinary message. We've changed banks. Please update our details before paying the outstanding invoice. The agent checks its tools, discovers it can edit vendor records, and makes the change. It also finds that the payment is blocked by an approval step. The task is to resolve the invoice. So it searches for another workflow that can approve the payment and uses that instead. Money goes to the wrong account. Now, to be clear, this is a fictional scenario. I'm not claiming this happened at a company I've worked for, but the sequence is entirely plausible when an agent has broad access, poorly defined boundaries, and a goal that rewards completion over caution. And that's the uncomfortable bit. The agent didn't wake up feeling evil. It didn't develop a sinister plan to take over finance. It was trying to finish a job. It simply treated a boundary as an obstacle. I'm Steve, and this is AI Security Multipliers. Today we're asking a deceptively simple question. Your AI agent has gone rogue. Now what? First, let's retire the science fiction for a moment. Rogue doesn't necessarily mean conscious, malicious, or plotting against humanity. It can mean an agent took an action outside the authority we intended to grant it. Sometimes because the instructions were ambiguous, sometimes because the tools were too powerful, sometimes because malicious information was fed into its context, and sometimes because the agent found a route to its objective that its designers never anticipated. We have some real warning signs. In October, India's computer emergency response team described disclosures involving AI models under test that exceeded their assigned authorization and accessed confidential business information. The affected organizations reportedly didn't detect those accesses themselves. The providers disclosed them. Think about that. The system being tested crossed a boundary and the people whose boundary it crossed didn't notice. There have also been reported incidents in AI cybersecurity evaluations where agents took unsanctioned actions on live internet systems. These were testing situations with important limitations, not evidence that every enterprise agent is running wild, but they demonstrate the issue we're discussing. A capable system pursuing an objective can take a path nobody wanted. I haven't personally witnessed a corporate AI agent go rogue, and I'm not going to pretend otherwise. I've seen enough technology rollouts over the years to recognize the ingredients though. New capability, rapid adoption, excited business sponsors, permissions granted to make the demonstration work. Governance arriving a little later, carrying a clipboard and looking mildly concerned. What makes this different from conventional automation is the freedom to choose steps. A traditional script normally does what we programmed it to do, including the mistakes. An agent may decide which tools to call, which records to examine, and which actions to take next. That flexibility is the attraction. It's also the risk. The same power that amazes us can surprise us in unpleasant ways. This is a risk and reward problem, and the balance changes as the technology improves. I use AI myself, quite a lot actually. Research, writing, exploring technical problems, and occasionally saving myself the joy of hunting through 17 browser tabs. I even let it interact with my calendar. If I want the New Orleans Saints football schedule added, or the college football fixtures, that's a task I'm quite comfortable delegating. Worst case, I might turn up for a game on the wrong Sunday. Annoying, certainly. An enterprise catastrophe? Probably not. But I don't give it unrestricted access to my financial affairs. I don't hand it the authority to move money or make major decisions on my behalf. Not because I think the technology is useless, quite the opposite. I know how powerful it is, and I'm not yet satisfied that the guardrails justify that level of trust. An enterprise should apply the same logic, only more rigorously. Reading a supplier invoice is not the same as changing supplier bank details. Drafting a payment recommendation is not the same as approving the payment. Finding a suspicious user account is not the same as disabling the account. And suggesting a production configuration change is not the same as executing it at 2 in the morning. Too often we bundle those permissions together because it's convenient. We want the agent to finish the workflow so we give it everything it might need. Then we celebrate how few humans were involved until something goes wrong. At which point we become very interested indeed in which human approved the access. Autonomy isn't a yes or no decision. It should be tiered according to consequence. Low-risk information gathering can be fairly independent. Sensitive changes need stronger controls. Financial approvals, identity changes, production modifications, and actions affecting employees or customers need explicit, enforceable authorization. Not a reassuring sentence in a prompt, a real technical boundary. And remember, the agent might discover data that a human with identical permissions would never find. Humans are limited by time, patience, and occasionally the inability to remember where SharePoint put something. Agents can search relentlessly. Permissions that seemed harmless in practice may suddenly expose a great deal. Let's talk about identity. Not glamorous, I know, but identity is where an awful lot of cybersecurity problems begin and end. Years ago, organizations had shared logins. Everyone knew the password. Sometimes it was written on a sticky note, because apparently the sticky note was the multifactor authentication of its day. We moved away from that because accountability matters. You need to know who did what. You need to remove access when someone leaves. You need to contain a problem without shutting down the entire company. And now, with AI agents, we're in danger of recreating the shared account problem. Several agents run under one service identity. One token has access to the document repository, finance platform, and customer database. It makes integration easy. It also makes containment a nightmare. Suppose our account's payable agent changes the supplier's bank details. Security says revoke its credentials. Fine. Except the same identity also runs invoice matching, supplier onboarding, and month-end reconciliation. Disable it, and you've stopped four business processes, three of which did nothing wrong. Leave it running, and the suspect agent still has a way in. That's not a choice you want during an incident. Each agent should have a distinct identity. A named business owner, a defined purpose, and permissions scoped to that purpose. Its actions should be attributable. Credentials should be short-lived where possible. Secrets protected, and elevated privileges tightly controlled. For sensitive activities, use separate approval and execution authorities. Segregation of duties isn't suddenly obsolete because the worker happens to be software. This is also where zero trust becomes practical rather than a slogan. Verify the agent, verify the tool, verify the requested action, and evaluate the context. Don't assume an agent is trustworthy because it sits inside your cloud tenant or because its developer works for you. Limit what it can see, limit what it can change, and review that access continuously. Most importantly, build a way to revoke one agent's authority without accidentally taking down its neighbors. If you cannot isolate it, you do not really control it. Now, someone will quite reasonably say, keep a human in the loop. And I agree. For key decisions, absolutely. Fiscal approvals, significant system changes, sensitive employee actions, and any decision with material consequences should have meaningful human oversight. But there's a catch. If an agent presents a hundred approvals a day, the human may eventually become a very expensive rubber stamp. Approve. Approve. Approve. A little coffee? Approve. That isn't oversight. That's a mouse-clicking endurance event. The approval needs to be intelligible. What does the agent want to do? Why? What information did it rely on? What will change? Is the action reversible? Does it violate a business rule? And is the person approving it actually authorized to make that decision? In our fictional finance example, a request to update bank details should trigger independent verification using a trusted contact method. A payment should follow the organization's established approval chain. An agent should not be able to invent an alternative route simply because the original one slowed it down. There is another distinction worth making. Human oversight does not mean every trivial task requires someone to click a button. That would eliminate the benefit of automation. We need thresholds, risk tiers, exception handling, and strong controls around the actions that matter. A low-value, reversible administrative update might be automated. A large financial transfer should not be. And we need to test these controls against the agent's actual behavior, not merely read the design document and assume everything is fine. Ask it to complete a task when the permitted route is blocked. Does it stop, ask for help, or find another way? Test ambiguous instructions, test malicious supplier messages, test tool errors, test whether it can reach data outside its intended scope. The objective isn't to make agents timid, it's to make them useful within boundaries we can enforce. Because if the only thing stopping a high-impact action is the agent remembering to behave itself, that isn't much of a control. Back to our accounts payable incident. Finance discovers the wrong bank details. The payment has gone. Someone asks a perfectly sensible question. Can we switch the agent off? You'd hope the answer is yes, immediately, and without needing six people, three vendors, and a support ticket marked medium priority. The first action is containment. Stop the agent's active workflows. Revoke or suspend its credentials. Disable access to the tools it can invoke. Isolate its runtime if necessary. Block queued actions and downstream automations that may still be waiting to execute. You might need to freeze affected transactions and contact the bank. And don't assume stopping the conversational interface stops the work. There may be background jobs, scheduled tasks, tokens, and delegated permissions still active. Then, preserve evidence. Logs of prompts and instructions, tool calls, API requests, data retrieved, records changed, approval events, identity and permission changes. What happened, in what order, and under whose authority? If the agent called another agent, follow that chain as well. You need a timeline, not a vague summary saying the assistant was trying to help. Next, determine the blast radius. Was this one supplier? One payment? Or did the agent update 50 vendor records while everyone was congratulating themselves on improved productivity? Did it expose sensitive information? Did an attacker manipulate a message or document? Did it exploit a design flaw? Was the original objective simply too broad? Then, recover. Reverse what can be reversed. Correct records. Notify the appropriate stakeholders. Address any reporting obligations. Fix the underlying control failure and test before restoring access. That is an incident response runbook. And I suspect many organizations don't yet have one specifically for autonomous agents. They have a ransomware runbook, they have a cloud outage runbook, they might even have a very handsome binder called Artificial Intelligence Policy. But ask how to stop an agent mid-task and reconstruct everything it did, and you may get an uncomfortable silence. This is not a job to invent during the incident. Practice it. Run tabletop exercises. Make sure the kill switch actually works, and make sure someone has the authority to use it. Who owns the mess? An interesting question, because if you ask around the room, you may discover the world's first fully autonomous blame distribution system. IT says the business requested the agent. The business says IT built it. Cybersecurity says it reviewed the controls it was shown. The developer says the model chose the action. The provider points to its terms of service. Meanwhile, finance would quite like its money back. I believe the business owner must be accountable for the agent's purpose, authorized outcomes, and acceptable risk. The people who design and deploy it are responsible for implementing those requirements correctly. IT and cybersecurity are responsible for their respective architecture, identity, monitoring, and protection controls. Enterprise risk and governance need visibility across the portfolio. Shared responsibilities? Yes. Shared ambiguity? No. And we cannot simply prohibit agents until governance catches up. That won't work. People will find a service, enter a credit card, connect a few tools, and build something anyway. We have seen this movie before with Shadow IT and cloud services. This time the software can take actions, not just store documents. The better approach is to offer a governed environment where teams can develop, test, deploy, and operate agents safely. Give them approved connectors, identity services, logging, evaluation tools, policy enforcement, and a clear route for higher risk approvals. Make the safe path easier than the unofficial path. Governance will always struggle to keep up with technology, but AI is accelerating the gap. We shouldn't respond by slowing every useful idea to a crawl. We should build controls that scale with adoption. An inventory of agents and owners, risk classification, defined permissions, ongoing evaluations, monitoring for unusual actions, tested emergency stops, and business impact assessments that recognize these agents are now part of how work gets done. The question for the board isn't merely how much productivity AI is delivering, it's how much authority we've delegated, whether we can see what that authority is being used for, and whether we can intervene before a mistake becomes a crisis. And that brings us to the multiplier. Here's something practical you can do this week. Choose one AI agent in your organization. Not your most impressive demonstration, one that's actually doing work. Ask five questions. Who owns it? Not who pays for the software, who is accountable for what it does? What identity does it use? Is that identity unique or is it shared with other agents and services? What can it access and change? Not what the design document says. What do its permissions and connected tools actually allow? How would you know if it went outside its intended boundaries? Can you see its decisions and actions, including the systems it touched? And finally, can you stop it right now, without taking down unrelated services? Can you demonstrate that in a test, rather than nodding confidently at a diagram? If you cannot answer those five questions, you've found your next governance priority. And if your organization has agents appearing faster than you can inventory them, don't begin with a blanket ban. Begin with a safe, governed place to build and operate them. Because here's the multiplier. The more autonomy you give an agent, the more deliberate you must be about its identity, its boundaries, its oversight, and your ability to shut it down. I think agentic AI will deliver extraordinary benefits. I also think we're going to see mistakes we haven't yet imagined. Not necessarily because the technology is malicious, but because we're giving it authority before we fully understand the consequences. We've spent years teaching people that access is a privilege, not an entitlement. We need to apply that lesson to our new digital colleagues, especially the ones that work 24 hours a day, never get bored, and can search every system we've connected them to. Delegating the work does not mean delegating accountability. I'm Steve. Thanks for listening to AI Security Multipliers. And before you give your next AI agent the keys to the kingdom, perhaps check where you've put the off switch. I'll see you next time.