The Day the AI Stopped Working

Episode transcript

Transcript provided for accessibility and search. Minor transcription errors may remain.

Introduction: The Day the AI Stopped Working

SPEAKER_02 0:04

This is AI Security Multipliers. AI, Cybersecurity, Governance with Steve. This week, the day the AI stopped working, artificial intelligence is becoming part of almost everything we do, from customer service and cybersecurity to software development and everyday business operations. But what happens when the AI we've come to depend on suddenly disappears? And what if our disaster recovery plans never considered that possibility? This week, Steve shares a real experience from Hurricane Katrina, explores the hidden dependencies we're creating with AI, and asks whether our organizations could actually operate without it. Because the technology that makes us more productive could also make us more vulnerable. Here's Steve.

SPEAKER_00 0:57

It's

When Disaster Recovery Fails: Hurricane Katrina

SPEAKER_00 0:58

Monday morning, 9.17.

SPEAKER_01 1:00

You've just arrived at the office, made yourself a coffee, and opened your laptop. Everything looks perfectly normal. The network is working, email is working, your applications are running, the cybersecurity dashboards are reassuringly green. Which, as we all know, is usually the point at which something terrible is about to happen. You open your AI assistant, type a question, and nothing happens. You try again, still nothing. Eventually, a message appears. Service temporarily unavailable. Please try again later. Fair enough. Technology occasionally has a bad morning. You get another coffee. Ten minutes later, your phone rings. It's customer service. Their AI agents have stopped responding. Then software development calls. Their coding assistants aren't working. Finance has a problem. HR has a problem, and the security operations team has just discovered that its AI investigation assistant is unavailable. Suddenly, somebody asks a rather uncomfortable question. Exactly how much of our business depends on this thing? Nobody seems entirely sure, but apparently it's rather a lot. And that's when someone asks to see the business continuity plan, which covers power failures, network outages, ransomware, data center failures, even the loss of a major cloud provider. But the AI services that have quietly become embedded in dozens of business processes? Well, nobody thought to include those. I'm Steve, and this is AI Security Multipliers. Today we're talking about what happens when the AI your organization depends upon suddenly stops working, and why this isn't just another technology outage. It's a question about how we're designing the future of our businesses. Now, this scenario might sound a little dramatic, but I've experienced something remarkably similar. Not with AI, with a data center. Over 20 years ago, during Hurricane Katrina, we had a data center in downtown New Orleans, and we planned for disasters. The data center was on the fourth floor, well above ground level. So when the floodwaters arrived, the equipment was still nice and dry. Excellent. The power went out, but we had a generator. Also excellent. Everything was working. Until the generator ran out of fuel, and we couldn't get any more. You see, having a generator is wonderful, but it turns out generators have this rather inconvenient habit of needing fuel. And when the surrounding city is flooded, roads are inaccessible, and emergency services are overwhelmed, getting somebody to deliver diesel becomes a little more complicated than calling your usual supplier. So, eventually, the systems went down. But that's alright. We had backups, stored off-site, in a very reputable, secure storage facility, exactly as you would expect. Unfortunately, we couldn't get to them either. Our backup tapes were perfectly safe. In fact, they were so safe we couldn't get to them. And that left us with a rather serious problem. The primary systems were unavailable, and the backups we needed to restore them somewhere else were inaccessible. The important lesson wasn't that we'd failed to plan. We had planned. We had a generator, we had off-site backups, we had considered flooding. But we made assumptions about what would remain available during a disaster. Fuel deliveries, road access, physical access to backup media. And when those assumptions failed, so did our recovery strategy. That experience has stayed with me because disaster recovery isn't simply about having backup technology. It's about understanding everything your recovery depends upon, including the things you assume will always be there.

AI Is Becoming a Critical Business Dependency

SPEAKER_01 4:37

One thing I've learned over the years is that businesses generally don't appreciate technology until it stops working. They treat IT rather like electricity. You walk into a room, flick the switch, and expect the lights to come on. You don't normally spend your morning wondering about the power grid, unless the lights don't come on. Then suddenly electricity becomes the most important thing in your life. IT is much the same. Email works, applications work, networks work, people do their jobs, and nobody thinks much about the infrastructure behind it until something breaks. Five minutes without a critical system can be inconvenient, five hours can be disruptive, five days can be catastrophic, and I've seen how quickly an organization can move from taking technology for granted to absolute panic. Here's another uncomfortable truth. A lot of business continuity plans aren't really plans for operating without technology. They're plans for restoring technology. There's a difference. If your network goes down, your recovery plan may assume access to cloud services. If your applications fail, it may assume access to backups. If your data center fails, it may assume you can recover somewhere else. But what if those assumptions are wrong? What if the alternative depends on the same thing that failed? And now we're introducing AI into that equation. Another utility, another dependency. Except this time we may have even less control over it. I use AI quite regularly myself. Research, writing, exploring technical questions, working through ideas. It's useful and it's fast. In fact, even ordinary internet searches increasingly give you an AI-generated answer before you've looked at a single traditional search result. It's becoming normal. And that's how dependency begins. Not necessarily with some enormous enterprise transformation program. Sometimes it starts with convenience. A tool makes something easier. People use it, they get used to it. Then somebody connects it to a business process. Then another, then another, and eventually what began as a productivity assistant becomes part of how the organization operates. Now introduce agentic AI. Systems that don't simply answer questions but take actions. They access applications, retrieve information, update records, execute workflows, make recommendations, and, within defined permissions, carry out tasks. That's a much bigger dependency, because now AI isn't simply helping someone do their job, it's becoming part of the job itself. And many of these capabilities depend on services outside the organization's direct control. We may not own the models, we may not control the infrastructure, we may not have visibility into all the underlying dependencies, and the service commitments may not match the criticality of the business processes we're building around them. That's what concerns me. Not that AI will occasionally fail, all technology fails. It's that our dependency on AI could grow much faster than our understanding of how to recover when it does.

The Business Impact of an AI Outage

SPEAKER_01 7:42

Back to our fictional Monday. It's now 11.30. The AI service is still unavailable. Customer service has switched to manual handling, which sounds perfectly reasonable. Until someone points out that the team now handles only a fraction of the volume it did with AI assistance. Cues are growing. Customers are waiting. Software developers are still writing code, but some of the productivity gains they'd come to expect have disappeared. Finance has employees trying to remember how a workflow operated before AI was introduced, and cybersecurity analysts are discovering that the investigations their AI assistant completed in seconds now require considerably more manual effort. At this stage, the organization isn't necessarily in crisis, but it's beginning to discover the difference between having AI available and being able to operate without it. Because those are not the same thing. And here's the interesting part. The business might not have considered these services critical when they were introduced. A helpful assistant, a productivity tool, a clever new feature. Nothing particularly alarming. But over time, people changed how they worked. Processes changed, staffing assumptions changed, and the dependency grew. Without anybody formally deciding that the service had become business critical. That's how risk quietly accumulates. Tuesday morning. The outage continues. Now the IT team is investigating why so many apparently unrelated applications are affected. Different vendors, different departments, different contracts, different applications. And yet, several depend on the same underlying AI model provider. Wonderful. We've diversified our software suppliers, but perhaps not our actual technology dependencies. This is something organizations really need to understand. You might purchase an AI capability from one software company, another from a different company, and a third from somebody else entirely, but those companies may rely on the same foundation model, cloud infrastructure, identity services, or other shared components. So what looks like diversification at the application level may not be diversification at all. We've seen similar problems with cloud infrastructure and other shared technology services. AI introduces another layer of concentration, and potentially another layer of complexity, because the business may not even know which underlying provider a particular application depends upon. Until that provider becomes unavailable. And suddenly several supposedly independent systems stop working together. That's not necessarily a failure of the individual vendors, it's a failure to understand the dependency, and you can't manage a dependency you don't know exists.

AI Vendor Dependencies and Recovery Challenges

SPEAKER_01 10:25

Now, somebody inevitably says, why don't we just switch to another AI provider? Excellent question. And in principle, that's a reasonable strategy, but it may not be quite as simple as changing your internet connection. Different models behave differently. They may have different capabilities, different interfaces, different context limits, different security controls, different data handling arrangements, different performance characteristics. And once you start talking about agentic AI, things become even more complicated. Agents may depend on particular tools, workflows, permissions, integrations, memory, orchestration, and application-specific configurations. Moving that to another provider isn't necessarily a case of changing a web address and carrying on. Some organizations will be able to design portable, provider-independent architectures, others won't. And the more deeply integrated the AI becomes, the harder migration may be. I suspect enterprise demand will eventually push providers toward better portability, stronger availability commitments, and more credible resilience options. Some organizations may also decide that critical AI capabilities belong within infrastructure they control more directly. But we shouldn't confuse what may become possible in the future with what we've actually implemented today. Having access to another AI provider isn't the same thing as having a tested recovery capability. It's rather like having a spare engine for your car. Very reassuring. Until you discover it's still in a box and nobody knows how to fit it. Wednesday. The AI service is still unavailable. And now we're discovering a second problem, the people. Over time, AI has taken over increasingly routine work, which is often a very good thing. Nobody needs to spend their entire career copying information between systems. But some organizations may also have reduced staffing, or stopped recruiting into certain roles, or allowed practical knowledge of manual processes to disappear. And that creates a problem. Because when the automation stops, the work doesn't necessarily stop needing to be done. Someone still needs to investigate the incident, process the transaction, help the customer, understand the system, and make a decision. But what happens if the people who used to do that work are no longer there? Or the people who remain haven't performed the process manually for years? We explored the talent pipeline problem in a previous episode. And it's directly relevant here. If AI removes the entry-level work through which people develop experience, where do tomorrow's experts come from? But today there's an even more immediate concern. What happens when the expertise required to operate without AI has already disappeared? You can have a perfectly documented manual fallback procedure, but if nobody has the skills, capacity, or experience to execute it, you don't really have a fallback. You have a document, and I've encountered plenty of those over the years, usually stored somewhere nobody can find when they actually need them.

Cybersecurity Risks and AI Reliability

SPEAKER_01 13:22

Let's talk specifically about cybersecurity, because this is where I think the consequences could become particularly serious. AI is increasingly capable of helping analysts investigate incidents. Gathering logs, correlating activity, reviewing endpoint telemetry, examining identity events, summarizing evidence, recommending containment. All valuable capabilities, and I expect organizations will become increasingly dependent on them. But here's something I've learned from working with experienced investigators. A great security analyst doesn't simply follow a checklist. Sometimes something just doesn't smell right. The evidence looks normal, the alert isn't particularly impressive, the dashboards are green. But the analyst notices something. A pattern, an inconsistency, a detail that doesn't fit, and decides to investigate further. That's professional judgment, experience, intuition. AI can analyse enormous amounts of information and identify patterns that humans might miss, but it can also be wrong. And I don't want us to lose the human ability to challenge its conclusions. Now imagine a security operations center that has reduced staffing because AI performs much of the routine investigation. Then the AI service goes down. Can the remaining analysts handle the workload? Do they still have access to the underlying telemetry? Can they investigate without the AI interface? Can they contain incidents manually? And have they actually practiced doing so? Because if the answer is no, we've created a serious resilience problem. The issue isn't that AI has made cybersecurity less effective. It may have made it substantially more effective. The issue is whether we've allowed that increased capability to become a single point of operational failure. Now here's a scenario that might be worse than a complete outage. What if the AI doesn't stop working? What if it simply starts working badly? Responses become unreliable, recommendations are incomplete, an automated workflow begins making incorrect decisions, or a model update changes the behavior of a system your organization depends upon. The service is technically available, the status page is green, the API is responding, but the business process isn't producing the outcomes it should. That's a very different kind of failure. Because when a service is completely unavailable, everybody notices. When it continues operating but produces bad results, the problem may be harder to detect. And if AI agents have authority to execute actions, those incorrect results can have real operational consequences. We don't need AI to become self-aware to have a serious problem. We simply need to give an automated system more authority than we can effectively monitor or control. That's why AI resilience can't be measured only by uptime. We also need to understand whether the service is operating correctly, whether outputs remain reliable, whether actions are appropriate, and whether humans can intervene when necessary. Availability matters, but so does trustworthy behavior.

AI Governance and Business Continuity

SPEAKER_01 16:23

So who owns this problem? The CIO? The CISO? Enterprise risk? The business? I think the answer is all of them, but with clearly defined responsibilities. Technology teams need to understand the architecture, the dependencies, the recovery options, the availability commitments. Security teams need to understand the risks, the permissions, the controls, the potential for misuse, and how to investigate failures. Enterprise risk and governance functions need to ensure the organization understands the exposure it's accepting. But business leaders have to be involved, because they're the ones who understand what happens when a process stops working. If an AI system supports customer transactions, that's not simply an IT service. It's part of a business capability, and the business owner needs to understand the consequences of losing it. I also think AI services should be included in business impact assessments, just like any other application or critical service. What does the AI do? What processes depend upon it? How long can we operate without it? What happens when it's unavailable? What alternatives exist? And who makes the decision to activate them? These are familiar questions. We've been asking them about enterprise technology for decades. The mistake would be assuming AI is somehow exempt, because it's new, because it's exciting, or because the vendor has a very attractive presentation. So what should organizations actually do? First, understand where AI is being used. Not simply which employees have access to an AI assistant, which business processes depend on AI capabilities, including AI embedded within other applications. Second, understand the criticality. An assistant that helps draft emails is not the same as an agent involved in financial transactions or security incident response. They shouldn't have identical recovery requirements. Third, identify the underlying dependencies. Which providers, which models, which cloud services, which integrations, and what happens if one of those components becomes unavailable. Fourth, define realistic fallback arrangements. For some processes, that may mean manual operation. For others, an alternative provider, a reduced service mode, a cued workflow, or simply accepting a defined period of disruption. Not every AI capability needs five nines of availability, and not every business can afford it. The important thing is making that decision deliberately. Fifth, test the arrangements. Actually test them. Not just read the procedure and decide it looks reasonable. Disconnect the AI service in a controlled exercise. See what happens. Find out whether people can continue operating. Because I can tell you from experience that disaster recovery plans often look excellent right up until you need them. And finally, revisit everything frequently. AI is evolving extraordinarily quickly. What your organization depends upon today may be completely different six months from now. A business impact assessment completed last year may already be outdated. Resilience can't be a one-time exercise.

SPEAKER_00 19:31

It has to evolve alongside the technology.

The Multiplier: Switch Off Your AI

SPEAKER_00 19:38

So here's this episode's multiplier.

SPEAKER_01 19:40

I want you to conduct a simple exercise. You don't actually need to disconnect anything, at least not without telling the people responsible. That would make for a rather interesting Monday morning. Pick one business process that uses AI. Ideally, something important. Now imagine the AI service becomes completely unavailable for 24 hours. What happens? Who notices? What work stops? What work continues? Now extend that to seven days. Can the business still operate? At what capacity? What backlogs develop? What commitments can no longer be met? Next, look at your dependencies. Do you know which AI providers and underlying services support the process? Are there hidden common dependencies? Then examine your fallback. Is it documented? Is it practical? Do you have the people and skills required to execute it? And most importantly, when did you last test it? Because having a recovery plan isn't the same thing as having a recovery capability. I learned that lesson during Hurricane Katrina. The equipment was protected, the generator worked, the backups were secure, but the assumptions around them failed, and that was enough. So don't simply ask whether your AI provider is resilient. Ask whether your business is resilient when that provider isn't.

SPEAKER_00 20:56

That's the multiplier.

Closing: Building AI Resilience

SPEAKER_00 21:02

Eventually, our fictional AI provider restores service.

SPEAKER_01 21:06

The assistants start responding. Customer service returns to normal. The developers are happy, finance catches up, and the cybersecurity dashboards are reassuringly green again. Everybody breathes a sigh of relief. Someone writes an incident report, and, if we're lucky, the organization learns something. Because I don't think the question is whether AI will become critical to business operations. I think that's increasingly inevitable. The real question is whether we're building organizations that can survive when AI isn't there. And I want to be clear, I'm not arguing against AI adoption. Quite the opposite. AI offers enormous opportunities to improve productivity, strengthen cybersecurity, automate processes, and create capabilities we couldn't realistically deliver before. But the faster we adopt it, the more deliberately we need to manage the dependencies we're creating. We need to understand the technology, the people, the providers, the risks, and the recovery options. Because resilience isn't about preventing every failure. It's about making sure a failure doesn't become a disaster. And if there's one thing my experience during Hurricane Katrina taught me, it's that the assumptions you never thought to question are often the ones that matter most. I'm Steve. This is AI Security Multipliers. And remember, AI should multiply your capability, not your mistakes. I'll see you next time.