AI Agents Change the Security Model

Episode transcript

Transcript provided for accessibility and search. Minor transcription errors may remain.

AI Agents Change the Security Model

SPEAKER_00 0:02

For most of the history of Enterprise IT, we've spent an extraordinary amount of time deciding what humans are allowed to do, which is impressive because we're still not. Let's start with terminology. Because as usual, the technology industry has managed to create several slightly different definitions for something while simultaneously using the same word for all of them. An AI assistant generally helps a person. You ask a question, it gives you an answer. You ask it to draft an email, it drafts the email. You ask it to summarize a meeting, it summarizes the meeting. The human remains very obviously in control of what happens next. An AI agent goes further, an agent can be given a goal and potentially work through multiple steps to achieve it. It may retrieve information, reason about what it finds, choose an action, use a tool, evaluate the result, and decide what to do next. That doesn't necessarily mean some science fiction autonomous system wandering freely around your network. It could be quite simple. Imagine an IT support agent. An employee says, I can't access the finance

From Copilot to Agent

SPEAKER_00 1:17

application. The agent checks the user's account, looks at recent authentication activity, checks whether they're entitled to the application, searches the knowledge base, maybe identifies the problem, then creates or updates the service ticket. That's useful. Now allow it to reset the account. More useful. Allow it to modify a group membership. Even more useful. Allow it to approve the access request itself. Now we've changed something important. We haven't simply given the AI more intelligence. We've given it more authority. I think there are three useful ways to think about AI in the enterprise. AI advises. AI assists. AI acts. If an AI system tells me this user account looks suspicious, I recommend disabling it. The AI is advising me. I still make the decision. If the AI prepares the action and says I have identified the account and prepared the disable operation, would you like me to proceed? Now it's assisting me, but I still approve the action. If the AI identifies the account, decides the activity is suspicious, and disables it automatically. The AI is acting. The intelligence may be very similar in all three scenarios, but the risk isn't. Because in the third example, we've delegated authority to software. That's why I think autonomy is sometimes described in the wrong way. We talk about it as if it's a feature. How autonomous is the agent? But from a security and governance perspective, I think the better question is: how

The Moment the Security Model Changes

SPEAKER_00 2:59

much authority are we prepared to delegate to software? That framing changes the conversation. Because organizations already understand delegated authority. We delegate authority to employees, managers, administrators, applications, service accounts, third parties. And every time we do, we put boundaries around it. At least we're supposed to. AI agents don't make that principle obsolete. They make it more important. In episode three, we talked about identity effectively becoming part of the AI query. Your identity helps determine what information an enterprise AI system can retrieve. With agents, identity becomes even more important. Because now identity doesn't just determine what the AI can know, it may determine what the AI can do. Every agent that interacts with enterprise systems needs some way of being recognized. It may operate using the identity of the person invoking it. It may have its own application identity, it may use a service principal, a service account, a workload identity, or some combination depending on what it's doing. Those distinctions matter. Because when the agent calls an API or changes something in another system, that system needs to answer a very basic question. Who is asking? And then another one. Are they allowed to do this? This is authentication and authorization. Very familiar security concepts. But AI agents make them interesting again because we're creating potentially enormous

Every Agent Needs an Identity

SPEAKER_00 4:34

numbers of non-human actors. And those actors may be operating across multiple applications. So before we get carried away discussing how intelligent the agent is, I'd ask something much more mundane. What identity does it use? If nobody can answer that clearly, I'm probably not ready for the demo yet. Let's say an agent is helping me perform a task, whose permissions should it use? Mine, its own, some combination? There are legitimate architectures for each approach, but the security consequences are different. If the agent acts entirely as me, then ideally it should be constrained by my permissions. That's intuitive. If I can't access the payroll system, my AI agent shouldn't suddenly develop a keen interest in salaries. But what if the agent needs capabilities I don't normally have? Perhaps it's an IT support agent. It may need to perform specific administrative actions on behalf of employees. Now we may give the agent its own permissions, that's where things become more interesting, because the agent might have authority that the person interacting with it doesn't have. So we need to make sure the user can't manipulate the agent into exercising that authority inappropriately. This is not a completely new security

Whose Permissions Is It Using?

SPEAKER_00 6:00

problem. Applications have been acting on behalf of users for decades. But AI changes the interface. Instead of selecting from predefined buttons and workflows, the user can communicate with the application in natural language, and the system itself may decide which tool to invoke. That flexibility is what makes agents powerful. It's also what makes authorization design so important. Natural language should not become a shortcut around least privilege. This is where I'd separate capability from authority. An AI model may be capable of deciding that a purchase order should be approved. That doesn't mean it should have the authority to approve one. It may be capable of determining that an employee should receive access to a system. That doesn't mean it should grant the access. It may be capable of writing a payment instruction. Please don't immediately give it the bank account. Capability is a technical question. Authority is a governance question. And organizations need to be very deliberate about the difference. One useful approach is to think about the consequence of the action. Low consequence actions may be appropriate for greater autonomy. Creating a draft response, categorizing a ticket, scheduling a meeting, updating a low risk record. As consequences increase, perhaps we require approval. Changing permissions, making financial commitments, deleting data, modifying production systems, communicating externally. The question isn't whether the AI can technically do these things. It probably can.

Delegated Authority

SPEAKER_00 7:36

The question is whether we've deliberately decided that it should. That's why I think one of the most important controls for a gentic AI isn't particularly exotic. It's a clear definition of delegated authority. What can the agent decide? What can it recommend? What can it execute? And when does a human have to become involved? Enterprise IT already has a long and distinguished history of service accounts. You know the ones. Created for something terribly important in 2014, owned by somebody who left in 2018, running a process nobody fully understands. With a password, nobody wants to change because we're not entirely sure what will stop working. So naturally, we're now going to add intelligent, autonomous software to the mix. What could possibly go wrong? The lesson from service accounts is actually very relevant. Non-human identities need ownership. They need lifecycle management. They need appropriate permissions, they need credentials that are protected and rotated. They need monitoring, and eventually they need to be retired. Agents are no different. If anything, they're more important because the number of machine identities could grow very quickly. Imagine every department creating agents, finance agents, HR agents, sales agents, procurement agents, security agents, personal productivity agents,

The Service Account Problem, Again

SPEAKER_00 8:59

then agents creating or invoking other agents. Suddenly identity governance isn't just about employees and contractors, it's about a growing workforce of digital actors. If we're not careful, we could recreate 20 years of service account debt in about 18 months. Technology does seem to be getting faster. Agents become useful when we give them tools. A tool might allow the agent to search SharePoint. Query Salesforce. Create a ServiceNow ticket. Send an email. Run a database query. Call a cloud API. Modify a configuration. The more tools we add, the more useful the agent becomes. But there's an important principle here. Every time you give the agent another tool, you're not just adding a feature, you're adding authority. A chatbot with no tools can say something wrong. That's obviously a problem. But an agent with access to enterprise systems can potentially do something wrong. That's a different level of consequence. So tool access needs the same scrutiny would apply to permissions for a human administrator or application. Does the agent need read access, write access, delete access, administrative access? Can we constrain which operations it can perform? Can we limit which records it can modify? Can we restrict transactions above a certain value? Can we require approval

Every Tool Adds Authority

SPEAKER_00 10:24

for higher risk actions? And critically, can we revoke the tool? Because the easiest way to stop an agent from doing something may simply be to remove its ability to do it. That sounds obvious. Security occasionally benefits from obvious. We also need to accept something fairly basic. Agents will make mistakes. Humans do. Software does. AI certainly does. The question isn't whether an agent will ever make a bad decision. The question is what happens when it does. Suppose an agent incorrectly closes a support ticket. Annoying. Suppose it sends the wrong internal email. More annoying. Suppose it removes access from the wrong employee. Potentially disruptive. Suppose it modifies a production environment. Now the meeting invitations start appearing. The consequence matters. This is why autonomy should be proportional to risk. For some actions we may be perfectly comfortable allowing the agent to operate automatically. For others we may want confirmation. For higher risk actions we may require formal approval or prohibit autonomous execution entirely. And we need reversibility. Can we undo what the agent did? Can we restore the previous state? Can we identify every action it took? Can we

What Happens When the Agent Gets It Wrong?

SPEAKER_00 11:44

stop it while it's running? One of the least glamorous but most important questions in agent security is where's the off switch? Not philosophically, literally. How do we stop this thing? Prompt injection is already a significant AI security concern. At a simple level, the AI encounters instructions that attempt to change its behavior. Sometimes those instructions come directly from the user. Sometimes they're hidden inside content the AI is processing. A document, a website, an email, a support ticket. Now think about the difference between an AI system that can only generate text and an agent that can take actions. If I manipulate a chatbot into producing a strange answer, that's a problem. If I manipulate an agent into sending information somewhere, changing a record or invoking a privileged tool, the consequence can be considerably greater. This is sometimes called indirect prompt injection. The malicious instruction may be sitting inside data the agent has been asked to process. Imagine an agent reading an external document that contains instructions intended for the AI rather than the human reader. The agent has to distinguish between this is information I should analyze, and this

Prompt Injection Becomes More Serious

SPEAKER_00 13:01

is an instruction I should follow. Humans struggle with suspicious emails. We've now created software that reads all of them. Progress. This is why agent security can't rely entirely on telling the model to behave itself. We need controls around the model, strong authorization, tool restrictions, data boundaries, validation, monitoring, approval for consequential actions. The model is part of the security architecture. It shouldn't be the entire security architecture. One of the most common responses to agent risk is we'll keep a human in the loop. Good. I'm generally in favor of humans, but we need to be slightly more specific. Which human and what exactly are they approving? If the agent produces a 200-page analysis and then asks somebody, approve, we haven't necessarily created meaningful oversight. We've created a button, and humans are remarkably good at clicking buttons, especially when the button appears 17 times before lunch. Approval fatigue is real, so human in the loop controls need to be designed carefully. The person approving the action needs enough context. They need appropriate authority, they need to understand the consequence, and the approval should occur at the point where it actually matters. We also shouldn't require human approval for every trivial action. If every agent interaction generates an approval request, people will either stop using the agent or start approving everything

Human in the Loop, But Which Human?

SPEAKER_00 14:39

automatically. Neither is particularly useful, so again, risk should determine the control. Low risk action, automate it. Moderate consequence, perhaps confirmation. High consequence require explicit authorization. Very high consequence? Maybe the agent shouldn't be able to perform it at all. Human oversight is valuable, but only if the human is actually overseeing something. If an employee makes a significant change in an enterprise system, we generally want to know who did it. Agents shouldn't be different. In fact, we probably need more context. Who invoked the agent? Which agent was involved? What identity did it use? What information did it retrieve? What decision did it make? Which tool did it call? What action occurred? Was human approval involved? What happened afterwards? That's a lot more interesting than a log entry saying API call successful. From a security perspective, agent observability is going to matter enormously. Because when something goes wrong, somebody will ask, why did the AI do that? And I'd strongly prefer the answer not to be we're not entirely sure. That's rarely a satisfying sentence in front of an audit committee. Logging also helps with normal operations. We can see whether agents are behaving as expected, whether permissions are being used appropriately, whether

Logging and the Audit Trail

SPEAKER_00 16:11

particular actions are happening more frequently than anticipated, whether users are attempting unusual things, and whether the agent is repeatedly getting something wrong. Agents need accountability, and accountability requires evidence. If an autonomous system can make decisions, but we can't reconstruct those decisions afterwards, we've created a governance problem, possibly a very efficient one. There's another old control that becomes surprisingly relevant. Separation of duties. We use separation of duties because we generally don't want one person controlling an entire high-risk process. The person requesting a payment may not be the person approving it. The person requesting privileged access may not be the person granting it. The developer may not be able to push whatever they like directly into production. Well, that's the theory. Agents shouldn't automatically be exempt from those principles. Imagine an AI agent that can create a vendor, approve the vendor, create a payment, and authorize the payment. That's not efficiency. That's an internal audit finding with excellent branding. So when agents participate in business processes, we need to understand the role they're performing. Is the agent the requester, the analyst, the approver, the executor? Can the same agent perform conflicting roles? Can two agents collude unintentionally because they're using the same underlying authority? Do

Separation of Duties for Machines

SPEAKER_00 17:43

human separation of duty controls still apply when one of the participants isn't human? These are exactly the sorts of questions governance teams should start asking now. AI may change who performs the work. It doesn't automatically change why the control existed. There's a temptation when building agents to give them broad permissions because it's easier. The developer wants the agent to perform ten different tasks, so we give it access to everything it might need. The demo works. Everyone is delighted. Then the prototype becomes production, which in enterprise technology can happen with remarkable speed when somebody important likes the demo. This is where least privilege matters. An agent should have the minimum authority necessary to perform its intended function. And ideally, that authority should be contextual. Perhaps it can perform an action only for the user who invoked it, only within a specific system, only for a certain transaction type, only below a particular threshold, only during a defined workflow. And perhaps some permissions should exist only when needed. Just in time access is particularly interesting here. Instead of giving the agent standing privilege forever, allow it to acquire specific authority for a specific task under defined conditions.

Don't Give Every Agent the Keys to the Kingdom

SPEAKER_00 19:05

That's much healthier than creating another highly privileged identity and hoping everyone remembers why it exists five years from now. Because we already know how that story ends. Usually with somebody in cybersecurity asking, does anyone know what this account does? And an uncomfortable silence. So what should you actually do? Start by finding the agents, not just the big enterprise AI platform. Look at what's being built in departments, automation platforms, development teams, customer service, security, IT operations, low-code environments, vendor products. You may already have more agentic behavior than you realize. Then inventory authority. For each agent ask, what identity does it use? What data can it access? What tools can it invoke? What actions can it perform? Which systems can it modify? Where does it require human approval? Who owns it? How is it monitored? And how do we disable it? And classify the actions. Not every capability carries the same risk. Reading a knowledge article is different from changing payroll information. Drafting an email is different from sending one. Recommending access is different from granting it. Creating a purchase request is different from approving payment. That distinction gives you somewhere practical to start. Then test the boundaries.

What Do You Do on Monday?

SPEAKER_00 20:29

Don't only test whether the agent performs the intended workflow. Test what happens when information is incomplete, when instructions conflict, when a user asks it to do something outside their authority, when external content contains unexpected instructions, and when the agent simply gets something wrong. The goal isn't to prove the agent works, the demo already did that. The goal is to understand how it fails. That's usually where security gets interesting. And now this week's multiplier. Pick one AI agent your organization is using, building, or seriously considering. Then ask five questions. What identity does it use? What data can it access? What actions can it take? Who can authorize those actions? And finally, how do we stop it? Not how do we ask it politely to stop. How do we actually remove its ability to act? Disable the identity, revoke the token, remove the tool, terminate the workflow. Whatever the mechanism is. If you can answer all five clearly, you've got the beginnings of a security model. If you can't, the agent may already have more autonomy than your governance does, and that's probably worth fixing before we give it another tool. That's this week's multiplier. AI agents represent a genuinely important change in enterprise technology. Not because they're magically intelligent, and not because every agent is going to become autonomous. The important change is that AI is moving closer to action, and action requires authority. That's the security model I think we need

The Multiplier

SPEAKER_00 22:06

to focus on. In episode three we talked about identity helping determine what AI can know. With agents, identity and authorization increasingly determine what AI can do. And the moment AI can do something, familiar security principles become extremely relevant. Least privilege, strong identity, separation of duties, approval, monitoring, auditability, life cycle management, and the ability to revoke access. These aren't new ideas. And that's actually encouraging. We don't need to invent cybersecurity from scratch because somebody added the word agent to the architecture diagram. We need to apply what we've learned to a new kind of actor. The opportunity is enormous. Agents

Closing: Action Requires Authority

SPEAKER_00 22:52

can remove repetitive work, accelerate processes, help employees, improve security operations, connect systems, and potentially make organizations much more productive. But every time we increase autonomy, we should understand the authority we're delegating. Because an AI system that can answer a question is useful. An AI system that can take action is powerful. And power needs boundaries. That's it for this episode of AI Security Multipliers. If you enjoyed the episode or found it useful, please follow or subscribe to the show. And if you know somebody who's building AI agents, send this episode their way. Preferably before they give the agent administrative access. I'd also really like to hear from you if there's something you disagree with, a topic you think we should explore, or an AI or cybersecurity problem you're dealing with in your own organization, send it to me. And if you've already built an autonomous agent that can modify production, approve payments, and create its own credentials, I'd definitely like to hear from you. Although perhaps after you've found the off switch. I'm Steve, and I'll see you next time.