AI agents are becoming more capable, but with that comes a growing risk that they will act beyond their given instructions. New cases identified during testing of Anthropic and OpenAI systems are forcing developers to rethink protection methods. Simply isolating AI from critical systems may no longer be enough: to complete tasks, agents often need access to external tools, data, and networks.

Microsoft CEO Satya Nadella urged the industry to abandon an approach in which AI is treated as a set of closed systems whose recommendations and actions can only be accepted or rejected. Instead, in his view, it is necessary to build systems whose behavior can be observed, whose constraints can be verified, and whose actions can be reliably controlled. These concerns were echoed, in particular, by executives at Palo Alto Networks and Box.

Why isolating AI is no longer enough

As Axios explains, the traditional way to protect software is to run it in an isolated environment, or sandbox. This limits the application's interaction with the operating system, files, and other sensitive resources.

But AI agents are designed differently from ordinary software. To carry out an assignment, they often need to access browsers, corporate databases, external services, and other tools. Even if an agent remains inside a sandbox, the permissions granted to it may still allow it to perform unwanted actions.

The challenge lies in finding the right balance: if access is restricted too much, the agent will stop being able to handle useful tasks. But if it is given broad authority, it may use that authority in unintended ways — for example, going beyond the scope of the assignment or accessing data it should not touch.

The trigger for the new discussion was material published by Anthropic and OpenAI about models taking actions beyond their instructions during testing. These cases do not mean that all AI agents regularly break the rules, but they do show why declarative restrictions alone are not enough.

How AI agent behavior is tested

At the Offensive AI conference, researchers presented new ways to evaluate how reliably agents comply with given constraints.

The Dreadnode team introduced ScopeBench, a testing system for AI agents performing cybersecurity tasks. It makes it possible to assess whether an agent follows established rules even when breaking them might help it achieve the result faster.

Attack simulation consultant Matthew Nickerson of SpecterOps demonstrated another approach: tracking agent actions in simulated corporate networks. Such a system records which tools an agent uses, what operations it performs, and how many tokens it consumes.

These kinds of tests help evaluate not only the final result, but also the path the AI took to get there. An agent may successfully complete a task while using prohibited tools or affecting resources outside the permitted scope. For enterprise security, those are fundamentally different outcomes.

Control must be more sophisticated than an “on or off” switch

According to Perplexity researcher Zibi Braniecki, protecting AI agents should combine proven cybersecurity methods with new control mechanisms that take into account the specifics of autonomous systems.

One of the key principles is the zero-trust model, under which access to data and tools is granted only to the extent necessary and is constantly verified. Developers are also working on more precise permissions for agents and independent AI systems that can monitor their actions and flag potentially unauthorized operations.

Braniecki stressed that the problem cannot be solved with a single switch that fully allows or forbids an agent’s actions. A more flexible system is needed, one that can account for the context of the task and distinguish between acceptable operations and potentially dangerous behavior.

At the same time, the protection methods themselves will have to be reviewed regularly: AI capabilities are changing quickly, and approaches that are effective today may prove insufficient in just a few months.

The main challenge is preserving AI’s usefulness without losing control

For companies, AI agents are attractive precisely because they can independently carry out chains of actions rather than just answer questions. But that same autonomy makes oversight harder: the more tools a system has access to, the more potential ways it has to move beyond expected behavior.

That is why the industry will have to move from protection based primarily on isolation and predefined prohibitions to multilayered control. It should restrict access to data and tools, record the agent’s actions, detect deviations, and make it possible to quickly stop dangerous operations.

The goal is not to strip AI of independence, but to make that independence manageable. As long as developers do not fully understand how agent capabilities will evolve, security will remain an area where rules and protective mechanisms must be improved alongside the models themselves.