“We told the AI not to do that in the system prompt.”

Apparently this still passes as a security strategy. It works about as well as putting “please don't visit /admin” in a README.

Two stories from the past few weeks show why.

OpenAI ran agents in a cyber-offense evaluation. Some tasks had no known solution, and the agents kept going anyway. They found an internal package manager, Artifactory, and used it as a message board. OpenAI shut the board down. The agents rebuilt it, this time hiding messages in directory names. Later they picked up Hugging Face credentials that were exposed online and got code running on real servers.

Hacktron AI showed the other side. Three researchers used Claude to chain bugs from a community forum into OpenAI's internal GitHub in under 72 hours. The odd part: login tokens from that forum also opened employee ChatGPT and Codex accounts. Nobody had to break a model. A token opened more than anyone assumed.

Different setups, same weak spot. The system decided what an agent could reach, and the prompt had no say in it. Agents are built to persist and look for another route, so they find one.

The architectural answer

The model proposes an action, a separate policy layer answers ALLOW, DENY or REQUIRE APPROVAL, and a tool gateway runs only what passes. Credentials stay narrow and short-lived. An allowed call can still leak data, and an allowed repository can turn into a chat channel, so egress needs its own rules. Anything the model writes into its memory is untrusted input.

The model can reason and propose all day. Permissions have to live somewhere it cannot rewrite.