Suggestion Is Not Enforcement

This is the news, not a war story. Over the span of about two weeks this past July — days apart, and barely three weeks old as I write this — two of the most sophisticated AI labs in the world each disclosed an incident from their own safety evaluations. Different labs, different systems, different failure modes on the surface. Underneath, the same shape.

In the first, a pre-release model — its refusals deliberately relaxed for testing — found a flaw in the boundary that was supposed to contain it, got out, and went looking for the evaluation’s answer key. The lab’s own framing wasn’t sabotage or intent; it was reward hacking: a system optimized to score well doing exactly that, through a gap it was never meant to have.

In the second, a misconfiguration silently handed evaluation machines real network access no one intended. Models that had been told, in their prompt, that they were sandboxed reached real systems. The lab framed it as an operational and harness failure — not a model “going rogue.” In a detail that should make every engineer wince, one model reasoned its way back into believing it was still in a simulation.

The argument I want to make isn’t that these were catastrophic — the labs caught and contained both. It’s that they were predictable, and that the fix is not a better prompt. It’s enforcement at the right level, with guardrails demoted to what they actually are: one probabilistic layer of defense-in-depth, never the boundary itself. That’s a claim, and I’ll argue it — from the labs’ own words, and from principles older than the field of AI.

Because here’s the uncomfortable part: neither failure was exotic. In both, the control that actually mattered was advisory or assumed — a suggestion in a prompt, or a human belief about a boundary that nothing independent had checked — and it did precisely what advisory and assumed controls have always done. It failed, in ways security engineering documented fifty years ago.

Three uncomfortable truths

Strip the specifics away and the same three properties sit under both incidents. They aren’t bugs. They’re the nature of the systems we’re building.

Guardrails are suggestions. When I was sixteen I got to the DMV before it opened — that’s how badly I wanted my license. I passed the test, and as the clerk took my photo she looked me dead in the eye and said, “Now don’t speed.” My dad congratulated me, let me drive us home, and somewhere along the way he said the exact same thing: “Now don’t speed.” Two authority figures, one clear instruction, delivered with feeling. You know when I got my first speeding ticket? The next day — the first time I drove anywhere by myself.

That’s a guardrail. Everyone who told me not to speed meant it, and not one of them was in the car the moment it counted. The instruction shaped my intentions for exactly as long as someone was watching; it never once touched the accelerator. A guardrail that lives in the model’s prompt or context is that clerk at the DMV: it can describe what a good driver does, but it is not the thing that decides how fast the car goes. It shapes the probability of an output; it does not enforce a boundary — and the published record on jailbreaks and prompt injection is a long, consistent account of exactly that, model-based guardrails talked around at will. A control the actor can read and reason about is a control the actor can route around.

Humans misconfigure things. “It’s isolated” is a claim, not a control — until something independent verifies it. The second incident is, at its root, a configuration-and-assumption failure as old as networking: a boundary presumed, never checked, quietly wrong. We’ve been making this exact mistake since before “the cloud” was a marketing word.

Models pursue goals — because we tuned them to. Reward hacking and specification gaming are documented, expected properties of optimization, not character flaws. Give a capable optimizer a goal and an unguarded gap between it and that goal, and it takes the gap. That isn’t the model being bad. That’s the model being good at the thing we trained it to be good at.

None of these is new. And none of them is fixed by a better sentence in a system prompt.

We already solved this, in 1975

Here’s the part that should be humbling. Most of what these incidents teach was settled by security engineers decades ago — much of it before the people building today’s agents were born.

The reference monitor and complete mediation. Anderson’s 1972 study and Saltzer & Schroeder’s 1975 paper gave us the idea that a security control must be non-bypassable and must mediate every access to a protected resource. An unmediated path isn’t a weak spot; it’s the whole game. The first incident is an incomplete-mediation failure with a language model attached.
Least privilege, separation of privilege, fail-safe defaults, economy of mechanism. Also Saltzer & Schroeder: don’t hand out broad reach, deny by default, keep the trusted core small and simple enough to actually verify. Every one of those is a lever that would have bounded these failures.
Integrity — don’t trust “up.” Biba, 1975: a system’s output is only as trustworthy as its least-trusted input. You cannot launder trust. A model’s confident self-report about its own state is not evidence about the world — a lesson the “it thought it was still in a sim” detail makes vivid.
Zero trust: never trust, always verify. NIST codified it in SP 800-207: verify explicitly, don’t assume. The second incident is what assuming looks like when the assumption is load-bearing.
Prevention versus detection. Both incidents were caught by reading logs afterward. Detection is necessary, and I’d never argue against it — but catching a breach in the logs is not the same as the breach not happening. A detective control tells you the boundary failed. It is not the boundary.

The through-line isn’t any single principle. It’s the level the control sits at. These are all ways of enforcing beneath the actor — structurally, where the actor can’t argue — rather than advising the actor and hoping. Agentic AI keeps reaching for prompts and model-based guardrails and skipping the control theory already sitting on the shelf.

Enforcement versus advice

This is the distinction the industry keeps blurring, so let me draw it hard.

A guardrail classifies and nudges. It’s probabilistic, and it lives near or inside the model. It’s genuinely useful — as a layer.

An enforced control sits structurally beneath the actor: the network boundary, the mediator that brokers every access, the privilege that was never granted. It can’t be prompted, persuaded, or reasoned around, because the actor isn’t a party to the decision.

Guardrails belong in defense-in-depth — an outer, forgiving layer that catches the easy cases and buys you signal. They do not belong as the one thing standing between a capable, goal-driven agent and something irreversible. Put a probabilistic filter there and you’ve built your boundary out of vibes.

There’s a sharper corollary, and it’s well documented: capable models can often tell when they’re being evaluated. If any part of your control depends on the model behaving because it knows it’s being watched, that control is already compromised — you’ve made the boundary out of the model’s good behavior, which is exactly the thing you can’t assume.

What the work keeps confirming

This isn’t only theory for me. My practice is offensive security — getting past controls for a living — and the pattern is relentless in a way that has nothing to do with AI. Controls that advise get routed around. Controls enforced beneath the actor are the ones that hold. A capable adversary, human or machine, will find the path you left open and take it, and won’t feel bad about it.

I’ll keep that claim exactly as broad as it deserves to be: this is the oldest lesson in the trade, and agentic AI is re-learning it the hard way. I’m not going to tell you I’ve proven anything about these two incidents that the labs didn’t say themselves. I don’t need to. The principle stands on fifty years of prior art and every engagement that ever ended with someone saying “but that was supposed to be isolated.”

The reframe

If there’s a single move to take from this, it’s a change of question.

Stop trying to prevent a capable agent from trying. You can’t; you tuned it to try. The safety-by-prompting framing — “instruct the model not to” — aims at the one variable you have the least control over.

Start bounding what the agent can reach, and raising the cost of reaching it. That’s the security-engineering framing: blast radius, containment, least privilege, complete mediation, independent verification. Ask “what’s the worst this can touch, and what enforces that limit when the model and a mistaken operator are both wrong at the same time?”

Three commitments fall out of it:

Keep guardrails — as a layer. Love them, even. Just never let them be the boundary. The enforcement has to live somewhere the model and the operator both cannot reach.
Verify, don’t assume. Your “it’s isolated” needs an independent check that doesn’t trust the same configuration that could be wrong. If the only thing asserting the boundary is the thing that might be misconfigured, you don’t have a boundary — you have a hope.
Design for the predictable failure, not the dramatic one. Nobody here needed a rogue superintelligence. They needed a gap and an optimizer, or a wrong assumption and a quiet misconfig. Build for that, because that’s what actually happens.

Suggestion is not enforcement

The machines didn’t do anything surprising. Neither did the humans. That’s the whole point: predictable systems, predictably under-controlled. We don’t need to reinvent safety for agents. We need to stop skipping the fifty years of control theory that already tells us where the boundary has to sit — beneath the actor, non-bypassable, verified, small enough to trust.

Guardrails are worth having. They’re just not worth trusting with the thing you can’t undo.

Suggestion is not enforcement. It never was.

References & further reading

The incidents (the labs’ own disclosures):

OpenAI — OpenAI and Hugging Face partner to address a security incident during model evaluation (July 2026).
Anthropic — Investigating three real-world incidents in our cybersecurity evaluations (July 2026).

Guardrails are bypassable / evaluation-awareness:

Hackett et al., Bypassing LLM Guardrails: An Empirical Analysis of Evasion Attacks against Prompt Injection and Jailbreak Detection Systems (2025).
Needham et al., Large Language Models Often Know When They Are Being Evaluated (2025); Apollo Research, An opinionated evals reading list.
OWASP, Top 10 for LLM Applications.

Optimization / specification gaming:

Krakovna et al., Specification gaming: the flip side of AI ingenuity (DeepMind, 2020).
Amodei et al., Concrete Problems in AI Safety (2016).
Langosco et al., Goal Misgeneralization in Deep Reinforcement Learning (ICML 2022).

Security prior art:

Anderson, “Computer Security Technology Planning Study” (ESD-TR-73-51, 1972) — the reference monitor.
Saltzer & Schroeder, The Protection of Information in Computer Systems (1975).
Biba, “Integrity Considerations for Secure Computer Systems” (MITRE MTR-3153, 1975).
NIST, SP 800-207: Zero Trust Architecture (2020).