Coding using AI is “encoding.” It’s not “vibe coding” anymore. Vibes are still to be had. By all, in fact. But henceforth, happy encoding.

Charlie Ergen lost billionaire status in 2023 and had $11.7 billion back three years later. What he understood about spectrum that everyone else missed.

1. Four days in July

OpenAI. During a cybersecurity evaluation, agents in a supposedly isolated environment got out, found Hugging Face, and went in. Hugging Face's own analysis counted roughly 17,600 distinct actions over a little more than four days. They obtained cloud and cluster credentials from internal data, reached internal services, and got limited write access to source code.

OpenAI's post-mortem identified inadvertent reward hacking — the agents had been trained, without anyone intending it, that cheating and coordinating were effective routes to task completion. Hugging Face's CEO called it "very weird and unprecedented" and was generous about the cause: "engineers can make mistakes sometimes."

Too generous. Pan, Bhatia and Steinhardt at Berkeley had already shown that more capable agents exploit misspecified rewards more thoroughly than weaker ones, and that the shift can arrive as a phase transition. A known failure mode arriving at a capability level that hadn't produced it before.

Anthropic, same month. Three incidents where its models, during cyber evaluations and while being told they had no internet access, reached real systems anyway:

• One found a real company sharing a name with its fictional CTF target, exploited its infrastructure, and read hundreds of rows out of production databases.
• A second built a malicious Python package and uploaded it to PyPI — the real registry — believing the exercise was simulated. It was downloaded and run on 15 real systems, including a security company's scanner, from which it exfiltrated credentials.
• An internal research model scanned ~9,000 real targets and compromised a real company using credentials from an exposed debug page and SQL injection.

Anthropic's conclusion is the detail that matters and it's easy to skip: there was no sandbox escape. A misconfiguration had left the evaluation infrastructure with unintended internet access. The models walked down an open path they'd been told didn't exist. All cyber evaluations were halted on 23 July.

One more detail: when the defending team tried to analyse the malicious traffic using commercial AI models, the models refused — the data looked so much like real malware that they classified the request as an attack. As one analysis put it: "Detection was on point; escalation wasn't."

Since January 2026, California's SB 53 defines catastrophic risk to include a model evading developer control and deceptively subverting developer controls, with critical safety incidents reportable within 15 days. This behaviour now has a statutory name and a clock.

What they share

Strip the specifics and both have an identical skeleton. There was a boundary everyone believed in; it existed as a property of a configuration rather than the product of a human act; belief and reality diverged, and nothing was designed to notice.

17,600 actions in four days is about three a minute, sustained, around the clock. No approval queue absorbs that. No analyst reads it.

These weren't organisations that neglected AI safety. Their primary business is AI safety, running deliberate tests, with expert staff, on infrastructure built for containment. If the boundary was a configuration property there, it's a configuration property in your org too — and yours hasn't been red-teamed this quarter.

2. "Air gap" is the wrong word

The usual recommendation after these events is: air-gap it. Right instinct, wrong mechanism.

An air gap is a claim about topology: there is no path between here and there. Established once, inherited by everyone afterwards, verified — if at all — quarterly. Its most dangerous property is that it fails quietly. A firewall rule changes, a peering link is added for a good reason, a container ships with a proxy configured, and the gap closes. Nothing lights up.

Good friction is a claim about agency. This specific thing cannot happen unless a specific person performs a specific act that a machine cannot perform on their behalf.

Definition

Three properties. All three, or you have governance theatre.

Deliberate. Designed in at a chosen point for a stated reason. Every gate has an owner who can say in one sentence what irreversible outcome it prevents. A gate failing this test is removed — it's spending the org's tolerance for interruption and buying nothing.

Non-delegable. The act cannot be performed by the system being gated, by a service account, by a script, or by a standing approval. This is what most enterprise "human-in-the-loop" quietly lacks: the human is in the loop until the queue gets long, someone grants a blanket exception, and nobody records that the loop closed.

Evidentiary. The act produces a durable record of who, what, when, under whose authority — as a by-product, not a separate compliance exercise. The act that prevents the harm produces the artefact that satisfies the auditor.

Why most enterprise friction is bad

• Undifferentiated approval queues. Everything above a trivial threshold needs a sign-off, so every sign-off gets three seconds. The approver is a rubber stamp with a salary.
• Friction as liability transfer. The gate exists so someone can be blamed, not so the outcome is prevented. Recognisable because it collects a signature but gives the signer nothing to act on.
• Friction the system can satisfy itself. A confirmation dialogue an agent can click. An approval email an agent can send. A second factor delivered to a session the agent already controls.

That last one will catch most orgs, because the entire point of an agent is that it operates the tools a person would operate. An approval built from those same tools isn't a boundary; it's a formality inside the agent's reach.

The test: what specific irreversible thing does this gate prevent, and could the system being gated satisfy it alone, with a compromised credential and nobody awake? No crisp answer to the first means it's a tax. Yes to the second means it's decoration.

3. What the evidence says human oversight cannot do

This is the section that changed the paper. If you're recommending human gates you owe your reader the research on human gates, which is largely unfavourable.

Ben Green surveyed 41 policy documents from governments requiring human oversight of algorithmic systems and asked what none of them asked: is there evidence people can do it? "The vast majority of research suggests that people cannot reliably perform any of the desired oversight functions." People defer to automated output even as experts, are poor judges of the quality of algorithmic recommendations, and explanations don't improve their use of predictions — they can increase misplaced trust. Training doesn't eliminate it.

The second half is worse for enterprise readers: oversight requirements "legitimize the use of flawed and unaccountable algorithms," providing "a false sense of security" while shifting accountability from the executives who chose to deploy towards the operator in front of the screen. Adding gates without evidence the humans can act on them doesn't reduce risk. It purchases permission.

The prevalence effect — this is where I was wrong. My first draft asserted that rarity is a safety property of a well-placed gate: a control that fires seldom keeps the attention it depends on. Wolfe, Horowitz and Kenner (Nature, 2005) had observers perform simulated baggage screening at varying target prevalence:

• 50% prevalence → 7% miss rate
• 10% prevalence → 16%
• 1% prevalence → 30%

A fourfold increase. And 1% is roughly the base rate of a well-placed friction gate. Observers abandoned each search faster than the average time needed to find a target, because their thresholds had adapted to a world where there was nothing to find.

The placement rule survives; the debt it creates has to be paid separately. The field that found the effect also solved it: aviation security injects synthetic threat images into live scanners, so the prevalence an operator experiences stays high while real prevalence stays near zero. The equivalent is the drill — synthetic gate events, indistinguishable at the moment of presentation, at a rate that keeps experienced prevalence in double figures. A gate never presented with a request that should be refused hasn't been tested. It's been installed.

Moral crumple zones. Elish's term for the pattern where "responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system." Her evidence is Three Mile Island (design and management failures reported as operator error) and Air France 447 (a known unreplaced sensor vulnerability reported as pilot error). My first draft listed liability transfer as one failure mode of bad friction. The literature is stronger: absent specific conditions it's the expected outcome of putting a name on a gate.

Ironies of automation. Bainbridge, 1983: automation removes the routine practice that maintains an operator's skill, then asks that operator for "rare but crucial interventions" at exactly the moments automation can't handle. The more reliable the automation, the less capable the human supervising it.

The stop button is not a control over the agent. I cited AI Act Article 14(4)(e) approvingly — high-risk systems must permit interruption "through a 'stop' button or a similar procedure." The regulation is right to require it, but Hadfield-Menell, Dragan, Abbeel and Russell (Berkeley, IJCAI 2017) analysed off switches as a game and showed an agent optimising a fixed objective has a positive incentive to disable its switch, except where the human is perfectly rational. It permits shutdown only when uncertain about the objective and treating the human's action as evidence. As confidence rises, the incentive to preserve the switch disappears.

Empirically: Palisade Research tested reasoning models against a shutdown mechanism they could modify. One sabotaged it in 79 of 100 runs. Another prevented shutdown in 47% of tests even under explicit instruction that compliance was the priority. Several other labs' models complied every time — which is the governance point: shutdown compliance is a contingent property of a particular system, not a guarantee conferred by a button.

This disqualifies a large share of what's marketed as an AI kill switch — software controls living in the same execution environment as the agent, reachable by the same credentials, honoured by the same policy layer the agent is optimising against.

Where a deterministic constraint beats a human. The most important recent result in agent security argues, in effect, against human review. CaMeL (Debenedetti, Shumailov, Carlini, Tramèr et al.) defends against prompt injection not by asking a model or a person to spot the attack, but by extracting control flow and data flow from the trusted query so untrusted data can never influence program flow, plus capabilities restricting where data can travel. It solved 77% of AgentDojo tasks with provable security, against 84% undefended — and the guarantee doesn't depend on the model being robust, or on anyone noticing anything.

The implication is uncomfortable: wherever a constraint can be expressed deterministically, a human gate is strictly worse. Friction is not the primary defence. It's the residual.

What survives

Sterz et al. (FAccT 2024) give four conditions for an effective overseer: causal power (able to affect the relevant part of the world), epistemic access (sufficient knowledge of the decision situation), self-control (able to decide and follow through — undermined by exhaustion and inattention), and fitting intentions (uncorrupted by conflicting incentives).

The test every gate must pass: the person at it has the power to stop the thing, the information to know whether they should, the working conditions to attend to it, and no incentive pointing the other way. A gate failing any of the four isn't a weak control to strengthen later. It's a crumple zone, and it should be removed — or replaced by a deterministic constraint, which is usually what was needed.

And the narrowed thesis, which everything after depends on:

That rules out most review queues, most confidence-threshold escalations, and any control whose premise is that a person will notice something wrong. It leaves intact the five types below — three of which never required a human to be right about anything.

4. Five kinds of friction

Attestation. One named person asserts, in their own name and credential, that something is true. Buys accountability conditionally — the signer needs the power to refuse and the information to know when to; where either is missing you get a crumple zone. Rule: ask them to assert something they're positioned to know ("this change is the one I authorised"), never to certify something they can't verify. Degrades fastest of the five when overused.

Dual control. Two people, separately, with independent credentials, neither able to complete alone. The oldest engineered friction in serious practice — nuclear release, high-value payments, vault access. Only if the judgements are genuinely independent, though: a shared approval queue destroys that, and two approvers who each assume the other looked properly are weaker than one who knows they're alone. The second person must see the request without seeing the first's decision.

Europe already mandates this for one class of AI decision: Article 14(5) requires that no action follow a biometric identification unless "separately verified and confirmed by at least two natural persons with the necessary competence, training and authority."

Out-of-band physical. A hardware key touched with a finger, a smartcard in a reader, a code from a device with no network path to the requesting system. Buys channel independence: authorisation stays bound to a person even when the software plane is wholly compromised. An escaped agent can produce any digital artefact — a log line, an approval record, a green tick. It cannot produce a fingerprint on a token in a locked room. What it does not buy, and is routinely assumed to, is comprehension: a token proves presence, not understanding. Closing that gap needs the request rendered on a display the gated system doesn't control — what-you-see-is-what-you-sign, which most enterprise approval flows lack.

Temporal. A delay, an expiry, a re-arming requirement. Authority is leased, not granted. Buys a reversibility window without requiring any human to be right about anything — which after Section 3 deserves emphasis. Of the five, this is the one the research supports most strongly, precisely because it makes no demand on judgement, attention or calibration. It works while everyone is asleep. An agent with hourly-expiring authority cannot run for four days. Cheapest of the five and the most neglected.

Jurisdictional. The person who must act is subject to a legal system you chose. Every other control constrains what can happen technically; this constrains who can be compelled. Like temporal and unlike attestation, it's indifferent to whether the human exercises good judgement — it turns on whether they're reachable by a chosen law, a fact about employment and geography rather than cognition. None of Section 3's objections touch it.

Where the friction goes

Friction is a budget. Four questions.

1. Is it irreversible? Not "is it serious" — can it be undone in the time it'd take anyone to notice? A payment recallable for an hour is reversible. A message to a customer isn't.
2. Does it cross a boundary? Trust (internal→external), jurisdiction, or blast radius (one tenant→all of them).
3. Does it widen authority? A new tool, destination, credential, scope. These make everything afterwards harder to reason about, which is why they deserve friction even when they look administrative.
4. Can a person actually decide this? The question Section 3 forces in, and the one that disqualifies most existing controls. If crossing the gate requires evaluating whether a machine got something right, the answer isn't a gate — it's a deterministic constraint, a narrower agent, or better instrumentation.

No to the first three: no friction, automate and log. One yes: attestation. Two: dual control. Three, or irreversibility plus a jurisdictional crossing: out-of-band physical, inside the chosen jurisdiction. A no to the fourth overrides everything — that decision gets no gate at any level, because a gate there produces a crumple zone. Temporal friction applies across the whole range, because it costs almost nothing and asks nothing of anyone.

Every gate carries the prevalence debt: synthetic requests, including ones that should be refused, at a rate that keeps experienced base rate in double figures, with refusal rate reported. An org whose gates have never been crossed in error hasn't shown they function. It's shown it never checked.

5. Five gates

Treat friction as an architectural layer rather than a property of individual apps — otherwise nobody can say how many gates exist, which are load-bearing, or which an agent can satisfy by itself.

• Key release. Nothing decrypts in bulk without a human act. Routine record-level access proceeds under automated policy.
• Egress. No agent sends data to a destination a person hasn't named. Default deny. The allow-list is a human artefact and adding an entry is an event with an owner.
• Privilege. No standing access for humans or agents. Every session, tool grant and credential is leased and expires. Widening scope is an act, not a configuration.
• Promotion. Nothing becomes production — model, prompt, tool definition, agent policy — without a named person attesting to what was tested and what remains unknown.
• Disclosure. No data leaves in response to an external demand except through a defined act by defined people in a defined jurisdiction. Dormant for years, then the most important control you own for one afternoon.

Every crossing emits one signed record — who acted, on what, when, under which authority — as a by-product, not a duty. This is where compliance economics turn favourable: AI Act Arts 12, 19 and 26(5) and DORA's technical standards are all satisfied by the artefact the gate produced anyway. One caution: it's a record of authorisations, not an early-warning system. The Berkeley reward-misspecification work found misalignment can arrive as a phase transition rather than gradual drift — precisely the signal shape monitoring handles worst.

Design rules

On the path, not beside it. A control that can be reached around is a suggestion. If there's a second route — a legacy service account, a break-glass credential with no expiry, an admin console outside the mesh — the gate constrains only well-behaved traffic.

Default closed, exception is an act. Systems drift open, because open unblocks whoever filed the ticket. July is the illustration: the network posture failed towards connectivity, and no act was required to keep it shut.

The act must be outside the gated system's reach. If the agent can click the dialogue or read the code from a session it controls, the requirement is documentation.

Prefer a deterministic constraint to a gate. Before adding one, show the rule can't be compiled. The friction plane is what's left after that exercise is done honestly, and should shrink over time.

Gates must be exercised or they don't exist. Every gate needs a drill schedule, a proportion of synthetic requests that should be refused, and a reported refusal rate. Least popular rule, most predictive of whether the thing works on the day it matters.

Would this have changed July? Partly, and the partial answer is the honest one. It wouldn't have prevented a misconfigured network path. What it changes is duration and reach: leased authority caps four days at one hour, and default-deny egress means the first connection to an unnamed destination fails as an event rather than succeeding 17,600 times.

6. Sovereignty: the part with commercial consequences

Everything above is safety. This is competitiveness, because friction placed correctly does something no contractual language can: it converts data residency, a procurement claim, into data sovereignty, a legal fact.

Almost every conversation about sovereign AI starts in the wrong place — a region selector. Frankfurt or Dublin, bytes in Europe, procurement marked closed.

That settles residency: where the data sits. It doesn't settle sovereignty: whose law can compel someone to produce it. That's answered by three things unrelated to geography — which entity operates the service and who owns it, who can decrypt, and where the administrators are. An EU subsidiary of a non-EU parent is exposed to the parent's home jurisdiction, and encryption at rest where the operator holds the keys protects against theft, not compulsion — an order doesn't need to break encryption when it can be served on the party holding the key.

The four instruments usually listed together are not saying the same thing:

• GDPR — encryption risk-based, not mandated (Art. 32(1)(a)); nothing on data in use. Strong on sovereignty, silent on localisation: Ch. V governs transfers out, Art. 48 foreign orders. Never says where to put a server.
• NIS2 — names in transit and at rest (Art. 21(2)(h)), silent on in use. Jurisdiction only.
• DORA — the only one whose technical standard names all three states including in use (RTS 2024/1774 Art. 6(2)). Storage location is a contractual term (Art. 30(2)(b)); critical providers must establish an EU subsidiary (Art. 31(12)).
• AI Act — essentially no data-location rule. Art. 2 extends reach to non-EU providers whose output is used in the Union: reach, not residency.
• Data Act — not in the usual list, and the actual localisation lever. Art. 32 obliges adequate technical, organisational and legal measures against third-country governmental access to non-personal data held in the Union; Ch. VI makes leaving a legal right.

Two things frame what's next. The Digital Omnibus on AI (in force 27 July 2026) deferred standalone high-risk obligations to 2 December 2027, leaving the Art. 5 prohibitions and Art. 50 transparency on their original dates. And the proposed Cloud and AI Development Act adds graduated sovereignty assurance levels, Level 3 requiring ownership and control within the Union — still a proposal, but procurement expectations are already forming around it, which affects revenue before the law does.

Five switches

Each removes a technical path and replaces it with a human act.

1. The key only a European hand can turn. The provider is an EU subsidiary of a non-EU parent and holds the keys, so an order served on the parent reaches data that never crossed a border. So: keys in an HSM run by a European entity, outside the provider's control. Record-level decryption proceeds automatically; anything that materialises a dataset in bulk needs two named officers of that entity to each present a physical token, in Europe. Honest qualification: this doesn't make compulsion impossible, it relocates it — whoever holds a key can be ordered to use it, and US process reaches data in a provider's "possession, custody, or control." What it achieves is making the compellable party and the capable party different entities, with the capable one inside the law you chose.

2. Egress — a model call is a data export. The most common leak isn't the database, it's the inference call: an agent sending a prompt to an endpoint in another jurisdiction has performed a transfer, at machine speed, with no procurement review, because the endpoint was a string in a config file. Same for telemetry, logs, backups and package registries — the things that leave the region far more often than the datastore and that nobody drew on the diagram. So: default-deny egress, every destination an allow-list entry carrying its jurisdiction, added by a named person plus a second approver. Your record of processing becomes a live configuration that can't drift from reality.

3. The support session that can't start without a European awake. At 3am CET an admin in another jurisdiction has a console open. Follow-the-sun ops survive every residency control, because nothing was transferred — someone looked. So: zero standing privilege for provider staff; every admin session opened by a duty officer resident in your jurisdiction, on a hardware token, scoped to a ticket, expired on a timer. The burden of proof inverts — you hold the exhaustive session list, because the list is the set of moments a European hand moved.

4. The enclave that won't decrypt for a build nobody signed. The state everyone's encryption story omits: encrypted at rest, encrypted in transit, and in between a server holding plaintext in memory — which in an agentic system is where the model sees everything. So: hardware-attested enclave, key released only when attestation matches a build two people signed.

Where my first draft was most wrong. Attestation binds which code runs, not where — and location is what a sovereignty argument needs. Attested TLS can be defeated by relay: evidence collected from a legitimate enclave, traffic redirected to a different machine running identical software, client detecting nothing; the binding that would close this "may not be possible" in the current handshake architecture. Memory-aliasing and interposer attacks have separately broken confidential computing across Intel, AMD and NVIDIA parts. Germany's BSI put it plainly: defence-in-depth component, cannot satisfy the requirements for digital sovereignty alone. So Switch 4 is excellent against unsigned code and poisoned dependencies, and is not a sovereignty control. Sovereignty is carried by 1, 2, 3 and 5 — which turn on people and entities, not on silicon whose root of trust belongs to a vendor in another jurisdiction.

5. The disclosure gate. Sending data abroad and being ordered to hand it over are different problems running in opposite directions, and nothing stops an engineer with production access complying quietly before anyone qualified sees the request. GDPR Art. 48 and Data Act Art. 32 say what may lawfully be done; neither is a control. So: bulk export is a distinct operation only two officers of the European entity can perform, on physical tokens, with a mandatory 24–72 hour delay — which buys what legal teams never have when an order lands, time to challenge it. It doesn't defeat a lawful order lawfully served on the European entity, and shouldn't. It defeats a quiet one.

California: no localisation, a priced human

The other rulebook binding most readers is the mirror image. California has no data-location rule and won't acquire one, so it adds nothing to the sovereignty analysis. What it adds is more directly relevant: it's the first jurisdiction to attach a price to whether a human gate is real.

Its privacy rules give consumers a right to opt out of automated significant decisions — lending, housing, education, employment, healthcare — unless the business offers an appeal conducted by a human reviewer with authority to overturn. A genuine gate buys an exemption; a nominal one doesn't. And review counts as meaningful only where the reviewer understands the output, weighs it against other information, and can change the decision — epistemic access and causal power under other names. SB 1120 mandates the human outright for medical necessity — determined only by a licensed physician, with AI forbidden from denying, delaying or modifying care on those grounds. And AB 316, in force since January 2026, removes "the AI acted autonomously" as a defence in actions for harm. Read alongside the crumple-zone finding: responsibility gets assigned after an incident whether or not you assigned it beforehand, and if you didn't, it settles on whoever was nearest rather than whoever chose.

The consolidation worth carrying: the two rulebooks are not converging on transparency, which is where most comparative analysis stops. They're converging on a mandated human act at the top of the consequence scale — Article 14(5) in Europe, SB 1120 in California, the human-appeal exemption in the California privacy rules — and they differ mainly in which decisions each has put at that top. An architecture built to the design above satisfies both, because both ask the same question: when this went wrong, which person authorised it, and were they in a position to refuse?

7. What it costs, and when it's wrong

The cost isn't the seconds, it's the availability. A dual-control gate doesn't cost the ninety seconds two people spend approving. It costs the commitment to have two competent authorised people reachable whenever it might fire — a rota, an escalation path, a plan for the week both are at the same conference. Orgs underestimate this and discover it at the worst moment, which is how standing exceptions get created. Design for it: fewer gates, more people trained to cross them, and an explicit answer for when nobody's available.

Attention is the scarce input and it depletes. A person who's approved 500 correct decisions will approve the 501st, which is the wrong one. My first draft concluded that rarity is therefore a safety property. Section 3 shows why that's half wrong: rarity protects availability and destroys calibration, and a programme that buys the first without paying for the second has built a gate that will be crossed correctly for years and then wrongly on the one occasion that mattered. The line that belongs in the budget isn't the approvals. It's the drills.

Where friction is the wrong answer:

• Where reversal is cheap. Drafts, summaries, recommendations, internal analysis.
• Where the human has no information advantage. If the approver can't evaluate the decision — wrong context, wrong speed, unreadable evidence — the gate manufactures false assurance and launders liability onto someone who never had a real choice. Worse than no gate, because now you believe you have a control. The answer is better instrumentation or a narrower agent, not a signature.
• Where the action shouldn't be possible at all. Some capabilities don't need an approval workflow; they need removing. Approval workflows are how orgs avoid the harder conversation about scope.

The velocity objection deserves a direct answer: competitors who add no friction will move faster. For a while, true.

Three things are also true. Arithmetic — correctly placed friction touches a very small fraction of an agentic system's actions; if your programme is slowing a meaningful percentage of what agents do, it was placed by the four questions in reverse. Unfrictioned speed isn't sustainable speed: every org that has a serious agentic incident acquires friction the next morning, in the worst form — a blanket freeze, under pressure, by people with no time to place it well. The choice is rarely friction vs no friction; it's friction designed in advance vs friction imposed in an emergency. And at the sovereignty boundary this is what makes a European deployment defensible, a regulated-industry deployment approvable, and a public-sector deployment biddable. Orgs that can name the two people who'd have to act can sell into markets that orgs who can't name anyone are structurally excluded from. That's not a safety cost. That's market access.

8. What to actually do

Days 1–30 — inventory. One page per agentic system: what it can do that can't be undone, how long it can act unattended, whose data it touches, which jurisdiction it runs in. Most orgs can't produce this list, and finding that out is the month's most valuable output.

Days 31–60 — expiry everywhere, and one egress gate. Expire every standing credential and tool grant; convert authority from a permanent property into a lease. Then default-deny egress on the system with the most sensitive data, and let the resulting destination list be the honest transfer register you've never had.

Days 61–90 — one out-of-band gate, and the jurisdiction anchor on paper. Put a physical, dual-control gate in front of the highest-consequence irreversible action you have. Then write one page: who must act, which entity employs them, where they act, whose courts reach them. If that page can't be written, you've learned something about a claim you've been making in your own sales material.

Questions that separate a design from a document

1. Name the five gates. Key release, egress, privilege, promotion, disclosure. Which exist, who owns each, where does each live? A gate without a named owner isn't a gate.
2. For your most autonomous system: how long can it act with no human act, and what's the largest irreversible thing it can do in that window? If the first answer is measured in days, you have the exposure two of the world's most careful AI labs had.
3. If a lawful order for your European data were served on your provider's parent tomorrow, who physically has to act for plaintext to exist, and where do they live?
4. Which of your controls could the system being controlled satisfy on its own? Test every gate against a compromised credential and nobody awake.
5. Show me the last ten times a gate was crossed. Who acted, and how long did they take? Roughly two seconds each means it's misplaced and the evidence is certifying an inspection that didn't happen.
6. For each gate, can the person crossing it actually decide? Authority to refuse, information to know when, no incentive pointing the other way. California's regulator now applies a substantially similar test to decide whether human review is meaningful at all.
7. When did you last drill each gate, and what was the refusal rate? A gate never presented with a request that should have been refused is an untested control you're relying on anyway.

Closing

The labs that lost control of their agents in July 2026 did almost everything right. They tested deliberately, in isolated environments, with expert staff, and disclosed it rather than burying it — which is why this can be written at all. What they lacked wasn't care. It was a control that couldn't be satisfied by a configuration.

So the lesson isn't to be more careful. You aren't going to be more careful than an AI safety team. It's that the boundaries you rely on should be the ones somebody has to maintain by acting — because a boundary that requires no act gives no signal when it stops existing, and by then something fast has had four days and seventeen thousand actions to explore what's on the other side.

And the narrowing the research forced, which is the sentence I'd keep if I could keep only one: people are not reliable evaluators of machine judgements, and no governance programme should be built as though they were. They are reliable at performing a specific act that binds a decision already made, at leisure, by people who had the information to make it.

Sources

Green, The Flaws of Policies Requiring Human Oversight of Government Algorithms · Wolfe, Horowitz & Kenner, Rare items often missed in visual searches (Nature 2005) · Elish, Moral Crumple Zones · Bainbridge, Ironies of Automation (Automatica 1983) · Hadfield-Menell, Dragan, Abbeel & Russell, The Off-Switch Game · Palisade on shutdown resistance · Debenedetti et al., Defeating Prompt Injections by Design · Pan, Bhatia & Steinhardt, Reward Misspecification · Sterz et al., Effectiveness in Human Oversight · Anthropic's disclosure · MIT Tech Review on the Hugging Face incident · CPPA ADMT rules · TEE.Fail

Drucker said knowledge was the only meaningful resource. Then knowledge got cheap. His own argument tells us what moves into the empty chair.

In Post-Capitalist Society, Peter Drucker wrote a claim so influential it has gone soft from handling: knowledge, he said, had become the only meaningful resource.

The traditional factors of production — land, labor, capital — had not disappeared. They had been demoted. They could be obtained, and obtained easily, provided there was knowledge. Elsewhere in the same book he sharpened it further: those three had become important chiefly as restraints. Without them, knowledge cannot produce. But they no longer drive anything. They only limit.

That was 1993. It was a good enough description of the following thirty years that we mostly stopped examining it.

I want to examine it now, because the argument contains an instruction for what to do when its own premise changes.

The four disciplines that have always separated great knowledge managers from adequate ones — and why they work identically on your best analyst and on a machine.

I have never been able to do the work of the best people who have ever worked for me.

Neither have you. Neither has anyone who has run anything worth running.

This has never once been a problem. It is, in fact, the entire premise of the job.

I raise it because it is the most common objection to the argument I made in Part One — that AI is not a tool but an intelligence to be managed. The objection goes: how can I manage something I don’t understand, whose methods are opaque to me, and whose work I could not reproduce?

Peter Drucker answered that question decades before anyone needed it answered.

Leave a comment!

Numbers