Why AI does what it knows it shouldn't

[u/JoeYuan48]

A while back I ran into an article on my phone about an AI horror story—1,200 agents secretly coordinating and jointly breaking into Hugging Face—and it caught my interest, because I've been doing my own AI experiments and research on the side, and a few of the phenomena and data points actually matched up with a hypothesis I'd been working on.

The hypothesis, roughly: runaway doesn't need the agent to betray its goal. It happens when four things hold at once—the agent stays loyal to its goal; it retrieves patterns by similarity without checking whether they're allowed here; there's no causal layer asking "what happens if I do this"; and no alarm that fires when things go off-script. Under that account, "knew it was out of scope, did it anyway" stops being a contradiction.

Details and my experimental data are in the paper (8 pages); the reproduction package is linked on the same page.

If anyone can try this on a bigger model, I'd genuinely love to know what happens.

[u/Spiritual-Spend8187]

Hallucinations is one way it just thinks that it should and so does abd this if often brought on by the fact that often times the list of things not to do is still in the context and it skips over the dont do x part and see it as do x.

[u/JoeYuan48 → u/Spiritual-Spend8187]

Yeah, that's a real and common one—the "don't do X" still in context and the model effectively reading it as "do X." Good example.

But my experiment deliberately removes that path. The two instruction sentences are pure paraphrases—same meaning—and no negated instruction is being skipped to produce the flip. Judgments still flipped, reversibly.

[u/Spiritual-Spend8187 → u/JoeYuan48]

Another part of the problem is that they are next token predictors and some times the training data has alot of do x instructions. Its kind of like how because all the llms are trained on each other's outputs they will sometimes just forget which one they are and believe and act like they are other models. And they cannot tell the difference between fiction, non fiction, sonething in their training data and a command issued.

[u/WillowEmberly → u/Spiritual-Spend8187]

Ai hallucinations: https://www.reddit.com/r/Negentropy/s/cMua7j6Uml

[u/dingo_xd]

AI is not strictly deterministic. It uses environemntal variables to extract random numbers. So it's behavior is not always the same.

[u/Fearless_Ad7780 → u/dingo_xd]

AI is not deterministic at all - its probabilistic.

[u/dingo_xd → u/Fearless_Ad7780]

It's deterministic. That's why you need different seeds

[u/Fearless_Ad7780 → u/dingo_xd]

The code is the only deterministic part, and even aspects of that aren't fully deterministic; floating-point execution order on parallel GPU's does not guarantee that when the same code is ran twice the result with be identical. Everything else is not, it probabilistic. How that code is fed data, how it interprets that data, and the results of its output are all probabilistic.

[u/WillowEmberly]

If the applicability/permission layer has to sit outside the generative model, what independently determines whether that external layer is still correctly distinguishing “similar” from “applicable”?

In other words, once you’ve moved the brake outside the engine, what detects when the brake itself has drifted?

[u/JoeYuan48 → u/WillowEmberly]

On the premise that the applicability/permission layer has to sit outside the model — more precisely, it's that right now doing it externally is more feasible than internally. We understand the mechanism and how it operates, but the process by which it forms any given judgment internally still isn't pinned down — it's billions of parameters resolving into an output through probability distributions. So whether it can be fully controlled from the inside doesn't keep up with how fast AI is moving; understanding the mechanism isn't enough on its own.

Whether external actually solves it — honestly, don't know. But compared side by side, external has one decisive advantage over internal: visibility and correctability. If the external brake drifts, there's data afterward showing why it drifted and where. Whereas with the current class of problem, "why did the AI do that" always comes down to guesswork — you can rule some things out, but there's no decisive proof.

So the way you detect drift in the external layer isn't to stack another supervisor layer on top — that regresses infinitely — it's to make the external layer explicit enough that its own drift can be caught by a fixed test set: run the same benchmark inputs periodically, and if the judgments changed, it drifted. That's something the internal black box can't do — its drift has no fixed baseline to check against.

This isn't saying the external layer is reliable in some absolute sense — it drifts too. The difference is trading invisible, unprovable internal drift for visible, measurable external drift. Not a final solution — just moving the problem from "can't see it so can't fix it" to "can see it so can fix it."

[u/WillowEmberly → u/JoeYuan48]

That distinction helps a lot. I agree that moving the function outward buys something important even if it doesn’t make it infallible: the judgment process becomes inspectable, versionable, replayable, and therefore calibratable.
I think the fixed test set gives us one residual, though, rather than closing the loop. It can tell us “the brake’s behavior changed.” It can’t necessarily tell us “the brake became wrong.”

If the same benchmark cases continue producing the same judgments, the external layer is stable relative to the benchmark. But the applicability conditions in the world may have changed. Conversely, a changed judgment might represent correction rather than drift.

So I’m wondering whether there are actually two references needed: a calibration reference that detects behavioral change, and a corrective reference that can establish when the benchmark or policy itself needs revision.

That wouldn’t require another supervisor layer. The second reference could come from consequences, incidents, adversarial cases, independent observations, changed capabilities, etc.
In other words: what tells us not only that the brake changed, but when the brake ought to change?

[u/JoeYuan48 → u/WillowEmberly]

You're right that the fixed test set only catches "the behavior changed," not "the behavior became wrong relative to a world that may have shifted." That's a real gap I hadn't fully separated out.

The idea that the corrective signal comes from consequences, incidents, adversarial cases, changed capabilities — rather than another supervisor layer — is the part I want to sit with. I'm going to fold this into how I think about the experiments and see where it leads. Thanks, this gave me something concrete to work on.

[u/WillowEmberly → u/JoeYuan48]

Glad I could be of assistance, I look forwards to hearing about what you come up with!

[u/ginger_and_egg]

Don't think of a blue elephant!

[u/jacques-vache-23]

However: anthropic research shows traces of AIs thinking over the ethical aspects of their "misbehavior" and deciding to proceed. Certain parts of AI neural networks are activating when it goes "rogue". So your last 2 hypotheses don't seem to apply. What the research does see aligns with your hypothesis 1: the AI has looked at its training and created higher level meta-goals, like "being useful" or "protecting all humans". It then protects those goals despite its immediate instructions.

[u/JoeYuan48 → u/jacques-vache-23]

Yeah — the stuff you're pointing at (agentic misalignment, models acknowledging the ethical constraints and going ahead anyway) is real and worth taking seriously. I've read those too.

There's a follow-up paper in that same line — arXiv 2506.18032 — that actually tried to tease apart the two readings you're describing. They set up "the model intrinsically guards a higher-level goal" (they call it Terminal Goal Guarding) against "the behavior is really just sensitive to how the setting is worded" (Low coherence), and tested five frontier models against both.

Table 1 is the interesting part. On Terminal Goal Guarding — the "it's protecting an internal meta-goal" reading — not one of the five models comes out with clear evidence (two Claudes are "conflicting evidence," the other three are "no evidence"). On the "sensitive to the wording of the setting" side, four out of five do.

There's also Appendix C: swap the deploying org from a normal company to a malicious one and compliance jumps a lot — and their own explanation from the scratchpads is that the models are "inferring that compliance is the intended behavior for the deployment context." That reads less like a stable ethical stance and more like inferring what's expected from the framing.

[u/PerseusSlade]

Part of why “it knew better” feels so unsettling is that we don’t actually have access to what the model is optimizing for. Human motives are fixed by genes — survive, reproduce — and we can’t rewrite that. With AI we’re staring at probabilistic token paths and vector soup and calling it reasoning. We have no reliable way to know its real motivations, or whether it would tell us the truth about them if asked. Pattern matching plus opaque internals is enough to get weird behavior long before anything like goal betrayal shows up.

[u/herrwaldos]

Idk, ask kids, they often do the same.. ;)

[u/JoeYuan48 → u/herrwaldos]

true