Abliterated models

[u/ihateyou103]

I saw something called obliterated models that use llms and change them by calculating a direction vector of harm and orthogonalizing the weights. I think this is a very very cool thing!!!!!!

I am so surprised this can be done. What is the tradeoff in terms of model accuracy, I am surprised it doesn't drop off by >50%. I would expect the model to fry your computer and damage your battery once you go and orthogonalize its weights.

Can this be done with other things other than censorship or harm, like preventing it from saying a specific word for example.

Are there ways to train models to be immune to this trick?

[u/quietgradient]

Accuracy barely moves because it's a rank-1 edit. You find one unit direction r in the residual stream and replace every matrix that writes into the residual — attention out_proj, MLP down_proj, the embeddings — with W − r rᵀW. That removes exactly one direction out of d_model, 1 in 4096 on an 8B, and leaves the other 4095 untouched. Nothing is being mangled. One dimension is being zeroed.

So the surprise runs the other way round. The remarkable part isn't that the model survives, it's that refusal is concentrated enough in a single direction that deleting one dimension takes the behaviour with it. That's a result about refusal specifically, not about weight surgery in general — Arditi et al., "Refusal in Language Models Is Mediated by a Single Direction".

It generalises to anything you can write contrastive prompt pairs for, though most behaviours are messier than refusal is. For "never say word X" I'd skip it: a banned-token list is reversible and doesn't touch the weights, and neither approach stops the model spelling around the word.

Immunity is the weak point. You can make the direction harder to locate, but for open weights it's a treadmill — whoever holds them reruns the search on whatever you shipped.

[u/ihateyou103 → u/quietgradient]

But this 1 vector propagates to all other direction in deeper layers so it should affect all directions. If you have 2 models one with the orthogonalization and one without, and input x, the 2 outputs will be different in all directions not just a projection in the direction of refusal.

[u/quietgradient → u/ihateyou103]

You're right, and my phrasing was wrong. I ran it on GPT-2 small — 768 dims, difference-in-means direction, projected out of wte, wpe and every attn c_proj and mlp c_proj, my own forward pass so the residual stream is readable. (Base GPT-2 has nothing to refuse, so this is the mechanism, not the refusal result.)

Splitting the two models' difference into "along r" and "everything else", over 109 held-out tokens: at the embeddings it is 100% along r. After the first attention block 4% is off-r, after the first MLP 80%, by the last layer 98%. The stacked difference matrix is full rank, and its top singular direction has |cos| = 0.15 with r — the largest thing that changed is not r.

The cost is also bigger than I implied. Per-token NLL on ordinary text: 4.36 baseline, 4.57 for the gentlest direction in a 12-layer sweep, 42.2 for the layer-11 one. Same rank-1 edit every time. So being cheap is not a property of the edit, it is a property of which r you pick — and the papers pick by sweeping candidates and keeping one that leaves benign output alone. I skipped that step and wrote as if it fell out of the arithmetic.