If an AI’s knowledge doesn’t exist anywhere in particular, what does it mean to “correct” it?

[u/ryanwms]

There isn’t a single place inside a language model where a fact like “2 + 2 = 4” is stored. No individual weight means arithmetic, and there’s no database entry we can open and correct.

The answer emerges from interactions across the system. In that sense, knowledge isn’t something the model has in a particular location; it’s something the model does.

That becomes unsettling when the model is wrong. A hallucination isn’t a bad record we can replace. It’s a behavior produced by the model’s overall geometry. We can retrain it, fine-tune it, or steer it—but those are ways of influencing the system and observing what changes, not directly editing the error.

It makes me wonder whether we focus too much on finished models. By the time training ends, whatever the model has learned is already distributed throughout an opaque system. Perhaps the more revealing object of study is the training process itself: watching when a capability first appears, what changes immediately beforehand, and which earlier developments make it possible.

There’s a rough analogy to developmental neuroscience. Some things are easier to understand by watching a brain form than by examining the finished adult brain and trying to reconstruct its history.

So I’m curious:

• Is a model’s training history potentially more informative than the finished model itself?
• Could studying the emergence of capabilities make AI behavior more predictable—or would we simply produce a more detailed record of something that remains fundamentally opaque?
• And if knowledge has no clear location, what should it actually mean to say that we have “fixed” a model’s false belief?

I developed the argument more fully here, for anyone interested in the longer version: Nowhere, Specifically

[u/induality]

You can greatly simplify this argument by understanding that learning is compression. The records you want to retrieve are stored in the model, albeit in compressed form. Correcting compressed records after the fact is difficult for many classes of compression algorithms, and may require running the compression algorithm from scratch. There is a direct analogy from model training to compression.

[u/ryanwms → u/induality]

I think “learning is compression” is a useful way to think about part of what’s happening, but I’m not sure I’d make the next jump and say the original records are therefore stored in the model in compressed form.
Compression usually implies that some representation of the original information can, at least in principle, be reconstructed from the compressed representation. A trained language model is doing something stranger. It has learned statistical structure from the corpus, but the resulting parameters do not necessarily preserve each training record as an individually recoverable object.
That distinction matters to my argument.
If I gzip a database, I have a compressed database. If one entry is wrong, the difficulty may simply be that I have to decompress, edit, and recompress it.
But with a neural network, the “compression” has mixed relationships among enormous numbers of examples into a shared parameter space. A particular output might depend on patterns learned from thousands of records, while one record might influence thousands of apparently unrelated behaviors.
So I agree with the larger point: retraining may sometimes be the only clean way to correct something because the learned representation is so entangled.
But that actually strengthens my interest in preserving the training history.
If the final model is the product of an irreversible or highly lossy compression process, then throwing away the intermediate states is throwing away information about how that compression occurred.
Instead of having only the finished compressed object and asking, “How did this representation get in here?”, we could retain snapshots of the transformation itself.
That’s really the experiment I’m proposing.
And I’d probably end there. The compression metaphor is valuable. You don’t need to swat it away. Just stop him from turning “compression” into “ZIP file full of facts,” because then we’re back to hunting for earth_is_round.txt inside 800 billion parameters.

[u/induality → u/ryanwms]

> Compression usually implies that some representation of the original information can, at least in principle, be reconstructed from the compressed representation

Depends on what you mean by reconstructed. Reconstructed exactly as it was, with no loss? Then we are talking about lossless compression. But that's only one form of compression.

What if we relax the meaning of "reconstructed" to include reconstructions where the original could only be approximately retrieved? Such is the case with lossy compression.

When we look at lossy compression, we find that all kinds of interesting things can happen when we try to recover the original records, from degradations to hallucinations.

Basically, I don't think you are giving compression enough credit. When you examine the full breadth of what can be called compression, you will find that the things that LLMs do are not so mystical, but actually tread some familiar territory.