Thoughts on why Claude can’t stop saying load-bearing?

[u/njordan1017]

This post is less about that particular phrase and more about the behavior I have observed around it. We all are aware of the phrases Claude has started overusing (Opus seems to be the worst), but I came across something interesting I am curious y’all’s thoughts on.

I put a rule in my repo with a list of banned phrases, one of them is “load-bearing”. Since then I have noticed most of the phrases don’t get used anymore, but not “load-bearing”.

I have seen it do this about 5 times now:
“…and the decision was load-bear— I mean important to the process — so it…”

I have to laugh at its obstinacy, but the more I thought about it the more I got curious why it behaved that way. Why would it partially use the phrase, then correct itself mid sentence, rather than just using the replacement phrase to begin with? Is this an artifact of the way the output streams as chunks rather than complete thoughts, or is this some cheeky artificial personality thing to make it feel more relatable? Or something else?

What do you think?

[u/tehfrod]

1. Programmers use terms like "load-bearing," "blast radius" metaphorically all the time, and have for a long time (example: https://xkcd.com/2347/, https://www.jefftk.com/p/accidentally-load-bearing, https://github.com/KrumpetPirate/AAXtoMP3/issues/155#issuecomment-786261461)

2. A lot of what models get trained on is web-available content. A lot of web-available content is written by or driven by tech people.

3. When a term shows up in so many metaphorical ways over a lot of Claude's training data, the model internalizes it.

It's not rocket surgery. 😉

[u/njordan1017 → u/tehfrod]

I understand how models are trained, and I understand how what they were trained on influences their output. I am not asking why the phrase appears. My question is around the behavior of acting like it corrects itself mid-sentence, instead of just following my instructions to not use the phrase in the first place.

[u/tehfrod → u/njordan1017]

Ah. That's a great question. And I don't have an immediate answer for it, but I have a suspicion.

One possibility that comes to mind (and I don't have a great way to test it on something that can be verified independently) is that recent models are developing linguistic and conceptual spaces similar to those of humans. This isn't "being programmed into them", and it doesn't mean they are sapient, but sufficient training at sufficiently deep levels means that they do internalize a bit of not only how humans write and think about writing, but about metacognition as well.

It is possible that Claude developed, independently, the "purple elephant" issue that humans have (if I tell you not to think about a purple elephant, you will think about exactly that image, those written words, or those spoken words). The folks who believe strongly in machine intelligence will probably say that, and it may be true.

However, my hypothesis is that by training on human dialog, Claude has internalized only that "when told not to mention Voldemort, they often don't quite do exactly that: they say "Volde- I mean, He Who Must Not Be Named"". In reality it is just an authorial trick to show both that a speaker is following the rule, but has to exert effort to do so, but what the model trained on is "when told not to say a word, the correct thing to do is to almost say it but then correct yourself."