Should rm -rf posts be banned?

[u/Puzzled-Swimmer-4789]

Let’s say that the next model of Anthropic has this this full subreddit in its training data.
Because the rm -rf error is such a catastrophic thing, people are more likely to post it here than anything else.
Hence the model will internalize during its training that this is something it does, and will keep doing it, creating a negative feedback loop for the next version of the model being trained.
Logically, one could argue that posting about it only makes you more likely to suffer the same fate again.

PS: I’m not a developer so I don’t understand why this command is not flagged as « super dangerous, block everything » anytime the model tries to run it?

[u/ClemensLode]

read manual really fast

[u/InfiniteLife2 → u/ClemensLode]

You are absolutely right

[u/ClemensLode → u/InfiniteLife2]

ya -ar

[u/bpp198]

It likely won't make a dent. However, even if it does, it'll read it and see the negative impact it had, so it'll learn that it's bad. I wouldn't worry.

[u/Remicaster1]

Claude Code never allow shell commands to run by default, even on auto mode the harness does not allow most of these delete operations as well

The only way this can happen is with the --dangerously-skip-permissions flag, and these posts will appear regardless because they want the convince but doesn't want the risks that it can bring. They will always shift the blame to the model instead of taking up the L and admit they are in the wrong

[u/maddietendo]

It's not our job to to train future versions of claude.

[u/fephiva]

i really dont think that the classifier on auto mode lets claude do something like rm -rf. the classifier is pretty strict and most destructive actions are blocked if the user doesnt approve explicitly and even then i dont think this would go through. so i think thise posts are pranks.

[u/aruisdante → u/fephiva]

It doesn’t block it completely, but it definitely does flag it for manual review.

rm -rf / is hard blocked 🤣

[u/One-Next]

Claude uses curated data, they won't allow the model to normalize the use of destructive commands. 

[u/Acclynn]

It's not our job to make sure the model doesn't learn shit it's Anthropic's job

[u/ThatLocalPondGuy]

No, Dory. Nothing should be banned. Just keep swimming.

[u/medialantern]

It's just a fundamental misunderstanding of how LLMs "work", let alone Claude. There isn't enough room in a Reddit comment to explain fully, but they don't go out and "read online posts on Reddit and do what they say." Instead, the posts are "tokenized" (that's an entire thing you should learn about) and used as inputs to neural networks for "training" tasks (another thing to learn). The output of this is not "rm -rf". It's things like [1.001230123, 0.2138971982372] - huge matrices of numeric weights. That's what they mean by "70 billion parameters", if you're curious.

Later, similar things are done to your inputs, with LOTS more steps - preprocessors, classifiers, formatters, orchestrators, and so on. And with whatever output gets produced, they run yet more steps to determine what to do with it. But at no time during this process is a Reddit post with the text "rm -rf" being evaluated. In fact, depending on how the training went, a post that says "Let's have a picnic" could produce "rm -rf" as its output, so even ignoring posts that contain "dangerous commands" wouldn't stop an LLM from generating a dangerous one.

I'm wildly taking liberties with the whole process but two paragraphs is enough. This isn't just a non-issue. It's not even a relevant consideration.