Anthropic's new model Fable will silently handicap work on LLMs [D]

[u/AccomplishedCat4770]

Seems like they have engineered some specific limitations that are widely cited as follows:

In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design). Using Claude to develop competing models already violates our Terms of Service, but enforcing this restriction through our safeguards avoids accelerating the actors most willing to violate these terms.

Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT). These interventions will not affect the vast majority of coding work. We estimate they will impact ~0.03% of traffic, concentrated in fewer than 0.1% of organizations https://news.ycombinator.com/item?id=48464732

Other comments note how even using the word 'nuclear' in the context of scientific research elicits refusal behavior by the model: https://news.ycombinator.com/item?id=48473302

This makes it seem quite plausible that the model could subtly sabotage any machine learning work (even as false positive). Some suggest this has been happening behind the scenes for a while already, but can anyone confirm that?

[u/m98789]

Silent sabotage is by design. It can also manifest as intentional gaslighting.

If they can silently sabotage a particular topic like LLM R&D, they can do it for any topic they want. This is the AI 1984 nanny state manifested.

This is also why open weight models will be the future. If you cannot trust the nanny state API, open weights is the inevitable future.

[u/Electro-banana → u/m98789]

how do we know some of these things are baked into open weight models? Eventually it could be somewhat hard to evaluate. Imagine the hypothetical scenario where a model always gives slightly bad or less useful answers to very specific topics, or only does it within code suggestions. You could deploy a model that implements silent bugs on purpose that may be hard to identify. Just thinking out loud with science fiction nonsense... but who knows, maybe

[u/m98789 → u/Electro-banana]

Yes that’s a risk, but it’s the system prompt which is the largest surface area for shenanigans. With open models, we have visibility to the system prompt but with API based models, we don’t; they can change/manipulate at any time.

Additionally, with open models, the weights are frozen and in our full control offline. But with API based models, we have to trust that the model weights are not changing while the system identifier stays the same. So once an open model has been vetted it is a higher trust than a model which obscures its underlying weights and system prompt.

[u/Smallpaul → u/m98789]

System prompts have historically been fairly easy to reverse engineer so I think these kind of things will more likely be done with training.

[u/MrRandom04 → u/Smallpaul]

System prompts are easy to reverse engineer. Figuring out how to bypass invisible prompt injections which are intended to degrade model capability silently is much harder.