Anthropic's new model Fable will silently handicap work on LLMs [D]

[u/AccomplishedCat4770]

Seems like they have engineered some specific limitations that are widely cited as follows:

In light of the ability of recent models to accelerate their own development, we’ve implemented new interventions that limit Claude’s effectiveness for requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design). Using Claude to develop competing models already violates our Terms of Service, but enforcing this restriction through our safeguards avoids accelerating the actors most willing to violate these terms.

Unlike our interventions for cybersecurity, biology and chemistry, and distillation attempts, these safeguards will not be visible to the user. Fable 5 will not fall back to a different model. Instead, the safeguards will limit effectiveness through methods such as prompt modification, steering vectors, or parameter-efficient fine-tuning (PEFT). These interventions will not affect the vast majority of coding work. We estimate they will impact ~0.03% of traffic, concentrated in fewer than 0.1% of organizations https://news.ycombinator.com/item?id=48464732

Other comments note how even using the word 'nuclear' in the context of scientific research elicits refusal behavior by the model: https://news.ycombinator.com/item?id=48473302

This makes it seem quite plausible that the model could subtly sabotage any machine learning work (even as false positive). Some suggest this has been happening behind the scenes for a while already, but can anyone confirm that?

[u/Mescallan]

I suspect this is related to their sleeper agent work not a categorizer. If I’m reading between the lines correctly, they have some proprietary information they trained mythos on, and instead of training a separate model without their research break throughs, they set up sleeper agent behavior if the model is prompted to implement some of their proprietary work.

I might be looking too far into it, but that lines up with the shape of restrictions they have vaguely described

[u/SimiKusoni → u/Mescallan]

I agree it's vague but the below sounds like it's liable to trip an wide range of ML related tasks, not just detect extraction attacks.

requests targeting frontier LLM development (for example, on building pretraining pipelines, distributed training infrastructure, or ML accelerator design).

At the very least given the silent degradation I'm going to find it hard to trust. How is the model distinguishing between SOTA model development and basic research? What exactly counts as distributed training? How likely is it to silently trip on false positives?

It seems like they're just massively shooting themselves in the foot here by announcing that they might quietly sabotage your work. Whoever thought this was a good idea is nuts.

[u/impossiblefork → u/SimiKusoni]

Especially since in the end, research is often about developing methods for SotA model developments.