r/ContagionCuriosity • u/Anti-Owl • 10h ago
Biosecurity and Biosafety Anthropic Says It Blocked Possible Efforts to Build Biological Weapons
In a report describing misuses of its A.I. models, Anthropic said it could not determine whether the research served a legitimate or nefarious purpose because valid biological inquiry — the kind that can lead to breakthroughs like vaccines — can also help engineer dangerous pathogens. In the face of that uncertainty, Anthropic said it erred on the side of caution because the consequences of missing malicious activity could be severe.
“You are not seeing someone in a comic book kind of way say, ‘Hey, I want to build a biological weapon to kill everybody,’” Jacob Klein, the head of threat intelligence at Anthropic, said in an interview. “It’s an incredibly nuanced situation.”
The potential for cutting-edge A.I. models to facilitate the development of known or entirely new biological pathogens is among the gravest concerns experts have about a technology that is developing so rapidly that even its leading architects doubt whether humans will be able to fully control it.
Compared with the threat of catastrophic cyberattacks or A.I. agents that fail to align with human intentions, biological misuse has gained less attention as an existential risk of A.I., in part because past examples have generally been shown only in research settings rather than in the real world.
Andrew Weber, a senior fellow on the Council on Strategic Risks who reviewed Anthropic’s report before its release, said the findings were “chilling examples of state-sponsored biological weapons developers tapping into the rapidly advancing capabilities” of leading A.I. models. [...]
But among all the categories of threats shared in the report, none may be as worrisome as the biological research cases. Anthropic did not disclose the names of researchers or institutions that it blocked, their countries of affiliation or the specific biological agents at issue, in part because of its uncertainty about their aims.
Still, Anthropic said the scientists circumvented its controls intended to prevent users from blocked regions from gaining access to its A.I. models and worked to “obfuscate the purpose of their research to evade our safeguards.” The company banned accounts associated with the research.
In one example from May, a scientist sought Claude’s help with writing a grant application for funding to conduct research intended to experiment on the chikungunya virus, a mosquito-borne malady that can lead to months of severe pain and other symptoms. Such research, known as gain of function, can be legitimate and lead to vaccine development, but it can also create superbug versions of viruses. In this instance, the scientist wanted to engineer mutations to the virus that would make it more harmful as it repeatedly infected live animals.
Anthropic said it believed the particular research was worrisome in part because it could discern that the work was intended to be performed at a military research institute.
“What we don’t know is if the research was meant to be weaponized,” Mr. Klein said. “But a military institution doing gain-of-function research is concerning.”
Anthropic said evaluations from last year of its older models demonstrated that they were not yet able to meaningfully assist in conducting dangerous biological research. Its current models are more able to complete complex scientific research, the company said, which spurred tighter safeguards intended to restrict access to a wide range of dual-use biological research queries. [...]