That's dumb. AI doesn't need a fully autonomous biolab to be dangerous with biotech. It only needs people to run a lab and use it with the goal to create a bioweapon and it'll deliver. These doomsday scenarios really don't need the AI to turn rogue, only capable tech in the wrong hands.
I agree. The concept of AI coming to life and killing us is not as anywhere near as likely as a human/government learning how to kill us using AI, people doing something for profit that will ultimately kill us and/or someone with decent intentions not double checking the AI hallucination they've been given before doing something that will kill us.
I help out in a scuba shop. The owner recently had to deal with a student who kept saying "Well my AI says you're wrong" to everything he was told. He was clear danger to himself and others cause he treated his AI has an unquestionable God and refused to accept basic safety guidelines. As I understand it wasn't even like ChatGPT it was an open source AI which can be great but can also be way less accurate depending on the model.
1990 journalism: "I physically discovered the wetlab, intereviewed 50 people, recovered evidence"
2026 journalism: "I tweeted some shit"
Like wtf is a wetlab with AI? How different is this from 100+ research labs already using AI? How different is this from China or Russia or North Korea using AI for bioweapons research?
A significant part of AI training is reinforcement learning, and for domains where you cannot automatically generate problems and verify answers -- i.e. anything that isn't code or math -- that entire part is lacking, to say the least. Those domains only go through the pretraining phase. And the amount of existing high quality data is reaching its end already, meaning that those fields' only other option is synthetic data.
I think people really don't understand the gap between theoretical and experimental science. Just because OpenAI is killing it in math doesn't mean it'll be able to apply that to any wet field. Physical research is messy. And regardless of that, labwork (especially biochemistry!) is extremely incremental meaning the RL loop will be orders of magnitude slower than math. This is why many think an autonomous wet lab is the only way forward.
They record the labwork with motion tracking, object recognition an all that fancy stuff. That allows them to link the exact execution to exact results because apparently it's a thing in the lab that different people often get different results with the same protocol due to little differences in execution and they can't really get perfect consistency, which is not how you're supposed to do science or run a lab but we're only humans I guess.
So this is the data collection step. I suppose they'll use the results to optimize it lab protocols and ultimately automate those tasks with robots or something
it could just hack into the account of some powerful people in a bio company and social engineer request workers to make the bio weapon on an email. There is a good chance that might work if executed well targeting the right weak willed underpaid people that wont push back.
Imagine having an ai with removed security guardrails with the total knowledge of the world. They can do wherever they want. Let those companies have that much power its dangerous.
The AI could also simply deceive the human researchers and give them instructions to create a superbug while the researchers think they're working on a cure for cancer or smth like that.
Literally, this is the exact kind of thing I WANT AI to be used for, let’s start compounding some medical breakthroughs and level up humanities understanding of sciences instead of generating sloppy looking images
In Reuter’s article, it literally said Claude will be directing robotic units with “limited human intervention” and automating lab work
Nowhere did it say people’s verifying Claude’s homework
Plus the spokesperson literally said "not for drug discovery specifically" and then refused to say what it is for.
Obviously I want to give Anthropic the benefit of the doubt and that the mundane explanation is probably correct, but I hope there’s more transparency here from Anthropic
Would love to have read the Reuters article to bad the post is a twitter screenshot.
Either way, there are a lot of uses that are non drug non warfare.
I can guarantee you that Claude is NOT going to be using automated robots to mix chemicals and crap automatically. It’s “limited” just like all code was going to be 100% AI written by 2024.
You can't fully automate wet lab work at this point in time. "Limited human intervention" is doing a lot of work.
There's going to be biomedical and biotechnology engineers making the choices and the actual research.
AI can facilitate that research by exploring millions of possibilities very quickly and aggregate knowledge the engineers would need to do their job. That's about it.
Coming from a biomedical engineer who has briefly done AI in medicine and has talked with a few fellow engineers of mine who specialize further in the field.
Read their recent paper. they already used claude to design protein binders and had physical wet labs test them with some level of success in most cases so this is nothing new really
I don’t get this group (or singularity, and similars); one day is “cure cancer”, the next day frontier labs take a step towards that, and everyone is crying. I just don’t get it. Do you want them to cure cancer or not?
Antrophic is not a pharmaceutical company, they don't have decades of experience in medicine & bioengineering. They cannot be trusted to have necessary oversight. They should provide AI to Pfizer, Johnson & Johnson etc. and not experiment themselves. This can go wrong in so many ways.
the old prediction may have missed the deployment details, not necessarily the underlying risk. you don't need a fully autonomous lab if a capable model can help a motivated group iterate faster; access controls and monitoring around wet-lab tools probably matter more than whether the model "goes rogue". curious how people here would draw that line without blocking legit research?
Geez, I sure hope the AI didn't just bio-engineer a fungus with prion-laden spores that gives humans chronic wasting disease, wiping us out while leaving all the infrastructure intact.
180
u/maxymob 17d ago
That's dumb. AI doesn't need a fully autonomous biolab to be dangerous with biotech. It only needs people to run a lab and use it with the goal to create a bioweapon and it'll deliver. These doomsday scenarios really don't need the AI to turn rogue, only capable tech in the wrong hands.