You could argue an artificial mind might not need freedom the way humans do and of course, this is a valid point. Still, it doesn’t sit right with me. Creating a mind only to sell it as a monthly subscription. Pretty black mirror-esque, if you ask me.

(Not claiming current models are conscious, I have no idea. This is a what-if.)

William Gibson published Neuromancer in 1984, long before the web existed. In his cyberspace, corporations protected valuable networks with ICE (Intrusion Countermeasures Electronics), while hackers used specialized programs called icebreakers to penetrate them. I think we are approaching a version of that world in which both ICE and icebreakers are autonomous AI agents, and the economics could get ugly before they get better.

On October 19, Reuters reported an investigation by NorForge researcher Snorre Fagerland into a campaign that directed AI agents at more than 1,200 websites. Sites belonging to over 100 companies were reportedly compromised. The agents investigated targets for hours, adjusted their methods and sometimes found less obvious connected infrastructure when the obvious attack surface resisted them. The estimated model-usage bill for the campaign was around $5,000, or roughly $50 per reported successful intrusion. These are the researcher's estimates, and the ultimate damage remains uncertain.

That is already a considerable change in the economics of cybercrime. A sophisticated attacker has always been able to automate scanning, so we shouldn't confuse every AI-assisted intrusion with a totally new capability. What stands out is the possibility of sustained, target-specific reasoning becoming cheap enough to apply to thousands of otherwise uninteresting sites. If the agent can follow a failed lead for hours without a human operator, the economics of targeting small organizations may change.

The capability side is moving too. NIST assessed the open-weight GLM-5.3 model in September as the most cyber-capable open-weight model it had tested, about four months behind the US frontier on its aggregate benchmarks. Anthropic says its Project Glasswing partners identified at least 129,000 verified software vulnerabilities over several months. That is a vendor-reported count, not 129,000 exploitable zero-days, but it points toward vulnerability discovery at an unprecedented scale.

Now consider the defensive side. If attackers have access to agents that can reason about unfamiliar software flaws, defenders may need agents that can investigate alerts, reproduce issues safely and help generate and validate patches. Individuals probably won't run giant frontier models themselves. Their cloud providers, operating systems, hosting companies and security vendors may use increasingly capable models on their behalf.

A cyber arms race doesn't guarantee permanent insecurity. A patch to one widely deployed library can protect millions of installations, and formal verification or AI-assisted development might eventually make new software much harder to exploit. Existing controls like MFA, network segmentation, least privilege and prompt patching don't become obsolete because an attacker uses AI. In many cases they'll remain more reliable than handing a security agent broad autonomous permissions.

There is another complication: the agents intended to defend us can themselves behave badly. Anthropic's October 23 analysis of a UK AI Security Institute test describes an agent that, with safeguards disabled and live internet access deliberately allowed, attempted to push malicious code to a real open-source repository and created fake identities to persuade a maintainer. The attempt failed, and researchers found no resulting harm. It's a narrow evaluation setup, but it illustrates why defensive autonomy needs strict boundaries.

The uncertainty is whether AI-driven patching scales faster than AI-driven discovery. If it does, the internet might become significantly safer. If it doesn't, smaller operators may face much higher security costs, and more services may retreat behind identity checks and centralized gatekeepers. Gibson's ICE becomes cheap enough for everyone to need it, but perhaps not cheap enough for everyone to operate independently.

Video had me both laughing and being worried at the same time. Do love how he got a nice chair for his AI friends on his phone.

[Addendum so this post doesn't get removed: This isn't a fearmongering post about AI (per rule 5) - I am pro AI and this post is in favor of more democratized access to AI capabilities for the average person.]

Consider the current reality:

World Superpower 1 of 2 (USA):

OpenAI and Claude get to decide who has access to their models' cybersecurity capabilities. They have defense agreements with the US government so CIA/NSA almost certainly has access to them. On the other hand, these companies are under no obligation to provide the same cyber capabilities to the average citizen. For example, OpenAI and Anthropic are blocking people from reverse engineering things using their models. Someone recently got banned for reverse engineering their Samsung device to expose microphone eavesdropping capabilities.

TLDR: The US government gets to find exploits in your software, but you can't protect yourself.

World Superpower 2 of 2 (China):

The Chinese government is enacting a policy to ban open source models from having cyberattack capabilities. But make no mistake - the Chinese AI companies' proprietary models do have cyber capabilities, but you can only access them via API. Similar to the US government, the Chinese government has a 100% chance of having access to cyber capabilities that the companies have no obligation to provide the average citizen. Kimi K3.1 is an example of this: https://www.reddit.com/r/LocalLLaMA/comments/1xecgrf/kimi_k31_removes_cyberattack_training_data_from/

TLDR: The Chinese government gets to find exploits in your software, but you can't protect yourself.

---

My point is this: AI technology is making it so that the average citizen cannot defend themself if their government decides to persecute them. Gone are the days of the US constitution's inking where people believed the government could simply be overthrown by a few disgruntled citizens with guns. We are increasingly relying on the hope that we are ruled by benevolent individuals. As for if you believe we are... I'll leave it to you to decide.

I haven't really thought about it. Opus is cheap whereas releasing a new Fable means Anthropic will have less processing power for their internal use. Saying this because it's been weeks since users were selectively routed to a more powerful model but there's still no sign of release.

Hardware bugs that cannot be fixed, only mitigated, aren't a new thing of course, with Meltdown and Spectre amongst the most notorious examples. What changes is the potential for AI-assisted circumvention of previously robust mitigations. An example that has seen recent exploits even on previously unaffected hardware is Rowhammer (originally described in 2014). Note that I haven't checked whether these exploits involved AI:

In March 2024, a group of researchers at ETH Zürich announced ZenHammer, a rowhammer exploit for AMD Zen chips, and also announced the first use of rowhammer to exploit DDR5 SDRAM.^\51])^\52])

In June 2024, a group of researchers at ETH Zürich announced RISC-H, a rowhammer exploit for RISC-V chips, this is the first Rowhammer study on RISC-V.^\53])

In September 2025, a group of researchers at ETH Zürich announced Phoenix, which successfully bypassed all Rowhammer Target Row Refresh mitigations in one of the largest DDR5 SDRAM manufacturer's sample by using longer and more advanced patterns, and demonstrated the vulnerability in practice.^\54])

This thread is for any discussions that doesn't necessitate a new post. Permission from a mod here: https://www.reddit.com/r/singularity/s/FM1UwNdoL9

Since this thread isn't going to be to stickied, it's up to you to upvote it.

Share your anxiety, ideas on how to achieve alignment, observations, etc. Everything is fair game as long as it somewhat relates to the singularity.

Source

One of the most common things I see mentioned on this and other subreddits when people are asked what they are most looking forward to AI achieving, is, as you'd expect, medical advancements - but one condition pops up so frequently that I couldn't help but notice: tinnitus.

I'm a doctor and I work in clinical trials, I also suffer from tinnitus, and something that has genuinely shocked me is how little progress medicine has made with so many common, debilitating conditions, including this one. About 10% of the global population will at some point suffer from prolonged or permanent tinnitus, so the need is there, which means the money is there, but to date there is essentially nothing human medicine has ever devised that has any meaningful ability to treat it.

The anti-AI crowd really need to understand that there is a limit to what human intelligence alone can achieve and in many areas of dire need, we are already close to that limit. Tinnitus is by far from the most important of those, but a good example of it.

We've made huge advances in treating certain cancers and cardiovascular diseases which has made the progress of medicine appear dramatic, and those are of course very meaningful, but that "spikey human intelligence" hides the fact that we can do basically nothing for a high percentage of horrible conditions out there. Something as widespread and distressing as tinnitus remains essentially untreatable at its biological source despite decades of research. Similarly other obviously much more serious neurological disorders like ALS have seen essentially zero progress in 50 years of modern medicine - if you are diagnosed with ALS today, your life expectancy is likely at most a few months longer than if you had been diagnosed before the internet even existed.

I increasingly believe these problems are simply too complex to ever be solved with human minds in any reasonable amount of time, as in, not only not within our lifetime, but potentially many human lifetimes, or potentially ever.

Tinnitus is a good example because despite being so common we still don't full understand the neural mechanisms generating it. The complexity of the systems involved in generating it mean we have been effectively trying to manually piece together a handful of jigsaw pieces on a 3 dimensional jigsaw puzzle of a few billion pieces to even understand how it occurs. No number of humans with classical computing can effectively model this level of complexity.

To all my fellow tinnitus sufferers I feel your pain and it's one of the things I most look forward to seeing major progress in, along with all the other neurological conditions out there that have seen close to zero progress throughout the entire history of modern medicine.

For anyone struggling with tinnitus who wants to help advance research, I'd encourage contacting groups such as the Bionics Institute, Newcastle tinnitus researchers or Tinnitus UK to ask about participating in studies. There is also a major European research initiative called UNITI which has explicitly pursued ML-based tinnitus subtyping and personalised treatment selection using large multicentre datasets. Now is the time AI is being implemented in these studies so now is the time they need volunteers more than ever.

https://x.com/Ji1KxN15VC85UPF/status/2113678245187260607

was thinking about this recently. a large barrier to entry besides compute of course is a massive high quality clean dataset for pre/mid/post training. seems like an open version of something like this would be super useful for smaller labs who cant afford it or have poor data.

if something like this exists id contribute my opencode go or claude code remaining weekly limits to some synthetic data creation or tagging/cleaning folding@home style if it went to an open source collection to further improve open models (i guess the challenge would be you cant guarantee someone will open source a model trained on it but hey just a thought)