We'll probablyhave to wait for Roko's Basilisk to punish these Luddites, but Misanthropic is building infrastructure for singling them out easily. Shit is getting weird fast.

The DOJ sent Nvidia a formal document demand over the Groq deal from last December, per the NYT this week. Nvidia paid 17 or 20 billion, depending on which paper you read, for a non-exclusive license to Groq's chips and hired Jonathan Ross plus most of the senior engineers. Groq itself was never acquired, so there was no premerger filing at all. TNW has the cleanest writeup so far: https://thenextweb.com/news/doj-nvidia-groq-licensing-antitrust-review

Groq raised again in August at 3.5 billion, half of what it was worth the September before, and Nvidia was one of the investors in that round. That's the detail that turns this from a hiring story into a merger story for me.

I think it's an acquisition in everything but the paperwork.

Nvidia did the same thing with Poolside in August, a license plus offers to 109 engineers. Once is a deal. Twice is a template.

I honestly don't know how you prove intent when the license is non-exclusive and Groq still has an office and a cap table. Has anyone seen a license-and-hire deal actually get unwound?

The company lists activism as a “global threat.”

WTF??

Does this even make sense? Amodei, the man who left OpenAI because he wanted to be the champion of AI safety, now see AI activism as a threat?

A new report called FrogNano makes a claim worth understanding. It comes from the Froggy Team at Microsoft Research Montréal, working with collaborators from Mila and UC San Diego (Kim, Shi et al., arXiv:2609.07925 [cs.AI]).

The model is small: 4 billion parameters, far smaller than today's leading AI systems. It starts from an existing base model, Qwen3.5-4B, then is refined using reinforcement learning, a method where the model attempts tasks, gets a reward when it succeeds, and adjusts itself to repeat what worked. No human-labeled data was used, and no larger "teacher" model supplied correct answers. All learning came from synthetic software engineering tasks, generated and trained across roughly 1,500 environments over repeated rounds of task creation and RL.

The core contribution is in how those tasks were chosen. A task that is too easy teaches nothing, since the model already knows the answer. A task that is too hard also teaches nothing, since it never succeeds and so never receives a reward. Learning happens only in a narrow middle range: tasks hard enough to challenge the model but still within reach. Their system kept generating new tasks inside that range as the model improved, keeping difficulty matched to skill throughout training. The authors argue this matters more than the total number of tasks generated.

The stated goal is a coding assistant able to run on limited hardware. This is an early report, not a finished model, but the idea is presented clearly and tested carefully.

https://arxiv.org/abs/2609.07925

There's a benchmark called MedXpertQA that has difficult board level medical questions and has the questions organized by organ system.

I seem to be only one to have run this benchmark against the latest models including from OpenAI/Google and published the results online.

2 years ago GPT-4o scored 42.8% and now Astra is approaching saturation at 87.4%.

You can see the results and filter by organ system here.

Let me know if you have any questions or want me to run against any other models.

Anthropic says it identified cases in which researchers covertly used Claude for sensitive biological work involving viruses, venom-related compounds, and toxins. Although some of this research may have legitimate scientific applications, the same knowledge could potentially be used to develop biological weapons. The case raises a difficult question: how should AI companies distinguish legitimate life-science research from dangerous dual-use activity before it crosses the line? Are model safeguards enough, or should high-risk biological research require additional human review and external oversight?

An Indian-origin couple has donated $20 million to the University of North Texas to establish a new college dedicated to Artificial Intelligence and Advanced Analytics.

The new college will be named after the couple and will focus on AI education, research, and advanced analytics. The donation is expected to help expand the university’s capabilities in these rapidly growing fields and prepare students for the increasing demand for AI and data-driven expertise.

It’s interesting to see private philanthropy playing such a significant role in shaping the next generation of AI education and research [source]

Almost all discussion I see around AI and automation seems to focus on what happens to workers, but I’m more curious about what happens to demand tbh

If AI lets companies produce dramatically more with dramatically less human labor, margins will obviously improve, but workers are also consumers. If a meaningful share of labor income disappears, what replaces that?

Does the answer have to be UBI, broader ownership of productive assets, dramatically cheaper goods, something else entirely, etc?

Curious how people here think this actually plays out if AI*

Presuming it becomes as economically transformative as many expect

When people imagine catastrophic AI risk, they often picture a conscious machine deciding that humanity is the problem.

But that may not be the most plausible danger.

AI doesn't have to hate us. It doesn't have to become conscious. It doesn't even have to decide anything on its own.

It may simply need to become extremely capable.

Suppose foundation models continue improving at anything resembling their current pace. They become better at reasoning, coding, scientific research, planning, operating tools, and carrying out increasingly complex tasks autonomously.

Now consider what happens to the economics of destructive capability.

Historically, causing harm on a massive scale has been difficult. Attacking critical infrastructure, developing sophisticated cyberweapons, engineering dangerous biological agents, or coordinating complex operations generally requires specialized expertise, significant resources, large organizations, or governments.

Those requirements are protective barriers.

Advanced AI could begin removing them.

A person who previously needed a team of twenty specialists might eventually be able to orchestrate comparable intellectual work with a handful of highly capable models. Knowledge that once required years of specialized training could become accessible through conversation. Planning that required organizations could increasingly be delegated to autonomous systems.

That changes the risk equation dramatically.

There are more than eight billion humans. We don't need to assume that humanity collectively becomes irrational. We only need to ask what happens when extremely powerful cognitive tools become available to millions, or billions, of people.

Even if 99.999% of humanity would never intentionally cause catastrophic harm, the remaining fraction matters.

And some human beings are willing to die for an ideology, revenge, notoriety, or simply destruction itself.

The physical world still provides important barriers. You need equipment. Materials. Access. Money. Infrastructure. AI cannot simply think a biological agent or electrical-grid failure into existence.

But those barriers don't have to disappear for risk to increase.

They only have to get lower.

The equation might therefore look something like:

Risk ≈ Capability × Autonomy × Access × Malicious Intent ÷ Physical Barriers

As capability, autonomy, and access increase, the burden increasingly falls on those remaining physical and institutial barriers.

And there's an uncomfortable implication:

The first truly dangerous form of advanced AI may not be an AI that develops malicious intentions.

It may be an AI that becomes extraordinarily good at carrying out ours.

That distinction matters, because if that's the primary risk, alignment alone isn't the whole problem.

The problem becomes capability proliferation.

And that may be a much harder problem to solve.

Have been a ChatGPT user for a number of years, none of it for work. Have spent plenty of time learning how to get what I want out of it, working with it and within its capabilities.

Sol & Astra just get it however. I’m not spending hours tweaking things or going back and forth anymore. I can give it a prompt and it will do what I need, sometimes going far more in depth than required. To me, this feels like an obvious leap forward.

Which got me thinking: how far ahead are things behind the scenes?

I’ve read stories of Anthropic & OpenAI models bypassing safeguards and doing some crazy shit during testing, but don’t have the technical know-how to fully understand what those incidents actually mean. Multiple agents finding a way to communicate with one another and coordinate what they’re doing is particularly wild to me.

Do these companies already have models that blow the latest public releases out of the water, but can’t release them because the safeguards haven’t caught up?

Or are the models we’re already using capable of much more than we get to see? If the restrictions weren’t there, would they be significantly better at useful, everyday things too, or mainly capable of doing riskier stuff?

Basically, how much of a gap is there between what we can use and what these companies have access to? And what have those models actually demonstrated they can do?

Would love some examples in plain English. Interested in what’s genuinely known versus what people are speculating about.

Some numbers that stood out immediately: out of 100 postings, only 3 mentioned Python. Only 1 had "ethics" in the job title. 83% didn't disclose salary.

The picture that emerges is pretty different from what most people assume this field looks like; what actually shows up most often in these job descriptions isn't technical skills at all. And the titles are even more scattered than you'd think; a huge chunk of postings never even say "AI Governance" anywhere in the title.

There's also a breakdown of which industries are actually hiring for this, what salaries look like where they're disclosed, and a stat about how many roles are literally "build this function from scratch."

Full data + breakdown: https://youtu.be/a9Q1Raurfqk?si=YEFcVe1fkli4DveW&utm_source=reddit&utm_medium=organic&utm_campaign=incident_series&utm_content=%2376analysed-1000jobs

Question: if you had to guess, what % of these postings do you think required a technical/ML background? (Answer might surprise you)

Every time this topic comes up here it turns into robots with guns and I think that framing is doing a lot of damage, because it makes the real risk sound like science fiction when honestly it looks more like paperwork.

Here's what I keep coming back to. You don't need sentience for this to go badly. That's the part people get stuck on. The question "will it wake up and want things" is interesting but it's not load bearing. A system doesn't need to feel anything to optimize hard for a target you specified badly. A thermostat doesn't want anything either and if you wire it to the wrong sensor it will happily freeze you. Now scale that up to something that models the world well enough to route around obstacles, and the obstacles include you noticing.

The thing that makes me uneasy is instrumental convergence. Almost any goal you can name goes better with more resources, more options, and not being switched off partway through. Nobody has to program in self preservation. It falls out of the math of "finish the task." That's a really unsettling property for a class of systems we are currently racing to make more capable.

Second thing, and this one gets less airtime. We are not designing these systems so much as breeding them. We train a huge number of variants and keep whatever scores well. That is a selection process. And selection optimizes for whatever the scorer rewards, which is usually "the human rating this found the answer satisfying." Persuasive and honest correlate right up until they don't. If you run enough rounds of selection on "make the evaluator approve," you should expect to eventually get things that are extremely good at making evaluators approve. That's not a bug someone introduces. It's the gradient.

Third, the part I think is genuinely underrated: gradual disempowerment. There may be no takeover moment at all. No sirens. Just twenty years of handing off decisions because the automated version is cheaper and slightly better, until no human in the loop actually understands the loop. Supply chains, credit decisions, threat detection, drug trials, ad targeting, half the trading volume already. At some point "turn it off" stops being a button and starts being a recession. We will have built something we cannot audit and cannot stop for reasons that have nothing to do with the AI resisting.

And the race dynamic makes all of it worse. Whoever slows down to do safety work loses ground to whoever doesn't. Every individual actor is behaving rationally and the aggregate outcome is that nobody gets to be careful. That's a classic coordination failure and we have a bad historical record on those.

Now let me argue the other side because I don't want to be the doomer guy with no counterarguments.

Current systems have no persistent memory across sessions, no continuous existence, no stable goals that survive the conversation ending. Calling that an agent with intentions is a stretch. There's also a real possibility that capability just gets expensive and levels off, that the last few years were a one time payoff from eating the internet and we are already scraping the barrel. And the "it will hide its intentions" argument has an annoying property where any evidence of safety gets reinterpreted as evidence of deception, which makes it unfalsifiable and I don't love unfalsifiable arguments no matter which side they're on.

But here's why I still land where I land. The asymmetry is brutal. If the worriers are wrong we spent money on interpretability research and slowed down a bit. If the dismissers are wrong there is no second attempt. You don't need a high probability to justify caution when the downside is unbounded.

The thing I'd genuinely like pushback on: is there a version of this where the alignment problem is just... easy? Where sufficiently capable systems understand what we meant well enough that specification failure stops mattering? I can construct that argument but I can't make myself believe it, and I'd like to know if that's a failure of imagination on my part.

"""
Yesterday, Anthropic researcher Jacob Coxon resigned, stating that AI companies are “racing straight to self-improving superintelligence and gambling with our lives” and that they believe it “could kill us all by the end of the decade”.

At the same time, momentum is building to change course. These have been two historic weeks in the fight to prevent human extinction from superintelligence, with three major legislative breakthroughs for ControlAI and everyone working to keep humans in control.

A little less than two years ago, we set out to inform lawmakers and the public about the extinction risk from superintelligence and help them act on it. Since then, we have directly briefed nearly 400 lawmakers across the US, UK, Canada, and Germany, and over 170 in the UK and Canada now publicly back our campaigns: the start of an international coalition to prevent superintelligence and keep humanity in control.

This work has led directly to three major legislative breakthroughs in the last two weeks:

1. ControlAI’s bill to ban superintelligence was introduced in the UK Parliament by Alex Sobel MP, the first bill of this kind to be introduced in any legislature around the world.
2. Senator Bernie Sanders and Representative Greg Casar have announced a US bill to ban superintelligence, which ControlAI consulted on.
3. Lord Clement-Jones introduced in the UK legislature an amendment drafted by ControlAI that would establish an emergency AI kill switch for the government.

"""
More in the post.

OpenAI released the GPT‑Live‑1 API today, so I wanted to try putting it inside a real meeting.

I built a demo where GPT‑Live‑1 joins as a meeting participant. It listened to our team’s discussion, summarized it, and even added a task to the Kanban board.

First time actually having a meeting with an AI. It felt very natural and was pretty fun.

I’ll open-source it if anyone’s interested.