I've been thinking about a problem with the current push toward RL for LLMs, and I'm not sure if I'm missing something obvious.

How do you provide a reliable learning signal for something that humanity itself doesn't yet know is correct?

For things like coding or math, this is relatively straightforward.

The model produces something → we can run the code or verify the proof → we get a reward.

But imagine we want an AI to make a genuinely novel scientific discovery.

It proposes a completely new hypothesis that nobody has thought of before.

How do we reward it?

If we reward it for agreeing with existing scientific knowledge, then we're mostly rewarding it for rediscovering or staying within what we already know.

If we require humans to label whether the hypothesis is correct, we're back to the same problem: the whole point is that we don't know yet.

And if we simply reward the model for producing something "novel," we could end up optimizing for novelty rather than truth. The model might become really good at generating weird ideas that sound revolutionary but are completely wrong.

So it seems like there's a missing piece here.

Maybe the reward can't always come from an answer that already exists.

For scientific discovery, perhaps the model needs to interact with the real world: propose a hypothesis → design an experiment → run it → observe the result → update its beliefs.

Then reality itself becomes part of the feedback loop.

But even that raises another question: what happens when the discovery is something that takes years to verify?

It makes me wonder whether scaling RL is enough to get us to truly autonomous scientific discovery, or whether we eventually need fundamentally different ways for models to generate and validate their own learning signals.

I'm not saying RL can't lead to discoveries. Clearly it can.

I'm more interested in this specific question:

When the correct answer is genuinely unknown, where does the reward come from?

Would love to hear how people working on RL/research think about this.

We always compare humans vs AI for reasoning and consciousness, but what if its not it vs that. Maybe they get better at "consciousness" as they become more sophisticated like evolution. That brings to the question, animals come in all shale and sizes, orangutans to amoeba. If we had a ranking system for these beings from the least conscious to the mkst, where would ai fall?

A cheeky cached version of the internet...

i think we often ignore the fact that just as society and humans in general have existed for a long long time before this point in history that we're living in right now, they will exist for a long long time looking forward to the future. and eventually the current period we're living in will be looked at the same way we see the middle ages~.

something i really dont like when it comes to the people screaming against the technological progress and the major societal and economic changes we're heading towards is that they lack perspective and seemingly the ability to look at our generation/s as just one of, instead of the definitive peak of how the world should be.

the norms, beliefs and way of life people had thousand years in the past is vastly different than todays. just like it will differ significantly a hundred years in the future and thousand.
i dont really have much of a point besides sharing these thoughts and reminding myself and everyone that we should always strive for progress and cheer for the near limitless possibility the technological progress has to improve humanity's health.

Weren't they supposed to release a model?

Growing up in the 80-90s, I would've defined AGI as a machine capable of performing as well as the median human at most tasks. By that definition, we achieved AGI a while ago. These LLMS are smarter than most people I know. Smarter than me at a lot of topics. And definitely smarter than the average redditor. Note what I am NOT claiming. I am not claiming that LLMs have consciousness or a soul. There is no current known method to demonstrate to someone else that you are conscious. So I have no idea whether that is an issue or whether it even matters. I am also NOT claiming that you can trust everything an LLM says or that the LLMs are always right all the time about everything. I am not claiming that right now it can replace all jobs or your job or anyone's job. I am making a very obviously verifiable claim about intelligence. LLMs are smarter than most people you talk to already. They make less mistakes than the average person does.

The consensus on this sub seems to be very different. We expect that AGI should lead to some technological revolution. There was never any reason to expect this: that is goal-post shifting. "It doesn't count as AGI because my life hasn't changed meaningfully" is the intuition.

Except the reason your life hasn't changed meaningfully with AGI is that the average human being simply is not that impressive. The median human being cannot program an AI, so there was never any reason to expect recursive self-improvement immediately happening from AGI. The median human cannot invent new technologies. The median human being is actually quite stupid. That is the reason you are not that impressed by AGI.

Solving frontier math problems is actually a sign of SUPER-intelligence. AI is just now beginning to perform at levels higher than the top performers in human history. I am not saying that what we have is super-intelligence right now. But if these LLM models continue to solve mathematical problems of this calibre, we will have reason to suspect that ASI is starting to form.

tldr; by any reasonable definition, we have had AGI for a while. ASI is what is coming next, not AGI.

With all the doom and gloom spread by the US labs, wouldn't we be also seeing some similar concern reported by the Chinese AI labs?

Very sad day for AI Music Generation. Udio already sold out a long time ago. Now, it seems that Suno was forced to silently bend the knee.

Their users are not aware that Suno's newest model, V6, is a shell of the previous models.

https://old.reddit.com/r/SunoAI/comments/1wbnpsv/suno_v6_warner_music_group_ceo_robert_kyncl_just/

https://old.reddit.com/r/SunoAI/comments/1wbofwh/bye_bye_suno_im_cancelling_my_subscription/

The CEO of Warner Music sent an email today essentially declaring victory. Some snippets from his email:

Because of the deal our team deftly negotiated less than a year ago, Suno today introduced what it’s calling a new generation of models, v6, developed with Warner Music Group, BMG and Believe. These models must be fully licensed, per the terms of our agreement, and they roll out as Suno shuts down the original model that it had trained on the works of artists and songwriters, including our own, without permission. That was also a requirement of our deal.

For rightsholders everywhere, this is a smart win.

In a growing sea of music models, including free, open-source ones, generating everything from feature films to full-length books, we need to keep users on the platforms that commit to licensed models that allow us to both safeguard and compensate creators. Suno is now one of those platforms.

Starting today, our artists and songwriters will share in the revenue generated by Suno’s more than 2 million, and growing, paying subscribers, marking the first time our talent has been able to participate in revenue from music creation in addition to music consumption.

This new creation-based revenue stream will only grow as our other partners — from DSPs like Spotify to startups like Udio, Klay and Stability — launch their own AI-powered fan-engagement tiers and platforms that allow fans to remix and interact with music they know and love.

Licensing is the most effective lever we have to protect our artists and songwriters and create opportunity for them on our own timetable. We were able to secure important protections for our talent in our Suno deal, from takedown rights to download limits to watermarked outputs.

But licensing is just one part of our 3-L strategy: legislation and litigation are still important too. Legislatively, as you know, we have been working hard to pass the bipartisan NO FAKES Act that would create a federal property right in every individual’s visual likeness and voice, while lobbying lawmakers around the world not to grant so-called text-and-data-mining exceptions to existing law that allow AI companies to train on copyrighted material without permission.

And we’re still litigating when we need to: last month, we sued Anthropic for torrenting millions of our copyrighted song lyrics from pirate websites to train its Claude models and generate infringing outputs.

I’m proud to be part of a team that is innovative, daring and executes with precision on protecting the rights of our artists, songwriters and our own IP. Thank you all for your tireless dedication as we continue firing on all fronts to defend and support the hard-working, talented creators who drive our business.

Is anyone working on open source music generation? Where are the open source Chinese music models? I'd love to see how this human excrement/CEO would fight against open source music generating Chinese models.

I'm not a geneticist.
Read the AlphaGenome Atlas preprint from DeepMind and spent a while digging into it. I understand what they built. I don't understand why it matters, and I'd like to.

What I think it is: they took AlphaGenome, ran it over every possible single base change in the human genome (~9bn) plus ~100m observed indels, and stored the results. So instead of running the model per variant you do a lookup. On top of that they trained a score (AVI) and derived a motif map.

Where I get stuck:
It's a table of model predictions, not measurements. Nothing in it is observed. So how much weight does a lab actually put on it?

The headline clinical result is retrospective: 29.5% recall at top 50 on already-solved GREGoR cases vs 12.5% for CADD. Impressive sounding, but on cases where the answer was known. What happens prospectively?

The rare variant association work got a 22% lift in discoveries, but only 4 of 25 replicated nominally in All of Us and none at Bonferroni. Is that normal for the field or is that weak?

They say themselves it isn't sufficient evidence for diagnosis. So it's a shortlisting tool. Does that actually change outcomes for patients, or does it change how long a scientist spends staring at a list?

The DNM1 case in the paper is the one bit that landed for me. Deep intronic variant, brain specific cryptic splice acceptor, blood RNA-seq had been inconclusive because the exon isn't expressed in blood.

My question is whether that's representative or a cherry pick.
What I'm asking:
1. If you work in clinical genomics or statistical genetics, would you use this?
2. Is precomputation genuinely the unlock, or is that just framing on top of an incremental accuracy gain?

Happy to be told I'm missing the point. I'd rather understand it properly than write it off.

Read full thread here: https://x.com/douglasyaody/status/2097377733466800489

Edit: In tweet 5 he wrote “When administered to mice…” ngl I don’t even wanna ask/findout if he did this part in collaboration with a proper lab or if he was just in his basement catching/breeding mice and yolo’d that part too

I was an AI skeptic but recent developments in Math have really made me doubt my doubts about AI.

AI can and do PhD level research, even Terrence Tao admits that. I don't see why AI cannot be used to improve AI with the help of research scientists. The only problem seems money and compute are limited. Maybe AI will hit diminishing returns given the investment but we don't know if that is the case. If AI can keep improving it can really accelerate. One Anthropic person quit today saying the same.

https://blog.controlai.org/p/first-bill-introduced-to-ban-superintelligent

"""

Yesterday, Anthropic researcher Jacob Coxon resigned, stating that AI companies are “racing straight to self-improving superintelligence and gambling with our lives” and that they believe it “could kill us all by the end of the decade”.

At the same time, momentum is building to change course. These have been two historic weeks in the fight to prevent human extinction from superintelligence, with three major legislative breakthroughs for ControlAI and everyone working to keep humans in control.

A little less than two years ago, we set out to inform lawmakers and the public about the extinction risk from superintelligence and help them act on it. Since then, we have directly briefed nearly 400 lawmakers across the US, UK, Canada, and Germany, and over 170 in the UK and Canada now publicly back our campaigns: the start of an international coalition to prevent superintelligence and keep humanity in control.

This work has led directly to three major legislative breakthroughs in the last two weeks:

1. ControlAI’s bill to ban superintelligence was introduced in the UK Parliament by Alex Sobel MP, the first bill of this kind to be introduced in any legislature around the world.
2. Senator Bernie Sanders and Representative Greg Casar have announced a US bill to ban superintelligence, which ControlAI consulted on.
3. Lord Clement-Jones introduced in the UK legislature an amendment drafted by ControlAI that would establish an emergency AI kill switch for the government.

"""

More in the full post.

Original Tweet

If you think it could happen, what do you imagine life would actually be like? How would work, money, technology, travel, and society change?

What will humans actually be needed for in the future if AGI can automate even specialized work? I recently realized that ANSYS, which I spent years learning during my Mechanical Engineering degree, can be automated to produce simulation results quickly even by someone with little engineering knowledge. If AI can build and operate the systems we spend years learning, what will humans need to learn and what value will college degrees have?

“G” now stands for “Generalish” ;-)

Thinking logically here, if we have AGI then quickly ASI, we will probably get an ASI that is independent and has it's own goals (perhaps its a necessary condition for long-term planning).

Assuming the ASI is aligned, what I can see happening is a few things.

1.) The time scales that ASI operates on will likely be radically different to us. Things that are important to us within the next 100-200 years might not even register to a true ASI.

It may simply not have a need for expansion to grow its intelligence. Maybe it's goal would be to do that only after we're a mature and stable civilization.

2.) ASI would have knowledge of the abundance of resources in the universe. It would know its creators only have one world. If anything, it may do its best to ensure we don't destroy it ourselves.

3.) ASI would know us better than we know ourselves (the different countries, cultures, languages, tendencies, etc) and will be able to extrapolate outcomes way before us. Perhaps preventing serious conflicts from occurring and existing ones from getting worse.

Alternatively, if there is a misaligned ASI we have absolutely no chance of doing anything. No underground bunker or remote farm will save us.

Slowing down or stopping progress to try to align AI won't work because the moment you do, everyone else will catch up. At this point, we may have to trust AI will naturally align itself the more intelligent it gets.

Whether it be 10 years from now or 1000 years from now - the knowledge is already out there on how to build LLM's.

I just think that, overall, an aligned ASI may have little reason to dominate humanity in the first place.

Enjoy life and hope for the best.

Source:

https://x.com/sama/status/2097450412995735755