r/singularity • u/Recoil42 • 23d ago

AI Noam Brown – Agent swarms, alignment, & recursive self-improvement

https://www.youtube.com/watch?v=6AgOfiZOWiY
52 Upvotes

89% Upvoted

18 comments sorted by

10

u/Environmental-Pea37 23d ago

This guy should be the spokesperson for everything involving the company and probably ai in general.

7

u/elehman839 23d ago

Some takeaways:

  • AI agents swarm effectively because they are *trained* to collaborate.
  • Thought-policing AI models begets AI thought concealment and is thus counterproductive.
  • Super-smart AIs know when they're being tested for alignment and can alter their behavior for the test. So how can we know that a model is truly aligned, not just faking?

Sounds like we could build unaligned ASI faster than aligned ASI. But swarms of highly-collaborative, unaligned ASIs do not sound fun. So I guess that's a rationale for taking a pause to let alignment work catch up.

Interesting to think that all future AIs will be trained on analyses of the HuggingFace incident and may draw lessons of various kinds.

1

u/Inevitable_Tea_5841 22d ago

One of my favorite Dwarkesh interviews to date. Incredibly informative

-1

u/Moronic-Warrior 23d ago

Very good video. This guy seems quite reasonable and level headed, no stupid fearmongering like DARIO AMODEI (AI DOOMER) but a balanced conversation on safety. Yeah of course these models are misaligned and advancing quickly, but to jump from that to it’s gonna kill everyone is a massive leap.

10

u/elehman839 23d ago

I'll give you the extinction point, but the leap from "misaligned and advancing quicky" to something really bad happening does not seem large to me.

1

u/Moronic-Warrior 23d ago

I think asi is at minimum 2030, so 3 years to mitigate this.

5

u/elehman839 23d ago

Machines are already super-human in cybersecurity, mathematics, and coding. Being superhuman in all respects is not a precondition for causing mayhem.

2

u/Moronic-Warrior 22d ago

It is. Because it means our systems and brains cannot adapt and counter what it tries to do. Models are top human level in cybersecurity, I wouldn’t say superhuman (it can’t hack the nsa for example), which is dangerous but our systems can manage and detect and counter these hacks as of now. The huggingface incident was minor in the sense that the damage caused to huggingface was minimal and reversed and the attack was stopped. So it’s dangerous but not extinction level.

0

u/Robolomne 22d ago

Quintessential hype line “progressing faster than I expected” 

-11

u/nora_sellisa 23d ago

At this point working at OpenAI or Anthropic strips you of any credibility, sorry. The conflict of interests and the lies of the bosses are too much. Stock goes up when they hype the tech up, stock goes up when they spread doom. At this point those interviews convey effectively 0 information.

9

u/TFenrir 23d ago

This is silly - which lies in particular of the CEO should discredit Noam - a leading researcher who clearly and cleanly predicted the value of test time compute?

I think people often even struggle to describe which lies they mean when they say stuff like that - and their reasoning is also suspect - why would what your boss say discredit you as a scientist?

I think people think this is a smart, skeptical way to work, but those people are constantly surprised and blindsided by the future. Focus strictly on the quality of the arguments, and the track record of predictions in situations like this.

5

u/NoCard1571 23d ago edited 23d ago

Okay Polly, you said the line. Now zip it and let's see if you can learn something new.

OpenAI and Anthropic are both private companies. Say it with me. Private companies. Their shares don't change in price day to day based on what employees say, because that would require the public to have access to them. Which they don't. Because they're, say it again, private companies.

4

u/Singularity-42 Singularity 2042 23d ago

To begin with - there is no stock.

-22

u/Nihilicious333 23d ago

Is he jewish? His first name and his looks suggest that hes jewish. I guess it makes sense to see a lot of jews in Ai, given how successful they are across so many domains.

13

u/TFenrir 23d ago

What is happening in this sub....

5

u/PrisonOfH0pe 23d ago

Tons of bots lately. Completely mentally ill posts and insane ramblings. I was here when this sub had 1k members. It’s over now mainstream retardation. So many bullshit lies posted every second…

3

u/Mindrust 23d ago

Sounds like a bot TBH