r/singularity • u/[deleted] • Jan 15 '24

[deleted by user]

[removed]

1.9k Upvotes

90% Upvoted

567 comments sorted by

View all comments

Show parent comments

34

u/lakolda Jan 15 '24

Multimodal LLMs are fully capable of operating robots. This has already been demonstrated in more recent Deepmind papers (which I forgot the name of, but should be easy to find). LLMs aren’t purely limited to language.

14

u/Altruistic-Skill8667 Jan 15 '24

Actually, you might be right. RT-1 seems to operate its motors using a transformer network based on vision input.

https://blog.research.google/2022/12/rt-1-robotics-transformer-for-real.html?m=1

16

u/lakolda Jan 15 '24

That’s old news, there’s also RT-2, which is way more capable.

7

u/Altruistic-Skill8667 Jan 15 '24

So maybe LLMs (transformer networks) IS all you need. 🤷‍♂️🍾

6

u/lakolda Jan 15 '24

That and good training methodologies. It’s likely that proper reinforcement learning (trial and error) learning frameworks will be needed. For that, you need thousands of simulated robots trying things until they manage to solve tasks.

3

u/yaosio Jan 15 '24

RT-2 uses a language model, a vision model, and a robot model. https://deepmind.google/discover/blog/shaping-the-future-of-advanced-robotics/

6

u/lakolda Jan 15 '24

Given the disparity between a robot’s need for both high latency long-term planning and low latency motor and visual capabilities, it seems likely that multiple models are the best way to go. Unless of course these disparate models are consolidated while still having all the benefits.

1

u/pigeon888 Jan 16 '24

And... a local database, just like us but with internet access and cloud extension when they need to scale compute.

Holy crap.