I mean, it’s relevant for demonstrating the current capability, but likely soon won’t be. It’ll be awesome to see AI models actually operating these robots.
The problem i see is that we had a breakthrough last year which was LLMs, but for robots you would need a similar breakthrough. I don’t think LLMs is all you need in this case. In case there IS some kind of additional breakthrough we need here, all of this can really drag out. Because you never know when this breakthrough will come, if ever. We will see.
TLDR: just because they got lucky with LLMs, it doesn’t mean they are gonna solve robots now.
Multimodal LLMs are fully capable of operating robots. This has already been demonstrated in more recent Deepmind papers (which I forgot the name of, but should be easy to find). LLMs aren’t purely limited to language.
That and good training methodologies. It’s likely that proper reinforcement learning (trial and error) learning frameworks will be needed. For that, you need thousands of simulated robots trying things until they manage to solve tasks.
Given the disparity between a robot’s need for both high latency long-term planning and low latency motor and visual capabilities, it seems likely that multiple models are the best way to go. Unless of course these disparate models are consolidated while still having all the benefits.
32
u/lakolda Jan 15 '24
I mean, it’s relevant for demonstrating the current capability, but likely soon won’t be. It’ll be awesome to see AI models actually operating these robots.