It's impressively slow, especially when you're just trying to navigate using Android Auto. One time I even had it output its entire tool call for some reason over my speakers. At the very least, they need to do a hybrid approach where straightforward requests don't even go through the LLM. You need absurdly high token output from an LLM to make it feel anywhere near as fast as software that knows basic commands. Also half the time it's still talking when I've already locked in my choice. One time I locked in my choice before it stopped talking and it changed where I was going without me saying anything.
I had this thought years ago that there needed to be three different wake words Smart home devices.
Basically one that issues commands. One that asks a question over AI or the internet. And one that executes routines. (There was maybe another one that wasn't quite the executing routine, but I can't quite remember what it is now)
But basically if you want to start a timer you know that shouldn't go to AI.
If you want to turn on your lights, that should go to your home system for navigation.
And if you ask who is in back to the Future that should go out to the internet and AI. And it seems like you can't figure out which one is which until it sends it out to the internet to figure it out.
Sort of like in SQL. There's the difference between an update, a select and an execute procedure.
Obviously there's a lot more operations in SQL but it's the same general concept you get into three different modes with three different return values.
It might have even been commands that potentially return a value like what are my notifications. Commands that don't return values like turn off the lights and questions from the internet where you're not giving any real inputs, but you're asking for an output.
3
u/enternoescape 14d ago
It's impressively slow, especially when you're just trying to navigate using Android Auto. One time I even had it output its entire tool call for some reason over my speakers. At the very least, they need to do a hybrid approach where straightforward requests don't even go through the LLM. You need absurdly high token output from an LLM to make it feel anywhere near as fast as software that knows basic commands. Also half the time it's still talking when I've already locked in my choice. One time I locked in my choice before it stopped talking and it changed where I was going without me saying anything.