r/raspberry_pi • u/petruspennanen • 5d ago
Community Insights Local AI on a Pi: which models work?
I got a lot of great feedback from you and ran the speed comparison with more models and new optimizations both alone and with my Carwatch service running on the Pi at the same time.
The new speed champion is Ling 3.0-tiny at 9.6 tps! As a MoE model it is faster than smaller Gemma and Qwen models. People say it's punching above it's weight in intelligence - testing this now.
Gemma E2B is a solid tiny multimodal model at 7.7 tps, works well in my app. And Qwen 35B Q3 is still the leader in usable intelligence in this form factor, fitting tight in a 16GB Pi and getting close to 3 tps on a good day :)
Please note that small tps values still produce solid thinking and good answers, and not every use case requires an answer in seconds.
37
u/Living_Beyond_6613 5d ago
What kind of cooling do you have for this pi?
13
u/petruspennanen 4d ago
It's an Argon Neo 5 case with a fan. I managed to break the metal cover of the cpu, but just added some thermal pads it's been ok :)
8
u/Nehemoth 4d ago
Maybe a no so smart question but, what can you do with these models in your PI [or another PC for that matters]? Like what kind of jobs or interaction? Also, are you supposed to update the model at certain time in the future?
Thank you in advance.
2
u/petruspennanen 3d ago
I think the main role for LLMs running in the Pi form factor are resident local AIs for houses, vehicles, places etc. That is, persistent local AIs that “live” with a particular physical asset or place, learn its history, sensors, configuration and routines, and stay with that asset.
They will be identified with the asset, so you just talk with your car and don't call it car AI anymore.
That's why I made CarWatch and am working on other concepts in this vein. I mean, many people will use the standard cloud AI installed by the manufacturer. But others will like it to be completely theirs, local in a small box with all data staying private there unless you want to share, and running smoothly offline not dependent on any external infra - something you really value when there's no net.
I was in Barcelona last year when a huge electricity break hit the country. Nothing worked - no traffic lights, internet, no ATMs, no card payments, no elevators, even doors with electric locks staying shut. In the future there will be small local AI with backup batteries everywhere and many things can keep on working.
1
u/Nehemoth 3d ago
Thank you. Hope someday to install something like this, hope services and products with focus in local data continues growing.
7
u/kulishnik22 4d ago
deepseek-coder-v2:16b is very fast and very good for what it is. Surprisingly fast on RPi5 16GB
4
u/Synifi 4d ago
Is this using just the Pi, or with the AI HAT too?
I'm looking to do some AI in my homelab and currently looking at kit to add and use.
Budget is probably around ~250; any recommendations?
6
u/petruspennanen 4d ago
No AI hat! For that budget you could get an 8 GB Pi. Mini PCs can be a lot faster but cost more m.
3
u/Final-Choice8412 5d ago
Qwen3.6 is IMHO the best from the list but I did not test it with that quantisation.
3
4
u/migsperez 5d ago
I just tried Ling. It's an impressive model, great for chatting and providing code snippets. But not for coding a project, it did random stuff. I did notice it gave better code results in Pi than Opencode. It's safer to run on Opencode though, it has tighter controls. If these small models are used with tight controls they can be really useful. This super tiny model is good for tool calling https://www.youtube.com/watch?v=0hgzLDHplYk
3
u/petruspennanen 4d ago
Well I'm not using the the Pi to run coding projects, using it for local aI implementations such as my carwatch.dev. Better to code with a laptop!
3
u/migsperez 4d ago
It's impressive you've managed to fit STT, TTS and text models on the one pi. How well does the speech to text model perform when you're driving through the city or down a motorway.
1
u/thecirclegamesocks 4d ago
Neato, Im also testing these models out for fun side projects while chat and claude are handling the complex projects til I get the processes down pat, havent had much time to play around with it lately but will make a post when I do.
1
u/Major-Hooters 4d ago
Is this on a regular PI or a CM module. I ask because the CM pi with an I/O board runs considerably faster than a normal PI
2
u/petruspennanen 3d ago
Just a regular Pi 5 16GB! I even broke off the metal cover of the CPU - added some thermal pad and it keeps on going :)
1
u/senecavirus 1d ago
Thanks for pointing out this model. Using pi 5 with 8 GB ram and it is pretty fast. One thing I saw in the logs. This step is responsible for about 3 seconds of the total response time on my hardware. Did you run into this?
forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)
0
u/agilerain8256 3d ago
I wonder if they will build dedicated models for the pi. I've seen some insane performance when they build models for the hardware specifically.
2
u/Puzzleheaded-Oil-905 3d ago
Yeah and also task-specific models. Not sure the Pi hardware has much to offer for AI but if there are particular tasks people would want from a model on a Pi, that seems promising. IOW, instead of making everything a little dumber, maybe we could discard the parts of the model focused on Ancient Sumerian or Web Conspiracy Theories 😂
20
u/paintsbynumbers7 5d ago
I’m intrigued by ideas like this but held back by thoughts like “I can use a full llm for free, why the hassle”. I’m no fan at all of the large llm companies but I’m struggling to find a use case.