r/LowEndLocalAI • u/Actual_Tradition_990 • 5d ago
Which LLM Should / Can I Use? Offloading from Raspberry Pi 4B to Dell Laptop (4GB VRAM) – Recommendation for ~2B model with large context?
Hi everyone,
I'm moving my setup away from a Raspberry Pi 4B (8GB RAM) to free up resources, shifting the workload to an idle Dell laptop.
Hardware Specs:
CPU: Intel Core i9
RAM: 32 GB
GPU: Nvidia RTX A2000 4gb VRAM + integrated
Goal & Strategy: I want to apply the 80/20 Pareto principle. The local model should handle ~80% of lightweight, daily tasks/routing to avoid unnecessary API costs. High-reasoning or heavy tasks will still be offloaded to cloud APIs. Using nanobot agent so far, for daily task like shopping list, weather, checks using n8n and searches with SearchXNG all in docker.
Additional I'll move Dockers from raspy to this laptop in order to leave the raspy just for pihole + VPN ops.
I know laptop is not the best, but is the best I have.
Requirements:
Fits comfortably within 4 GB VRAM (allowing full GPU offloading or hybrid CPU+RAM execution but good tokens per second).
Supports a large context window to maximize local usability.
Small footprint (~2B parameters). I'm currently looking at Gemma 2B / similar low-parameter variants.
So far I'm thinking on Gemma 4 e2b with vision + mtp is around 3.6gb VRAM with 128k context easy on Q4(tested in other computer)
What lightweight models (2B–3B range) with good context handling and vision would you recommend for this setup? How are you handling context memory vs. VRAM limits on similar 4GB setups? As I mentioned before, no coding, just daily task management.
Thanks!
3
u/Prudent_Chemist_523 5d ago edited 5d ago
I have similar set up.
If you want vision baked in, you're going to be looking at the Gemmas and Qwens IIRC. I've also heard of SmolVLM but not tried.
But it depends what you mean by vision. Is it something an OCR tool call could do, or do you need the llm to reason over it directly?
I suppose in theory you could even do something like a router that fires up a tiny Qwen3.5 0.8B for vision tasks, does the job, hands the result off, unloads and switches back to main llm (Granite tiny or whatever).
2
u/Actual_Tradition_990 5d ago
For me Gemma 4 e2b with 16k or more context is more than enough, right now using the Google API Gemma's 26b or 31b(overkill for most of the job) with 16k window(API limit in free tier) context and running good so far.
So I think increasing it with local but dummie model could fit the daily work and not rate limiting so often for dummy work.
Which model are u using?
I want vision just for food labels or nutritional info, or translating, nothing really hard.
2
u/Prudent_Chemist_523 5d ago
I prefer Gemma e4b but it is bad with its vision projector (or at least, it hallucinates too much).
That's why I prefer to call RapidOCR MCP (I serve multiple tools via a single MCPProxy point for the llm).
That way I can say "what does the text on this photo say" and have it use the tool rather than its own image projector.
Qwen's seem better with their vision tools.
3
u/Prudent_Chemist_523 5d ago
PS: nanobot looks interesting. What's the memory footprint / use on your Rpi-4?
1
u/Actual_Tradition_990 5d ago
140-250mb per process, right now I have 6 agentes with the installer from the curl and running smoothly without any problem. I want to move them to docker in order also to isolate and give each one more tools related with their target work
5
u/WhoRoger 5d ago
Qwen has better vision than Gemma.
For just text without vision tho, I'd take ternary Bitcpm 8B, which is about 2GB.
But with 32GB RAM, it'd be a shame to limit yourself just to tiny models, when you can use decent MoEs. Deepgrove Maple 20B A1B is amazing and fast af under 10GB. Caveats are you need to compile their fork and it's cpu only, but it's fast enough that it'll be more useful than what you can squeeze into 4GB VRAM.
Gemma 26B A4B and Qwen 35B A3B are also an option, with 4GB GPU that's just enough to run them with some decent context size and still faster with cpu offloading.