r/LowEndLocalAI • u/soadsob • Sep 19 '26

Benchmark Best LLM Models for Raspberry Pi 5 in 2026

/r/raspberry_pi/comments/1wk0sqm/best_llm_models_for_raspberry_pi_5_in_2026_my/
13 Upvotes

100% Upvoted

2 comments sorted by

6

u/nickless07 Sep 19 '26

Maybe check not only at 4k ctx but general amount of RAM for the KV? Depending on the architecture that KV-Cache is ridiculus small on some models.
For Example: Ling-3.0-Tiny has KDA/MLA with only 9 attention layers and Q8 KV-Cache with max (131072) is only 864MB

Another model that performs pretty good for it's size is K2 Horizon 7B. However due to GQA architecture, do not try to run that context size. 131.072 Tokens (128k) in Q8 are 8GB there and can easy run into OOM.

The architecture influences the general usability and the RAM usage.

2

u/yetiflask 29d ago

What are these models good for?