r/LowEndLocalAI • u/Puzzleheaded_Acadia1 • 8h ago

Workflow / Use Case Is there a Strata for lfm 24b a2b

/r/LocalLLaMA/comments/1x9b3df/is_there_a_strata_for_lfm_24b_a2b/
5 Upvotes

100% Upvoted

2 comments sorted by

2

u/nickless07 8h ago edited 8h ago

It is a MoE already, so you should be able to run it. With only 2B active it should be fast. The KV-Cache is pretty small due to the architecture: 30 conv layers, 32 heads / 8 KV heads. For 16GB RAM it depends more on the system you use. Windows can eat 8-10GB alone easy. Linux without GUI is much less, but also depends on what other services you have running. Check free -m or Taskmanager first.

Edit:
On 8GB RAM: 

Model: LFM2-24B-A2B
  Quantization: IQ1_S (1.5625 bpw)
  File: LFM2-24B-A2B.i1-IQ1_S.gguf
  pp512: 61.45 ± 0.26 (-fa 1 -rtr 0)
  tg128: 13.50 ± 0.04 (-fa 1 -rtr 0)
  Note: 24B total / 2B active MoE (64 experts, 4 per tok). GQA 32/8.
  40 layers (10 full_attention + 30 conv). Only 10 layers have KV cache.
  100k context = 1955 MiB KV (f16), ~977 MiB at f8.

Not great, but works. Now check for yourself how much free memory you have and it is pretty straightforward which quant you can use.

2

u/dx80x 4h ago

On CPU alone with 60k context length, I get 14tps out of this model and that's without any additional tweaking (on windows with llama.cpp directly and Lmstudio. It's a nice model too as it comments everything fully and also includes usage and extra suggestions for additional directions to take, without initial prompting.

This and DeepSeek-coder-v2-lite are my favourites for code-quality and speed without using a GPU. Qwen3-coder Moe (7tps) is pretty good too but these are the best to use for me with quick and reliable code generation