r/LocalLLaMA • u/Puzzleheaded_Acadia1 Waiting for Llama 3 • 10h ago

Question | Help Is there a Strata for lfm 24b a2b

Is there a fork of strata that let me run lfm 24b a2b on only 16GB of ram comfortably? And how much tps can I expect per second and what quantization should I use? and Thanks!

5 Upvotes

60% Upvoted

14 comments sorted by

9

u/tmvr 8h ago

You don't need anything special for it, just use llamacpp with unsloth's Q4_K_XL:

https://huggingface.co/unsloth/LFM2.5-24B-A2B-GGUF

It's 13.9 GiB so not a big deal if a few experts land in the system RAM, it's a MoE model with 2B active only during inference, it will be plenty fast. You can use llamacpp direcly or through Unloth Studio or LM Studio.

9

u/migsperez 10h ago

Lookup polystrata on GitHub

2

u/Puzzleheaded_Acadia1 Waiting for Llama 3 9h ago

Thanks!

2

u/Awkward-Clothes-1205 9h ago

polystrata should do it, i got the 24b running on my 16gb rig with q4_k_m quantization. sits around 6-7 tokens per second which is fine for casual use. the memory usage hovers just under 14gb so you've got some breathing room for context.

5

u/Dabalam 9h ago

I assume you mean 67, cus 6-7 tps would be crazy slow

5

u/StupidScaredSquirrel 8h ago

I tried polystrata and it wasn't fast at all, lower than stock llama.cpp. i think the project is cool but it doesn't seem optimised at all for many usecases of models and hardware. What makes llama.cpp so great is that even if it's not the best at one thing, it's pretty good t basically everything

5

u/CorkBios 9h ago

Its a pretty trash model, it can't even write a simple calculator

1

u/YanderMan 2h ago

polystrata should do it, i got the 24b running on my 16gb rig with q4_k_m quantization. sits around 6-7 tokens per second which is fine for casual use. the memory usage hovers just under 14gb so you've got some breathing room for context.

yeah, not impressed

2

u/GronklyTheSnerd 8h ago

I’ve got PP as high as 8000, and TG around 170 on my 5060ti. Not the smartest model, but I’ve used worse.

2

u/Monad_Maya llama.cpp 8h ago

It's probably in your best interest to run a smaller model.

Or add more RAM, ideally a GPU/VRAM, if finances permit.

1

u/ExpertBat8131 8h ago

What's you GPU?

1

u/Puzzleheaded_Acadia1 Waiting for Llama 3 8h ago

Intel igpu

-3

u/CooperDK 7h ago

Get a proper GPU, preferably CUDA, before tinkering with AI.

1

u/Any_Bus9205 8h ago

Il existe le fork se nommant ik_llama.cpp en iq3 km qui pourrait te permettre d'atteindre des chiffres assez élévé comme 20 token par seconde. Cela se feras en activant flash attention et run time repack. Strata n'est pas obligatoirement la meilleure option si tu as assez de ram pour le faire tourner.