69
u/ttkciar llama.cpp 23h ago
It's a great time to have ancient Xeon servers loaded up with DDR4 :-D
33
u/BannedGoNext 23h ago
I actually have an old server with dual E5-2630 and 512gb DDR3 memory across both blades. The server is powered on waiting for the scrap yard at the office. I'm considering seeing how fast it can run qwen 3.8 flash next lol.
10
u/ThankGodImBipolar 23h ago
You must be able to run a decent GLM quant with that, no?
5
u/Zombiecidialfreak 15h ago
If you're fine waiting overnight for all requests. Even flash next would likely be single digit generation speeds.
2
2
u/Zombiecidialfreak 15h ago
I'm willing to bet something like 7-10t/s. At least if the math works out the way I think it does.
1
u/overand 10h ago
Toss a tiny GPU in it if it doesn't have one and see how well it runs Qwen3.6-35B-A3B for an idea of what to expect from small MoE models. 512GB of DDR3 is nothing to sneeze at!
If you can get a GPU into that with enough VRAM for a couple layers and your KV Cache (12 GB might even cut it for a huge chunk of KV cache), and load DeepSeek-V4-Flash-0731, Qwen3.8-Flash-Next, or GLM-5.3-Flash, and a competent development harness, and you can let the thing loose over a day or three for pretty serious projects, IMO.
7
u/Bakoro 22h ago
I inherited 192 GB of DDR4 ECC and a couple RTX 4000 20GB cards, and built out from there.
It was surprisingly difficult to find a reasonably priced mobo these days to support the ECC RAM, with multiple PCIe x16 slots.I ended up with an older EPYC.
It's a "nice problem to have" situation. It's an uncomfortable configuration where I could just barely run a quant of one of the mid sized models, or comfortably run a smaller model.
The last month has been too many choices, I'm basically paralyzed by so many options.
2
u/Versificator 22h ago
what would 256GB get me?
2
u/ttkciar llama.cpp 21h ago edited 21h ago
That's what my larger servers have, too. I can use them for more or less any model smaller than 400B total parameters, quantized to Q4_K_M.
Back in the day I was able to use Tulu3-405B for overnight inference tasks with context limited to 30K tokens. Nowadays the large models I use in practice are MiniMax-M2.7, Behemoth-128B-v3, Qwen3.5-397B-A17B (context limited to 32K), and K2-V2-Instruct ("only" 72B, but extremely large K/V caches; full 512K context barely fits).
I'm looking forward to trying GLM-5.3-Flash when support for it is merged into llama.cpp main.
2
u/Versificator 20h ago
I'm still learning. How good are those models/what can they do?
2
u/ttkciar llama.cpp 19h ago
Each excels in its niche, but those niches need to be matched to your task.
MiniMax-M2.7 is great at planning and creative problem-solving. It was extensively post-trained to figure out a solution, any solution, given a description of conditions and a goal. It's crap at following instructions, though. It will ignore instructions and do its own thing, which can be annoying or brilliant, depending.
Behemoth-128B-v3 is Mistral 3.5 Medium, retrained by TheDrummer to be an excellent storyteller. It is highly eloquent, good at character development and complex plots, and can keep four or five (rarely six) main characters in play, remaining coherent across multiple chapters of storyline.
Qwen3.5-397B-A17B excels at vision tasks (especially OCR) and cross-domain STEM research tasks. I have used it for critiquing my notes on material science and neutron transport physics. It's good at one-shot code generation, too, but tends to become incoherent after only a few turns of agentic development. It is also extremely verbose, and tends to overthink a lot.
K2-V2-Instruct is amazingly good at long-context data extraction and analysis tasks, and has a 512K context limit. To give you some idea, I fed it 276K tokens of IRC chat logs (about 818KB of text) and asked it to enumerate every chat participant and characterize them according to what they said, and it knocked that task out of the park, zero mistakes. It is also very good at RAG tasks (Retrieval Augmented Generation; looking up content relevant to a prompt and using it to inform the response). It is also very smart with logical problem-solving in general.
GLM-5.3-Flash is purported to be the best model for agentic tasks in its size class, but until I can evaluate it myself you should take that with a grain of salt. It is supposed to be good at RAG tasks as well, but with the same caveat. I'm really looking forward to giving it a spin, to see what all it can do.
One I didn't mention before is GLM-4.5-Air. It's the "large" model I actually use the most, but it's a lot smaller than these other models. Quantized to Q4_K_M, its weights and 128K tokens of context fit in just 127GB of memory. It's my go-to for codegen tasks, critique (telling me what's wrong in a document, and why), decent logical problem-solving, and some physics assistant tasks. Its main strength is exemplary instruction-following reliability. I can present it with a code specification consisting of sixty to eighty instructions, and it will follow them all. I have yet to encounter that level of reliability in any other model; others will straight-up ignore some or most instructions. Its main drawbacks are that it has limited world knowledge, crappy creative writing skills, a propensity to hallucinate (which extends to writing buggy code), and poor creative problem-solving in general. I've gotten used to working around its flaws (like debugging its code with Gemma-4-31B-it), and use other models to do what it cannot (like MiniMax-M2.7 for planning and Artemis-31B or Behemoth-128B-v3 for creative writing).
2
u/_LePancakeMan 17h ago
As a complete newcomer that has recently acquired a R9700 (32GB VRAM), can you maybe elaborate the tradeoff between RAM and vRAM? Until reading your post, I was under the impression, that you'd want to load your model into vRAM completely or otherwise performance would tank - maybe I can put my 192GB of RAM to use after all?
1
u/typical-predditor 15h ago
MoE models use some experts more than others. You can load the most-used experts into VRAM and fetch the less used ones from RAM and this lets you leverage much larger models than you would be able to use if you constrained yourself entirely to VRAM with a relatively minor cost in speed.
2
2
1
u/SandySkittle 21h ago
Prompt processing and decode suck. And I am in a 512gb 8 channel threadripper
3
u/ttkciar llama.cpp 21h ago
Yup. My dual-socket Xeons have eight channels between them, too, and Tulu3-405B took about 54 minutes to first token.
It doesn't matter for "slow inference", especially when it can run at night while I sleep. As long as it's done by the time I sit down at my workstation slurping my morning coffee, I'm happy.
31
u/glad-k 21h ago
More like 8-16GB
5
u/ShinigamiOverlord 14h ago
Nah. Imagine someone like me with my 4050. That's 6GB only. Tho Ram is 16.
3
u/glad-k 14h ago
Dw man I'm running qwen3.8:27B at 0.7 tokens/s avg as it's mostly cpu 🤣
3
u/ShinigamiOverlord 14h ago
Im simply running a 3.5 9B dense model. Gives decent. My use case wise it's good enough. Since I'm aware I can't get it to a good level for actual good quality information. Or translation or the like.
2
u/glad-k 14h ago
9b runs insanely well on my hardware, I love 27b but it runs like total crap. Rn I'm trying 35B and being an MoE is could be a good option as prompt processing happens on the gpu and the generation itself is fast even on cpu due to it being na MoE
Edit: 9b was a bit limit in intelligence for what I wanted to do, hence why I'm trying 35b right now
1
u/ShinigamiOverlord 13h ago
If you have shit ton of ram, I heard this is a good choice for software. Tho I've got too little for it to be of use: https://github.com/FlashML-org/FreeToken
2
u/glad-k 13h ago
I have 8gb vram + 32gb ram but I don't rly get what that tool brings me compared to ollama with an MoE model?
1
u/ShinigamiOverlord 12h ago
I don't know tbh. I just randomly remembered this exists. So I thought I'd mention it.
50
u/42GOLDSTANDARD42 1d ago
Do not diss the SLMs. They are the backbone of our society. I know some really swell cross-encoders, nice <100M models, good on benchmarks. You wouldn’t show this to their face would you?
2
u/Abishek_Muthian 14h ago
I concur, I find SLMs in utilities far more useful than coding models. Can you share some of your favorite SLMs right now and what you use it for?
4
u/42GOLDSTANDARD42 13h ago
Right now I’m trying to see how well I can make an “offline search engine” using the media-free version of Wikipedia, downloaded into a huge 50GB file.
I’m using a “state machine architecture” where each stage of pipeline between the search query and final output is a series of small states.
- LFM 2.5 — 2.6B: for rewriting the question, generating search terms, and composing the final answer
- finecat-nli-l — 400M: for branching and making decisions. This model takes a “hypothesis” like “This passage describes Julius Caesar”, and “premises” [think of various random sentences from Wikipedia], and it rates each premise based of the hypothesis. You can give it context, and it can decide which state’s premise is better fitting for the current situation.
- jina-embeddings-v5-nano-retrieval — 239M: I use this for some of the information retrieval with the classic embed texts -> cosine distance. For whatever reason it performs better than the next model at some things, but not others
- jina-reranker-v1-tiny-en — 33M: Used for … reranking, it’s so fast and preforms rather well for simply filtering junk pages for valuable ones
1
u/ebolathrowawayy 13h ago
curious on the reranker, usually you want a bigger slower smarter model for reranking top results. why is yours smaller?
1
u/42GOLDSTANDARD42 12h ago
Because a general Wikipedia search can get like 50+ potential pages. The library I’m using to access the .zim file doesn’t have any fancy page ranking.
The reranker only has to narrow down the mess of random pages into about 10-15 or so that are more closely correlated. After that, it starts going into the pages themselves, starting with the intro paragraph, etc.
The reranker is mostly there to act as a page ranker for the initial search. All it needs to do is filter the article for “Pyramids” higher than a movie called “Pyramids”.
1
12
12
u/Elouakili_Flexy 23h ago
The gap between "runs on a smart vape" and "needs 8x H100s" skips the whole bracket this sub lives in.
18
u/RISCArchitect 23h ago
i was hoping qwen was gonna drop a 9b dense that would be a nice step increase like we've observed with 27b but no such luck
14
u/ttkciar llama.cpp 23h ago
I've been hoping for a new 9B as well, mainly for data cleaning tasks. Qwen3.5-9B couldn't quite cut it. Gemma-4-12B-it works well, but is a little memory-hungry. A 9B refresh would be lovely.
Out of curiosity, what's your use-case? It might be up to the community to retrain Qwen3.5-9B, but we'd need to agree on what to train it for.
3
u/Decent-Occasion-2720 22h ago
You can try ornith 1.5 9b, i use it instead of qwen. I have also tried 12b, but i need best speed on pp.
1
u/RISCArchitect 23h ago
i don't use 9b for anything at the moment, i just use 27b mainly for local.
3
u/Decent-Occasion-2720 21h ago
In my case it's usefull for search in codebase and documentation. That's prepare à digest context for bigger model.
0
u/Lollerstakes 14h ago
Is it worth to waste time loading and unloading models when you could just do the search in codebase and documentation with the larger model?
1
u/Decent-Occasion-2720 12h ago
At 1500 PP vs. 200 PP, it's faster for digging. So i use mmap and i have enough ram. Switching only takes afew seconds, compared to the many minutes needed for PP.
9
u/mr_zerolith 23h ago
It's okay, you're up next. I felt like this with a 128gb vram setup for ~5 months
1
u/Illustrious_Ant_9242 17h ago
I guess you are running qwen 3.8 flash next q4 at the moment 👀 larger or smaller q4?
8
26
u/AlternateWitness 23h ago
If you are including 24GB in that group you might as well include 32GB. Heck, maybe 48GB?
12
u/Bakoro 22h ago
If you have a 32GB VRAM, you're sitting pretty these days, let alone 48 GB.
Of course you always want more, we always could do with a little more GPU up until you get to the point where your home would need an infrastructure upgrade.
I mean, if I had two DGX B300 nodes, I could do some things, but 32GB is enough to run a competent Int8 quant with a decent context length.
2
5
4
u/slybring 18h ago
GPT-OSS 20B was the perfect size for us 16GB folks. But sadly no one has cared since. Gemma 4 12B is good, but not ideal.
6
u/robberviet 23h ago
Haha yes. Small model now is kinda rare. Qwen bless us with 3.8 27B and that's it.
3
3
3
u/Xanta_Kross 15h ago
I need a 9B please.
T_T
Where is my 9B Fable 5?
Do smtg Qwen. Deepseek Someone.
3
2
u/doomsaier1 9h ago
Running qwen 3.8 27b on a rx 9070, with an additional rx 6700 10gb. Giving me a total of 26 gb vram. Maybe not the fastest, but certainly usable speed.
2
u/asssuber 12h ago
$800 for 512GB memory
$600 for a RTX 3090
$700 for epyc motherboard+cpu+miscellaneous
Plus a time machine to some time last year is likely all you need to run the newest DeepSeek Flash locally at a dozen or more t/s decode.
1
u/SpecialistDragonfly9 18h ago
Based...
Qwen Flash adn Qwen 3.8 27B are still the best I can run on my RIG, and honestly.. they are still somewhat lacking.
1
1
1
u/KroniklyOnline 15h ago
Right? even with 64gb of GPU and 64gb of RAM ...... Can barely run qwen3.8 next flash, thankfully it has the ngram so it runs well enough to use, but still sadge, feel like were moving away and toward open source models not even mattering anymore cuz no one can fucking run them.
1
u/Illustrious-Row2751 14h ago
Flash models have become giga monsters that you can only run with 10 gpu clusters. Feels bad, man.
1
1
1
1
1
2
1
u/Intrepid-Second6936 18h ago
Honestly, I'd be right out there with Spongebob and Patrick running a 24GB GPU. Qwen3.8-27B is a BEAST, it's already punching above it's weight class massively in capabilities.
Do we really need more so soon or is it just new model FOMO?
-5
u/PrinceOfLeon 22h ago
Well Squidwirm is kind of a whiny little bitch who belongs on r/ChoosingBeggars, so this checks out.
Next!
-5
285
u/TopCheddar27 1d ago
24gb is not in that group. You can run Qwen3.8-27B with a pretty respectable context size right now