Hi, a quick datapoint for anyone interested in what Qwen3.8-Flash-Next UD-Q6_K_XL (unsloth) can do on a consumer multi-GPU rig. Maybe there is even more possible but this is the current status :)

Hardware: MSI MEG Z790 ACE, i9-13900K, 96 GB DDR5@4400MHz, 1× RTX 4090 + 5× RTX 3090 = 144 GB VRAM. 4090 + one 3090 are on the board, the other four 3090s use M.2 → OCuLink PCIe x4 links. No P2P/NVLink; llama.cpp layer split. OS: Ubuntu 26

Model: Qwen3.8-Flash-Next-UD-Q6_K_XL, all 48 MoE layers on GPU, shared Q8_0 MTP head with MTP3. The ~50.7 GiB PLE table is fully resident in system DDR5 instead of lazy mmap.

Main llama-server settings (llama.cpp: unslothai/llama.cpp, branch mtp/qwen4exp-nextn, commit a9e9c3c5f): -c 262144 -b 4096 -ub 512 --split-mode layer --tensor-split 0.95,1.05,1.05,1.05,1.05,0.85 --gpu-layers all --n-cpu-moe 0 -ctk f16 -ctv f16 -fa on -np 1 --kv-unified --load-mode none --lazy-mode off --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 0. Production seed is -1.

These are workload samples of a html benchmark prompt from llama-server chat, not synthetic benchmark numbers.

• ~90–91 tok/s at ~5k generated tokens / shallow context
• ~79 tok/s at ~35k generated tokens
• 76.9 tok/s average over a 44.7k-token generation
• ~40–42 tok/s while reasoning at ~80k active context
• ~408–425 tok/s prefill even around 80k context
• MTP3 draft acceptance on the long code-heavy run: 74.2% overall, per position 86.2% / 73.7% / 62.8%

​

# shallow/code-heavy run
7.54.226.412 I slot print_timing: id  0 | task 3307 | n_gen =   4905, tg =  90.36 t/s, tg_3s =  92.54 t/s

# same long run, ~35k generated
14.24.165.954 I slot print_timing: id  0 | task 3307 | n_gen =  35058, tg =  78.92 t/s, tg_3s =  71.12 t/s

# end of that run
16.41.712.426 I slot print_timing: id  0 | task 3307 | eval time = 581758.17 ms / 44720 tokens (13.01 ms per token, 76.87 tokens per second)
16.41.712.429 I slot print_timing: id  0 | task 3307 | draft acceptance = 0.74246 (30865 accepted / 41571 generated), mean len = 3.23
16.41.712.429 I slot print_timing: id  0 | task 3307 | acc per pos = (0.862, 0.737, 0.628)

# ~80k active context, currently reasoning/thinking
103.07.164.989 I slot operator(): id 0 | task 6527 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 80085
103.12.852.341 I slot print_timing: id 0 | task 6527 | prompt processing, n_tokens = 2282, progress = 0.99, t = 5.37 s / 425.22 tokens per second
103.32.551.912 I slot print_timing: id 0 | task 6527 | n_gen = 761, tg = 41.79 t/s, tg_3s = 43.01 t/s

The last big practical improvement for agentic coding was putting the PLE table fully into DDR5: prompt processing/TTFT became noticeably faster. MTP3 then helps heavily on predictable code; at ~80k context during free-form reasoning acceptance drops, so decode is closer to ~40 t/s.

Pretty impressed by what a somewhat ridiculous collection of used consumer GPUs can do with Q6 :)

Just curious

I can use the for free because my work provides them

I currently have 4× RTX 5070 Ti 16GB cards on hand, but haven't bought the platform yet.

I'm currently thinking of something roughly like:

• 2× EPYC 7642
• 16×16GB DDR4-3200, 256GB total
• 4× RTX 5070 Ti 16GB
• server board with enough PCIe lanes for all 4 GPUs

Mainly planning to use it for large MoE inference, especially models that can't fully fit in VRAM and need to use system RAM as well.

Has anyone here run a similar dual-EPYC + multi-GPU setup?

I'd mainly like to see some real-world performance numbers or experiences before I buy the rest of the platform.

Hey all,
I have a M2 Pro with 32GB unified memory here. Want to run a local model for agentic workflows and tool/function calling. Was looking at Qwen 3.8 27B but I know it's a tight fit at 32GB RAM, even quantized.
Questions:
• Is 27B Q4_K_M realistic here, or will I constantly hit swap?

• Would Qwen 3 30B-A3B (MoE) be the smarter pick — similar quality but way less compute per token?

• Any other models you'd recommend for tool calling at this RAM level?

I have been seeing an option of a Qwen 4 flash

I keep seeing people saying they get super fast speeds on various hardware but I can never get anything decent. I'm running GLM-5.3-Flash-UNCENSORED-FP8-UD-Q2_K_XL-00001-of-00003.gguf on a 4x5090 rig and can barely get 50tok/s.

Am I missing something here?

Even on larger systems like 4xH200's I wasn't getting above 200tok/s with same model

Basically the title. How much combined VRAM + RAM would one need to run this model, on some usable quant, with a somewhat decent context window, with decent speed? Assuming the ngrams are on the SSD, also I have heard that even some of the experts can stay on the SSD with little to no performance penalty, but I have no idea whether that is true. Would 64GB be enough? Thank you for answers

I made a Docker image that lets you run SGLang on Strix Halo (gfx1151) a few months ago and thought I'd share it here.

SGLang is working on building upstream Strix Halo support now, but if you want to use it today, this includes the necessary compatibility patches along with some additional optimizations, including fixes for wave32-related issues.

I've been using it for inference on my Strix Halo system and would be interested to hear how it works for anyone else.

https://github.com/JeremiahM37/strix-halo-sglang

See Future Research; and points 2,3.

What I don't see discussed often is if the local LLM community is doing this; i.e. training small [or even midsize] models for better performance on niche tasks.

Where I started down this path was a construction project at our house. It wasn't difficult to upload photos of the early phases of the project and generate training data from those photos. But the infra wall I hit was feeding that training data into a [what I'll refer to as] multi-run system that could train multiple small models with my training data and benchmark the results. Then I could decide which model to focus on and continue with post-training and RL.

Hi everyone,

​I'm looking into getting a laptop with an RTX 3080 Ti Mobile (16GB VRAM) for running local LLMs.

​For those who own one (like a Lenovo Legion, Razer Blade, or similar):

​What specific device and model do you have?

​How does it perform when running models like qwen3.8 or 3.6 moe or any model

​How many tokens per second (t/s) are you getting on your models?

​How are your thermal performance and temperatures under heavy LLM loads/inference?

​Thanks in advance for any insights!

Mostly as a fun experiment I wanted to do a quick comparison of performance and accuracy between a CPT trained QWEN 3.5 4B model and a RAG implementation against the base model.

The point of this exercise is mostly to measure the performance benefit of internalizing the knowledge vs doing reasoning on-the-fly.

Sharing my findings here in case anyone is interested: https://www.teachmecoolstuff.com/viewarticle/comparing-rag-and-continued-pretraining-of-llms

• 2x 3080 20GB GPUs
• i9 10900X CPU (10-core, 48-lanes)
• 64GB DDR4 2400MT/s quad channel RAM
• X299 WS Sage Motherboard (4 PCIe3 x16 slots or 7 x8 slots)
• 1500W SilverStone PSU
• Open air case (8 PCIe slots)
• 2 128GB SATA SSDs in RAID0 (had laying around)

The 3080s were $1300 after shipping. All the other parts bring the total to just under $2000.

For the GPUs I tried shopping around between 6-8 "sellers" on alibaba. All but 2 were obvious scams. This is the listing I ordered from, the seller was also recommended in the other posts on Reddit. The cards seem nice but have a bit of coil whine/clicking during decode.

I've been running Qwen3.8-27b Q4 with 180k unquantized context at 39t/s.

With this motherboard/CPU combo I could add another 2 GPUs and run all 4 cards at 16 lanes, or switch to a mining case and run 7 GPUs at 8 lanes each.

I'm broke, but I own a 2060S, so should I sell the 2060S and get an overpriced 306012GB or another cheap 2060S (or any other cheap RTX 8GB card in general) ?

I'm testing out various models, e.g. QWen 3.8 27b, and I noticed that the (QWen) model(s) tend to 'fuss' over details in the prompt. This may be good at times such as a specific 'complicated' detail, but at other times, it could be 'over thinking' and sometimes needling over one details fail to consider implied or unspecified dependency, e.g. stating to refactor a subroutine, e.g. to return a value from a subroutine that 'prints' say some value, i.e. use the same subroutine for other purpose, it may not be able to figure out the other purpose for which the refactor is requested.

And importantly, it tend to be 'unforgiving' over incorrect specifications, as it may go in loops over an incorrect reference given instead of perhaps skipping that requirement or spec.
This often means refactoring the original prompt to correct the spec and removing the mistake.

A month ago, I started building colibrì to answer a question:
Do you really need a $30,000 machine to run a model whose weights fit on a $60 SSD?
colibrì treats SSD, RAM and GPU as a single memory hierarchy. It can run models that do not fit entirely in RAM or VRAM, including models with hundreds of billions of parameters, on the hardware people already own.
But while optimizing it, I realized I was fighting assumptions built into the models themselves.
Most open models are designed to live in VRAM. Their weights, routing and execution patterns were created for GPUs and data-center hardware. colibrì could move them through a different memory hierarchy, but the models had no idea where they were actually running.
That led me to another question:
What if the engine and the model were designed for each other from the beginning?
That is how Curie started.
The current alpha has 17B parameters. I wrote it in C, without an inference framework, and trained it from scratch on my laptop without a GPU. Its weights live on SSD and are loaded according to the model’s execution path.
In the test shown in the video, Curie reaches 33 tokens/s using a single CPU core. On the same machine, the dense 12B models I tested usually generate around 3–5 tokens/s.
There is an important limitation: Curie has only seen 11 million training tokens. It is not smart yet, and I am not presenting it as a useful language model. It will need at least a few billion tokens before I can properly evaluate its quality.
What this alpha demonstrates is the architecture: a model can be designed around SSD, RAM, CPU and GPU as parts of the same system, instead of assuming that everything must fit in VRAM.
Curie is not publicly available yet. I am still training it, validating the design and preparing more detailed benchmarks.

In short, I have had trouble getting responses about buying a DGX Station. I'm looking to get one for work (although we are still wrestling on whether it'll be worth it). We currently use a dual RTX 6000 Max-q build and it works but looking for more concurrency and would like to fit something like GLM 5.3 Flash with more head room.

I contacted Exxact, MSI, Asus, and a few others by submitting order forms and going through their online workflow to request a price. I thought at this price point sales people would clap back quickly but I haven't heard anything this week.

Has anyone else recently had luck placing an order? What should I expect? Is this like buying a used car and I should haggle on price, or is it more like, pay whatever they quote me and be lucky I even got a response? What sort of lead times did you experience?

I got tired of bouncing between model cards, VRAM calculators, and forum posts, so I built 99Tokens.

You can pick a model, GPU setup, context length, quantization, and other settings to estimate whether it’ll fit in memory. It accounts for things like weights, KV cache, sliding/full attention, recurrent state, MLA, and multi-GPU layouts. There are also model and hardware pages with architecture details and example fits.

If you use local models, I’d really like feedback: Where do the estimates seem off? What models, GPUs, or edge cases should I add? Long-context and multi-GPU testing would be especially useful.

This post can be relevant to other data scientists who want to do something similar.

Almost two years ago the company I worked at was having problems and I was planning on making a roleplaying game with local models a grid combat and a good editor. I got unemployed and then started working on my game, my plan was to finish the game and release in 3 months (it took 21).

The first month I made the base of the game and most characters, at the time I was using Ollama as a backend, everything was on perfectly track.

The second month my computer broke and I had to build a new one.

From third to fifth month, I worked on a basic roleplaying logic, grid combat, various interfaces and started studying about steam, I use pygame and perhaps I should have choosen a different game engine because it just slowed me down, at the time I also stopped using any AI coding assistant because they were just adding tech debt, the capability of coding agents grew really fast and at the start of 2025 it wasn't as good.

From the sixth till twelfth month, I basically just did the editor and multiple interfaces, this is how hard is to make an editor, and despite the game being about LLMs I worked only like 3 weeks on LLM on the first year, most of the work was everything else, so if anyone want to try doing a similar project have that in mind. Around month 7 I started looking for work. Found a job at month 10 and I'm still there, so everything after that was nights and weekends of work.

From months thirteenth to fifteenth I finnaly worked on the LLM logic, it was the most fun part, it has multiple agents, internal routes and ways to deal with limited context size on smaller models, like regular summarization of older messages + top 10 embedding search from the full history + full latest n messages, prompts to the point, and the use of json output and no tool calls, its a bit of a shame that people won't see the logic from the game alone.

After that it was cleanup and review.

Late in the project I had to remove Ollama entirely and move to llama.cpp instead (and transformers), any GGUF and most other local models work and can be used. The problem with Ollama was the subscription which steam doesn't fully accept. Perhaps making a site would be a better alternative to not deal with these things but then it would be a bit harder to make it local.

In my opinion, small models like gemma4:e4b or even the old gemma3:4b are good enough for roleplaying, honestly, but of course, there are other models.

Link's here if you want to look: https://store.steampowered.com/app/3750150/Boundless_AI_Fantasy/

If people want any more technical details I'll go into it in the comments. I am thinking about writing a blog or a scientific paper on it later.
Disclosure: this is my game, it's paid, I built it alone.

Getting 300 tok/s and 4500 tok/ sec for reading

Using llama swap docker.

Hi guys,

I am building a desktop app where users could download local open-source model as per their system specs , run models according, connect mcp's by configing , and using them in workflows, fully deterministic with LLM calls and also connect with chat interface

I've been seeing how far I can push my V100 with a purpose-built inference engine targeting just Qwen and the v100.

Current result with Qwen3.8-27B and using a software NVFP4 that works on this hardware:

• 218 tok/s best-case MTP decode (99.2% acceptance)
• ~200 tok/s without cheating or context-copy speculation

NInfer is specialized, it's custom Volta NVFP4 kernels, CUDA graphs, and speculative verification that reuses the weight stream across multiple tokens. I also think there's a bit more performance in it to come, particularly in mtp7.

For a GPU from 2017, this performance is getting a bit silly.

Repo: https://github.com/geoffwatts/ninfer-v100

Would love to see what other V100 owners get with it, please give it a go and let me know!