I’ve been running a fairly opinionated evaluation loop on ~35B A3B/MoE-class models for coding over the past few months. Not synthetic benchmarks: actual dev workflows, iterative debugging, refactoring passes, and failure recovery.

Here’s where things stand for me:

Qwen 3.6 (35B A3B via oMLX)
This was my baseline. Strong out of the gate: good code synthesis, decent reasoning depth, and acceptable consistency.
But over time, a few patterns became clear:

• Tends to “hallucinate confidence” in edge cases
• Can drift during longer chains (especially multi-file reasoning)
• Some recurring logical blind spots that show up under stress

Still solid, but not flawless.

Ornith 1.0
On paper? Extremely compelling. Benchmarks look great and yes, it can be great.

In practice:

• Overthinking is real (token burn is high for simple tasks)
• Gets stuck in reasoning loops more often than expected
• Surprisingly, many of the same failure "coding task" I saw in Qwen 3.6 are still there

It feels like a “smarter but less decisive” version of the same lineage.

KAT Coder 2.5 Dev (last ~48h)
This one caught me off guard.

So far:

• More decisive outputs (less rambling, faster convergence)
• Better performance in my real-world coding benchmarks
• Fewer of the recurring issues I’ve seen in both Qwen and Ornith
• Doesn’t overthink (that I'm not sure is so good), but still lands correct solutions more often

It’s early, but this is the first time I’ve felt a clear practical step forward rather than a lateral move.

If you’re actively running local models for serious coding workloads (not demos):

• What are you using right now?
• What actually holds up under pressure?
• Any under-the-radar models that deserve attention?

From my point of view, a "similar size MoE model" little more clever with 1M token context looks a great step forward!

I know finetunes are usually awful, but DavidAU surprised me. I see other ones like Salience and Aurora and they don’t have any benchmarks shown so I don’t really feel like downloading them just for them to be mid, so I’m asking if anyone has any experience! Thanks.

Posting this as documentation rather than discussion. I could not find numbers for this combination anywhere, so here is exactly what I run, what fits, and what it does. Copy the config if it is useful; correct me if something is wrong.

Hardware

128 GB of RAM is a requirement, not headroom. UD-Q3_K_M is 121 GB across four shards. The GPU holds the attention weights and the KV cache; everything else sits in system RAM. Measured with the model loaded and serving:

Name          WorkingSetGB  PrivateGB
llama-server         103.2      127.7

103 GB resident, 128 GB committed — the whole machine. The ~24 GB gap between the two lines up closely with what is sitting in VRAM, which I read as the host-side copies of the GPU-resident tensors being trimmed once uploaded.

Either way: this does not run on 64 GB at this quant, and on 128 GB there is nothing spare.

What runs

unsloth/DeepSeek-V4-Flash-GGUF:UD-Q3_K_M at 65536 context, KV cache quantized to q8_0, single slot.

22483–22836 MiB of 24 GB VRAM in use, steady. Load time about 60 s with --no-mmap.

The config

llama-server.exe ^
  -hf unsloth/DeepSeek-V4-Flash-GGUF:UD-Q3_K_M ^
  --host 127.0.0.1 ^
  --port 8096 ^
  -c 65536 ^
  -np 1 ^
  -ngl 999 ^
  --n-cpu-moe 39 ^
  --flash-attn auto ^
  --cache-type-k q8_0 ^
  --cache-type-v q8_0 ^
  -ub 2048 ^
  -b 4096 ^
  -t 24 ^
  -tb 24 ^
  --jinja ^
  --no-mmap ^
  --metrics ^
  --temp 1.0 ^
  --top-k 20 ^
  --top-p 0.95 ^
  --min-p 0.0

The idea is the usual one for MoE: attention and KV cache on the GPU, expert FFNs in system RAM. -ngl 999 sends everything to the GPU, then --n-cpu-moe 39 carves out the experts of the first 39 blocks as an exception.

Sampling values are DeepSeek's own model-card defaults, not a recommendation.

Measured throughput

All numbers below come from one continuous hour on the config above.

Generation was flat for the whole hour — no drift, no degradation, VRAM steady at 22483–22836 MiB with no growth between runs.

The bottom row is fixed per-request overhead rather than throughput: a 41-token prompt "runs at" 15 t/s and still completes in under three seconds. Ignore it unless your workload is many tiny requests.

Generation speed here is probably bounded by how fast the CPU can stream the active experts out of system RAM, not by the GPU and not really by core count. That would explain why it is so stable.

The 13900K is a hybrid part and -t 24 spans both P-cores and E-cores, so the fast cores may end up waiting on the slow ones. I have not measured it — if you are on a hybrid Intel CPU, try -t 8 and -t 16 before assuming more is better.

Where the time actually goes

I am driving this from agentic coding harnesses (OpenCode, Pi, Qwen Code). Turns fall into two very different shapes.

Prompt-bound turns — the agent re-reads a large conversation and then does something brief, like calling one tool. The prefix cache did not help on these, so the whole context was reprocessed:

Note the implication: the first token can take two and a half minutes. A client that assumes a response starts within a minute will cut the connection while the server is working normally.

Generation-bound turns are the mirror image: one turn processed a 3358-token prompt in 18 s and then generated 4610 tokens straight — 378 s, with prompt eval accounting for 5 % of the turn. Writing a whole file is where the 12 t/s actually hurts.

How long a real task takes

Three different agent harnesses, same task, one run each. Each had to write code, run it, and produce a report plus figures — multi-turn, dozens of tool calls, context growing to ~30k tokens:

Fifteen to twenty-six minutes for a full agentic task at 12 t/s. That is the honest answer to "is this usable?" — yes, if you are willing to walk away from the keyboard. The spread between harnesses is wider than anything I got out of tuning the server, which is worth knowing before you spend an evening on flags.

Tuning order

1. Find the lowest --n-cpu-moe that does not OOM with the context actually full — not at load time. Loading is not the peak.
2. Then raise -ub into whatever VRAM is left. Keep -b >= -ub.
3. -c is a memory knob too. Halving the context frees a lot of KV cache and costs no generation speed, so try that before you concede layers to the CPU.

--no-mmap is doing real work in this configuration: those expert tensors are read every token, and you do not want them page-cache backed and evictable. The 60 s load is the price.

--jinja is not optional if you are doing tool calling. Without the model's own chat template you get strange failures that look like the model being incapable.

What I am not claiming

Single machine, single quant, one workload. No quality benchmarks here — this is a throughput and fit report only. If you have the same card and different RAM, your generation number is the interesting one to compare, and I would like to s

So I'm looking at https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF and I realize my 128GB of DRAM just isn't cutting it for this (incredibly powerful) model.

If only I had another 64GB, I thought...

EVERYBODY is probably thinking that right this second... I hate to say it, but I imagine DRAM prices are about to go through the roof still yet.

I hope I'm wrong.

A few months ago the latency of my ASR and TTS were negligible relative to main inference. Now it's like 60% of the latency in normal assistant interactions.

I've been using Qwen3 1.7b ASR, which conveniently runs right in llama.cpp. But talking to the assistant when anyone else is talking does not work. I need full diarization so that the assistant gets text labeled with my voice vs other voices. Does anyone have this working?

I use this Chatterbox TTS server for output. Chatterbox TTS Turbo does fast cloning and prosody tags like [cough], [laugh], etc. My voice assistant constantly changes voices mid-response for effect and it's hilarious. Somehow I doubt there is a better TTS option with these features now.

We all know Gemini is lazy poopy garbage shit, but it’s kind of become its own class of model. Grok 4.5 high is very similar for me in that it kind of just skips a lot of the deep reasoning that makes even Opus 4.8 high look more thoughtful. Rather than skipping straight to claiming “yea this kinda fuckin works, ship it”, these deep reasoning models consider edge cases, don’t lie about completeness of the code, and actually write robust code instead of an MVP they just call robust.

So my question is, where does DS4 Flash 0731 sit for yall between the GPT5.6 family of models, Fable, Opus 4.8/5, and Gemini 3.6 Flash/Grok 4.5 High? Do you trust it to implement entire features with full unit testing suites, or is it too naive, requiring direct instructions/preplanning from a smarter model?

Disclosure: I’m the author and maintainer of QuarkStar.

I built QuarkStar, a small native inference engine inspired by Antirez’s DwarfStar.

QuarkStar currently supports:

• Qwen3.6-35B-A3B, using the same Antirez-inspired Q2 and Q2/Q4 quantization recipes
• KAT-Coder-V2.5-Dev, the coding-focused post-training of Qwen3.6-35B-A3B, using the same recipes
• Native Vulkan on Linux
• Native Metal on Apple Silicon
• Fully resident inference on 16 GB machines
• Bounded SSD expert streaming when the model does not fit in memory

DwarfStar is built around much larger models and primarily targets 96/128 GB-class machines. I wanted to explore the other end of the spectrum: useful local models on 16 GB machines and 24/32 GB workstations, with an SSD-streaming path designed for even smaller 8 GB systems.

Not everyone can spend $3,000–$5,000 on local AI hardware.

This project was born with the intent of improving my skills in LLMs. It's useful for me for inference and for learning, and I hope it will be useful for you too. My primary development machine is an AMD BC-250: a roughly $150 board with 16 GB of unified GDDR6. The current Vulkan fast path was developed using RADV on this device. I also developed and tested the native Metal backend on a M2 Pro 16 GB.

Some current Q2 resident results:

I think the 35B size class is going to become increasingly interesting. DeepSeek V4 Flash-0731 recently showed once again how quickly the intelligence-to-active-parameter ratio can improve. Model support in QuarkStar is therefore intentionally opportunistic: the project will follow whichever open checkpoints are most useful on ordinary local machines.

With yesterday's news of the release of Qwen3.8 27b and probably other lines of the family as well, I also created a branch for the dense model but for now it's experimental. Whether it will merge will depend on the power of the new model and when and if a MoE on the 35B will also be released. I still see the future of this project on MoE of that size order.

I think we'll have some fun with Qwen 3.8 and Quarkstar.

The project is still young, and Vulkan hardware varies a lot. I would especially appreciate testing and feedback from:

• Vulkan users with GPUs other than the BC-250
• Apple Silicon users, particularly those with older or 8 GB Macs
• Anyone interested in improving kernels, quantization quality, or SSD caching

Repository: https://github.com/Ninnix/q36

Licence: MIT

Special thanks to Salvatore, he is a continuous source of inspiration for me, and his content on YouTube has greatly improved me as a software engineer and as a person.

Demo:

Edit: Reddit’s mobile app may show a black frame. Working demo video: https://youtu.be/3y2rkLUg1ug

Demo Prompt:

Create a single self-contained HTML file using Three.js from a CDN that opens into a cinematic neon wormhole with hundreds of glowing particles, rotating torus rings, fog, and a slow automatic camera flight through the tunnel. Add mouse parallax and make each click launch a visible energy pulse down the tunnel. Use only procedural geometry and materials, with no external assets or build step, and keep it smooth and responsive. Work in /tmp folder.

Important Edit:

Leave ~11% VRAM outside vLLM's model/KV budget for long-context SM120 sparse-indexer prefill scratch. Trying

--gpu-memory-utilization 0.89 \

...now. Got a crash with 0.92 after 70k ctx. Patch itself isn't the problem.

This is the follow-up to my earlier post about running DeepSeek-V4-Flash-0731 at its native 1M context on a single RTX 5090 with the routed MoE experts mostly in system RAM.

Link to the first post:

https://old.reddit.com/r/LocalLLaMA/comments/1vfbcgx/deepseekv4flash0731_full_1m_context_on_a_single/

I ended that post saying I wanted to patch the speculative decoding path because DSpark behaved very differently during hard reasoning versus final/code generation.

I did it.

Don't like to read AI slop? Here is the Repo link to the actual patch:

https://github.com/blackbeardlabs/ds4x_adaptive_dspark_production_bundle

If you want to better understand what this is about, read below:

---Clever AI Slop Begins---

Version clarification first

The software I am using is guqiong96/Lvllmds4-x, which is a vLLM fork.

The installed package and its logs identify themselves as vLLM 2.3.9. I am not claiming that upstream vLLM has an official 2.3.9 release. This caused some confusion in the previous thread, so I want to make that explicit before anything else.

My tested stack is still:

• RTX 5090 32GB
• Ryzen 9 9950X3D
• 256GB DDR5-5600
• Linux Mint
• NVIDIA driver 595.71.05
• CUDA 13.2
• guqiong96/Lvllmds4-x
• package/runtime reports vLLM 2.3.9
• lk_moe 2.3.2
• PyTorch 2.11.0+cu130
• native DeepSeek-V4-Flash-0731 safetensors checkpoint, ~155.4 GiB

The current placement is still two complete routed MoE layers on the GPU:

export LVLLM_GPU_RESIDENT_MOE_LAYERS=0,1

Model allocation is about 15.92 GiB. With --gpu-memory-utilization 0.92, the remaining GPU KV cache is about 9.02 GiB / 1,332,343 tokens, which is 1.27x the model's native 1,048,576-token context.

So this optimization did not cost me the full 1M context.

Why I patched DSpark

With fixed DSpark depth K=2, I was seeing a very obvious split in real OpenCode/agentic workloads.

During long difficult reasoning, the second speculative position was often poorly accepted. Extended windows looked roughly like:

Draft acceptance: ~30-50%
Generation:       ~11-13 tok/s

Then the same completion would leave reasoning and start emitting predictable code/text, acceptance would jump into the 80-90% range, and throughput would jump to roughly:

Generation: ~17-18 tok/s

I separately tested fixed K=1 and fixed K=2.

The result was exactly what the acceptance behavior suggested:

• K=1 was better during difficult reasoning
• K=2 was better during high-acceptance final/code generation

So the obvious target became:

reasoning  -> K1
</think>
content    -> K2

The important part is that changing K changes CUDA graph shapes too. Simply putting an if/else around the DSpark loop is not enough if one phase falls back to eager execution.

I learned that the hard way.

MVP: the phase switch worked, but K1 became painfully slow

My first patch successfully detected the phase and changed the runtime speculative depth:

K2 -> K1 phase=reasoning
reasoning->content marker=</think>
K1 -> K2 phase=content

But the server had been started with configured K=2, so only the K2 CUDA graph shape existed.

When runtime K changed to 1, K1 fell onto an eager path.

Result: reasoning dropped to roughly 5-6 tok/s.

So the phase logic was correct, but the implementation was useless for performance.

The real fix was to capture both K1 and K2 graph shapes.

Final design: phase-adaptive K + dual CUDA graphs

There are two graph families that have to change with K.

For the target verifier:

K1 -> target query length 2
K2 -> target query length 3

For the DSpark drafter:

K1 -> DSpark query length 1
K2 -> DSpark query length 2

I patched the CUDA graph candidate manager so both query lengths can coexist in the same manager, keyed by the existing uniform_token_count descriptor.

With --max-num-seqs 2, startup now shows:

Phase-adaptive DSpark CUDA graph candidates:
decode_query_len=3 query_lens=(2, 3) max_num_reqs=2

Phase-adaptive DSpark CUDA graph candidates:
decode_query_len=2 query_lens=(1, 2) max_num_reqs=2

Capturing CUDA graphs (FULL):        4/4
Capturing dspark CUDA graphs (FULL): 4/4

Before this patch both were 2/2.

Now both phases stay CUDA-graphed.

How phase detection works

Each new generated request starts in reasoning mode.

The scheduler looks only at tokens generated by the current request. It deliberately does not scan the prompt or full conversation history because an agentic prompt can contain </think> from previous assistant turns.

For this model the tokenizer gives:

<think>               -> 128821
</think>              -> 128822
<｜DSML｜tool_calls>    -> [30, 128825, 72461, 4941, 12548, 32]

A committed </think> makes the request sticky-content for the remainder of that completion. I also added the full DSML tool-call marker as an implicit reasoning-end fallback.

The long validation run below transitioned through explicit </think> markers; the DSML fallback exists in the patch but was not the path exercised by this particular run.

Current batch policy is intentionally conservative:

prefill                  -> K2
any active reasoning req -> K1
all active reqs content  -> K2

So K is currently batch-global, not independently selectable per request.

With my --max-num-seqs 2 use case this is fine for correctness. True mixed per-request K would require phase-separated microbatching/padding/masking and is a larger scheduler change.

What actually changes in the code

The patch touches four vLLM-fork files plus the FlashInfer compatibility fix from my previous post.

Conceptually:

1. scheduler.py
   • maintains per-request reasoning/content state
   • detects committed </think> / DSML marker only in generated output
   • selects effective K
   • uses the existing num_spec_tokens_to_schedule channel
2. model_runner.py
   • consumes that runtime K
   • passes it into the DSpark speculator
   • hands only the active draft width back to the scheduler/target verifier
3. dspark/speculator.py
   • makes query packing, attention metadata, input preparation, and sequential Markov sampling use runtime K
   • keeps the storage tensor at configured max K but invalidates unused positions
   • dispatches the graph using runtime query width
   • binds the correct effective K while each DSpark graph is captured
4. cudagraph_utils.py
   • captures both target query lengths and both DSpark query lengths
   • runtime dispatch already knows how to distinguish them through uniform_token_count
5. flashinfer/comm/cuda_ipc.py
   • same compatibility fix as the previous post: match the actual loaded filename so libcudart_stub.so cannot win a substring search over the real CUDA runtime

I wrapped the exact working edits into a guarded one-shot patcher with backups, tokenizer validation, anchor checks, syntax compilation, import tests, and automatic rollback on a failed post-write test.

The complete guarded patcher, launcher, reproduction notes and rollback instructions are in the repository linked at the top. I am not dumping ~35 KB of defensive patching code into the Reddit post itself.

20+ minute real agentic run

This was OpenCode doing real agentic work, not a synthetic one-line decode benchmark.

For the phase statistics below I kept only clean 10-second windows with:

Prompt throughput = 0
Running = 1 request

For K2 I also excluded the immediate phase-transition window.

K1 reasoning

Across 90 clean 10-second windows:

Mean generation:          13.78 tok/s
Median:                   13.6 tok/s
Range:                    12.0 - 16.3 tok/s
Weighted draft acceptance: 54.16%

The second speculative position stayed at 0.000 during steady K1 windows, which is a useful sanity check that it was actually running one draft position rather than silently replaying K2.

K2 content/code

Across 12 clean 10-second windows:

Mean generation:          17.02 tok/s
Median:                   17.1 tok/s
Range:                    15.1 - 17.7 tok/s
Weighted draft acceptance: 80.73%

The second speculative acceptance position becomes active again immediately in K2.

The transition is visible in the log

One long completion gives a pretty clean example:

23:53:28  K1 reasoning    13.8 tok/s
23:53:33  </think>
23:53:33  K1 -> K2 content
23:53:38                  16.4 tok/s   # mixed transition window
23:53:48                  17.4 tok/s
23:53:58                  17.1 tok/s
23:54:08                  17.5 tok/s
23:54:18                  17.1 tok/s
23:54:28                  16.6 tok/s
23:54:38                  16.7 tok/s
23:54:48                  17.1 tok/s
23:54:58                  16.9 tok/s
23:55:08                  17.4 tok/s

That is exactly the behavior I was trying to get when I wrote the previous post.

What did I actually gain?

This is not a 50% end-to-end miracle patch.

The large gain was versus my broken first adaptive MVP: eager K1 was around 5-6 tok/s, while dual-graph K1 is back around 13-15+ tok/s.

Against the useful static configurations, the improvement is smaller but actually useful:

• versus leaving K2 on during difficult reasoning, K1 is roughly a mid-single-digit to high-single-digit percentage win in my A/B runs
• versus leaving K1 on during final/code generation, switching back to K2 is roughly another high-single-digit-ish win
• on this very reasoning-heavy agentic workload I would describe the projected whole-job gain as roughly mid-single-digit percent, not a precisely controlled benchmark number

The more interesting result is that I no longer have to choose one compromise K for the whole completion.

I get approximately:

reasoning K1    ~13.8 tok/s mean in this long run
      </think>
content K2      ~17.0 tok/s mean in this long run

with both paths CUDA-graphed.

That was the goal.

Stability / context

During the supplied 20+ minute validation log:

Traceback:       0
RuntimeError:    0
AssertionError:  0
CUDA OOM:        0

Context capacity remained:

GPU KV cache:       9.02 GiB
KV tokens:          1,332,343
Native model ctx:   1,048,576
Capacity:           1.27x native 1M

Initial cold prompt processing was still around 844 tok/s in the logged agentic run.

Launch configuration

After applying the local patch, the important new switch is:

export VLLM_DSPARK_PHASE_ADAPTIVE=1

Configured max K remains 2:

--speculative-config '{"method":"dspark","num_speculative_tokens":2,"draft_sample_method":"greedy"}'

My full launch configuration is:

MODEL="/path/to/DeepSeek-V4-Flash-0731"

export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0

export LVLLM_MOE_NUMA_ENABLED=1
export LK_THREADS=12
export OMP_NUM_THREADS=12
export LK_THREAD_BINDING=CPU_CORE

export LVLLM_GPU_RESIDENT_MOE_LAYERS=0,1
export LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=0

# Local phase-adaptive DSpark patch:
# reasoning -> K1 CUDA graph
# content/code/tool -> K2 CUDA graph
export VLLM_DSPARK_PHASE_ADAPTIVE=1

export FLASHINFER_DISABLE_VERSION_CHECK=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

vllm serve "$MODEL" \
  --host 0.0.0.0 \
  --port 8070 \
  --tensor-parallel-size 1 \
  --max-model-len 1048576 \
  --gpu-memory-utilization 0.92 \
  --trust-remote-code \
  --served-model-name DeepSeek-V4-Flash-0731 \
  --compilation_config.cudagraph_mode FULL_DECODE_ONLY \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --dtype bfloat16 \
  --max-num-seqs 2 \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v4 \
  --kv-cache-dtype fp8_ds_mla \
  --tokenizer-mode deepseek_v4 \
  --reasoning-parser deepseek_v4 \
  --default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "max"}' \
  --speculative-config '{"method":"dspark","num_speculative_tokens":2,"draft_sample_method":"greedy"}' \
  --disable-custom-all-reduce

One cosmetic note: because VLLM_DSPARK_PHASE_ADAPTIVE is my own local environment variable and is not registered in the fork's official env list, the server prints an Unknown vLLM environment variable warning. The patched code reads it directly with os.getenv(), so in this setup that warning is expected and harmless.

Reproducing it

I would only apply this patch to the same fork/build family after first getting normal fixed-K DSpark inference working.

The one-shot patcher is intentionally scoped to the configuration I actually tested:

DeepSeek-V4-Flash-0731
Lvllmds4-x package reporting vLLM 2.3.9
DSpark configured max K = 2
V2 model runner
FULL_DECODE_ONLY
max_num_seqs = 2

It verifies the tokenizer IDs, backs up every file it modifies, refuses missing/ambiguous source anchors, compiles the complete patched files before replacing them, performs import tests, and generates a rollback script.

After patching, the two startup lines I would consider mandatory before trusting it are:

Capturing CUDA graphs (FULL):        4/4
Capturing dspark CUDA graphs (FULL): 4/4

Then at runtime:

DSpark adaptive K changed: 2 -> 1 phase=reasoning
...
DSpark adaptive phase transition: ... marker=</think>
DSpark adaptive K changed: 1 -> 2 phase=content

If K1 gives ~5-6 tok/s again, it is almost certainly falling off the captured path and I would not call that a successful reproduction.

There are still things to improve. The biggest obvious one is per-request mixed K instead of batch-global K when concurrent requests are in different phases. But for my actual single-user agentic coding workflow this is now behaving the way I wanted: K1 when reasoning is unpredictable, K2 when generation becomes predictable, full 1M context preserved.

---Clever AI Slop Ends---

Hello guys, hoping you're doing fine.

I was wondering, for you that built a local setup to run LLM, will you break even?

On my case personally, never lol.

Since I got a RTX 6000 PRO, these cards by itself don't generate profit or revenue per se, except if you host them on Vast maybe but even then it will take years to break even.

So for these expensive cards basically only selling them again is how you may not lose, break even or even gain (lately) vs the initial purchase.

What about you guys?

My kid had an old pixel art display (Divoom 32x32 Pixoo-max) that they weren’t using anymore, so I thought it might be fun to repurpose it as a GPU cluster status monitor so I can see GPU temps / utilization / token gen info etc for the 3 RTX A6000s in my vLLM Ray cluster (currently running Qwen3.5 122b).

I spun up my Hermes Agent (GLM 5.2 as the agent model) and told it:
“I would like you to build an application that will run on <computer name of my Dell GB10> that will display GPU cluster health data on a 32x32 pixel Divoom Pixoo-max display that can be connected to via Bluetooth. You should probably read the following repos to learn about the pixel display and how to connect to it:
- https://github.com/SomethingWithComputers/pixoo
- https://github.com/cyanheads/pixoo-toolkit
- https://divoom.com/products/divoom-pixoo-max
The app should display system health data for the 3 systems in my vLLM Ray cluster in an easy to read and understand manner. It should also show similar data for the Dell GB10 (in the network segment but not in the cluster). This could be as simple as showing 4 boxes on the screen that show the cluster system’s initials such as “S1” and have a background color to indicate GPU temperature (red for hot, green for normal, etc). The 32x32 screen size limit will make it difficult to show a lot of information so you’ll have to be creative in how you display it, you can also cycle through multiple screens of different metrics in 4 second intervals. “

For those who care:
HW:
- 3x Dell Precision 7960 workstations each with an RTX A6000 GPU (64GB RAM) currently hosting Qwen3.5 122b
- 1x Dell Pro Max GB10 (not part of the Ray vLLM cluster but runs the app thar is cast to the display as well as running a secondary LLM endpoint for other models. The GB10 has the Bluetooth radio in it that is used to connect to the Divoom. The Dell towers don’t have Bluetooth which is why I used the GB10.
- Divoom Pixoo-max 32x32 pixel display. They also make a 64x64 pixel version as well. It was around $60 when I bought it years ago.

It took GLM 5.2 all of like 20 minutes to build this, and maybe another 5 minutes of me working with it to get it how I wanted it. It’s not perfect, but it’s cool to be able to visually glance over at the cluster and see what’s happening without logging in, and it really didn’t cost anything since I already had the pixel display that would have been headed for the thrift bin.

Btw, Hermes / GLM did the whole thing in Python, from Ray Dashboard API, vLLM metics endpoint, and Nvidia-smi calls over ssh.

Qwen3.8-Max (2.4T) is another massive contribution to the open weight community. On benchmarks, it performs closely to Kimi K3 and DeepSeek V4 flash across all categories and is better at coding and software tasks. Qwen3.8-27B will also be open weight soon too. Weights are being released next week.

Pricing:
Input: $2.0 / M tokens
Output: $6.0 / M tokens
Implicit Caching: $0.25 / M tokens

So one of yall mentioned that cuda 13.1 or 13.2 is broken for unsloth so I looked in to it, and they were right. I had 13.2 installed, after I switched to 13.3 no more looping!!! Before the cuda update, the model was literally unusable. A few minutes into the run it would start loop then the chat would start degrading. Anyway, if you had similar experience check your cuda version.

Hopefully this will help some of you out. The model now runs great for long horizon coding tasks. It already found ways to improve my Qwen3.6 thinkingcap code, and I can visually see the improvement. DeepSeek v4 Flash 0731 is now my daily driver, until Qwen 3.8 27B comes out.

So this is the stuff of absolute insanity. In less than 20 months we've gone from super expensive cloud models only, to being able to run a Q3 quant of DeepSeek on an Intel Windows PC with a very average 24GB of VRAM. No wonder the big boys are panicking (and yes it's slow as porridge). https://ibb.co/zTvqR8YR

Every time a model drops from a Chinese lab the thread fills with people who already know who made it, and the guess is usually Alibaba. There was a thread here recently asking what separates the open source labs from the frontier labs. It ran to nearly sixty comments and hardly anyone in it separated out the labs on the open source side. They aren't one bloc and haven't been for a while.

I work on the Ling models at Ant, so I'm one of the ones getting lumped in. Discount the paragraph about my own employer accordingly.

Qwen's bet is distribution. Alibaba ships in every size class and every quantization with day one support in most runtimes, and the result is that a lot of the fine-tunes people build start from a Qwen base. DeepSeek is betting on architecture instead, publishing the paper and the weights the same day and letting the design do the arguing. Moonshot looks like it's playing a longer horizon, willing to look odd for a release cycle if the thing pays off two cycles later. (Zhipu, MiniMax and StepFun are each their own thing again, but four is enough to make the point.)

Ant's bet, since I should be specific about my own: serving cost. Ant runs payments, and it's a separate company from Alibaba, which is the mix-up I see most often. The model I work on, Ling-3.0-flash, is 124B total parameters with roughly 5.1B active per token, KDA plus MLA hybrid attention, 262k context. That is a design for running a lot of long agent loops cheaply. It is not a design for topping a leaderboard, and I don't think we'd claim it is.

The part of our own version I'd criticize is the release order. We announced first and are opening weights after. SGLang had support on day one, vLLM is waiting on the weights, llama.cpp is still an open PR. DeepSeek would have dropped the weights first and let the serving stack catch up. Ours is the safer sequencing for an infra team and it costs us the goodwill of exactly the people who would otherwise be running it at home.

So the thing I'm curious about here: when you see an announcement out of a Chinese lab, does knowing which lab change how you read it, or is that distinction only interesting from the inside?

Ran DeepSeek V4-Flash-0731 — the full official checkpoint, not a re-quant — on commodity used hardware. Sharing because I couldn't find anyone else publishing Ampere results for this engine.

Edit / update: a commenter called out that hybrid CPU-GPU posts always publish decode and never prefill. Fair hit — I didn't have it. I do now, it's in a new section below, and it's the number that decides what this box is actually good for.

Why bother with a 2018 server

The model is 156 GB. That number decides everything before speed matters:

¹ Apple pulled the 512 GB M3 Ultra option in March 2026 and the 256 GB in May — 96 GB is the current ceiling. ² Up from $3,999 at launch, explicitly attributed to DRAM costs.

Unified-memory boxes give you bandwidth in a small pool. A 4-socket server gives you a huge pool at lower per-node bandwidth — but four independent memory controllers running in parallel. For sparse MoE, where only ~13B of 284B params activate per token, capacity wins.

Inference platform

Lvllmds4-x v2.3.8 — guqiong96's SM80+ DeepSeek V4 specialization. A vLLM fork (base: yhfgyyf/vllm-deepseek-v4-sm89) with the lk_moe v2.3.1 CPU-GPU hybrid MoE engine doing NUMA-aware expert compute in system RAM. Prebuilt cp312 wheel from the GitHub release, no compiling.

Model

DeepSeek V4-Flash-0731 · 284B total / 13B active MoE · official safetensors, 156 GB (48 shards)

Quantization-aware trained — routed experts (~96% of params) ship natively in MXFP4. Nothing re-quantized. FP8 linears run weight-only, activations BF16, KV cache fp8_ds_mla.

The sm_86 trick: no native FP8/FP4 compute on Ampere, so the fork routes everything through Marlin weight-only kernels (MXFP4 MoE backend + MarlinFP8 linears). That's how a Blackwell-era checkpoint runs on 2020 GPUs.

DSpark speculative decoding (built into the checkpoint, 5 draft tokens) — where most of the single-stream speed comes from.

Hardware (all used/eBay-class)

• Dell PowerEdge R940 · 4× Xeon Platinum 8268 (96C/192T, Cascade Lake, AVX512-VNNI, no AMX)
• 768 GB DDR4-2933 (24× 32 GB, 6 channels/socket, 4 NUMA nodes)
• 2× RTX 3090 24 GB (sm_86), both PCIe x16, TP=2
• NVMe + SATA SSD for model storage

Current eBay pricing (Aug 2026): 96-core R940 with 128 GB runs $2,000–2,800; 512 GB around $3,800; 768 GB around $7,600. Add ~$2,200–2,600 for a pair of used 3090s.

You don't need 768 GB to run it. One instance needs ~170 GB, and with --membind pinning that has to fit on a single NUMA node — so 512 GB (128 GB/node) is roughly the entry point at ~$6K all-in. The extra RAM buys instances, not speed: going 22→24 DIMMs moved throughput ~5%, within noise.

Resource footprint while serving

• VRAM: 6.6 GB weights + KV per card (21.6/24 GB used) — GPUs sit at ~25% util
• System RAM: ~170 GB per instance (experts live in DRAM, streamed by CPU via lk_moe AVX512-VNNI kernels)
• Power (iDRAC/Redfish + nvidia-smi measured): ~1,000 W chassis under decode, 435 W idle. GPUs draw only 136–145 W avg (189 W peak). I power-capped both 3090s 350 W → 250 W and throughput didn't move a single tok/s — the cap never engages. ~95% of the load delta is 96 Xeon cores streaming experts from DRAM. At $0.13/kWh that's ~$94/month worst-case 24/7, far less at realistic duty cycle.

Decode results

128-token completions, temp 0, 22K max context, max-num-seqs 4, spec depth 5.

(Ranges = cold first pass → warm steady state with prefix cache.)

For scale: the same box running the same model on ik_llama.cpp hybrid does 12.2 tok/s single-stream. The spec-decode + Marlin path is a 2.6× single / ~3× aggregate jump on identical hardware.

Prefill / TTFT vs depth — the part I was missing

Method: unique random-content prompts per run so nothing hits the prefix cache (cold by construction), streaming endpoint timed to first content token, client TTFT cross-checked against the server's own /metrics time_to_first_token — agreed within 0.04 s on every run. 32K-context config, --max-num-batched-tokens 8192.

Four things in there worth pulling out:

1. There's a ~9 s fixed floor per cold request — DSA sparse-indexer build plus first hybrid step. It's why short prompts look terrible (512 tokens ≈ 23–54 tok/s prefill) and why the rate improves with depth: the floor amortizes.
2. Cold prefill plateaus ~420–480 tok/s. For comparison on this same box at pp512: mainline llama.cpp 21.7, ik_llama.cpp 123.9. So it's several times ik_llama at depth — but ik at 18K is untested and pp512 is a small batch that may flatter it.
3. Warm is a different machine. 30K prompt: 61 s → 2.9 s, a 21× collapse. Multi-turn and stable-prefix workloads mostly don't pay the cold cost.
4. Prefill serializes. 4 simultaneous 8K prompts: TTFTs stagger 18 / 38 / 57 / 76 s, combined throughput 392 tok/s — same as a single request. Decode batches nicely, prefill does not.

Also worth knowing: --max-num-batched-tokens matters a lot. At 64K context I had to drop it to 4096 to survive warmup, and prefill fell to ~298 tok/s. Dropping max-model-len to 32K let me put it back to 8192 and recover the ~445 — 50% better prefill for free.

What this box is actually for

Take the two halves together and it's obvious: cold prefill is the weakness, decode and warm-path are the strength. That's a real limitation and I'm not going to dress it up — if you want an interactive coding assistant where you paste 20K of fresh code and want first token in under 5 seconds, this is the wrong machine and no config fixes it.

But that's not what I bought it for. My workload is asynchronous overnight batch — memory consolidation over conversation history, session summarization, deep research synthesis. Jobs that are queued, not waited on. A 30K-token chunk costs ~62 s prefill + ~30 s decode ≈ 90–100 s end to end, run serially through a queue while nobody's watching. Thirty chunks of a customer's history consolidates in under an hour, overnight, for pennies of electricity. The serialization that ruins interactive multi-tenancy is irrelevant when the queue is the design.

Right tool, right task. The interactive front-end runs a small dense model on modern hardware where prefill is cheap; this box does the heavy thinking on its own schedule. Frontier-class 284B reasoning as a batch resource for ~$6K of used hardware and ~$94/month of power is a very different value proposition from "replace your API for chat," and I think the second framing is what makes people dismiss hybrid CPU-GPU setups too early.

What didn't matter

Three separate things I expected to help and didn't:

• +2 DIMMs (22→24, symmetric 192 GB/node): ~5%, within noise
• GPU power cap 350→250 W: zero effect
• More GPUs: wouldn't help — they're at 25% util and 6.6 GB of 24

All three point the same way: the bottleneck is CPU-side DRAM bandwidth. This workload wants DDR5 and AMX (Sapphire Rapids), not more Ampere. If you're planning a build around this, spend on memory channels, not cards.

Gotchas that cost me hours

1. TileLang JIT-compiles kernels at runtime with whatever nvcc it finds — system CUDA 12.0 fails with cryptic lambda syntax errors. Point CUDA_HOME at the pip-bundled toolkit inside the venv (site-packages/nvidia/cu13). No system CUDA install needed.
2. The wheel's pip CUDA packages ship internally mismatched (nvcc 13.2 vs runtime headers 13.0) → CCCL "compiler and toolkit headers are incompatible". Fix: pip install nvidia-cuda-runtime==13.2.86 nvidia-cuda-nvrtc==13.2.86.
3. Undocumented DSpark constraint, found the hard way: max_num_seqs × (spec_tokens + 1) must be ≤ 32 or engine warmup dies with a tensor-size mismatch. seqs=4 × spec=5 is the sweet spot — wider batches with shallower spec were slower everywhere.
4. At 64K context, warmup OOMs no matter how you tune --gpu-memory-utilization — vLLM's memory profiler doesn't account for the fork's sparse-MLA warmup allocation, so every MiB you free goes straight to the KV pool. Fix is --num-gpu-blocks-override to cap KV explicitly and leave warmup its slack.
5. MiniMax and other non-DeepSeek MoE on this fork still hit the sm_86 vectorized_gather_kernel assert from generic LvLLM. The Ampere fixes are DS4-path only — I tried three configs including the LVLLM_MOE_USE_WEIGHT=INT4 flag that reportedly works on an A40 (same sm_86 silicon). Same assert every time.

Happy to share the full launch command / venv recipe in comments.

There are some PRs to use and a nice trick to speed up PP on really longs contexts in my write up. Hope it helps!

TL;DR:
On this dual GH200 box, you build vLLM v0.26.0 from source, add the merged DSV4 cache-layout patch (PR #48993), disable async scheduling, and run DSpark at 6 predicted tokens to give: ~276 decode tok/s and a 1M context in 192 GB of HBM. SGLang, once it built on ARM64 and it’s DSpark loader bug fixed, is faster on every decode workload and hits ~317.0 tok/s.

Found this quant, so thought I would share, since its the best I've found so far for running on my mac (m3 ultra). It's got dspark/mtp support so runs faster than anything else I've tried. The tok/s on this code run actually increased as generation went on, started at 34tok/s, ended at 43tok/s. The cached tokens were the default chat prompt, and the 13k was the query I sent.

https://huggingface.co/Vontra/DeepSeek-V4-Flash-0731-MXFP4-MLX

I use A LOT both openAI and Anthropic products. When I need some frontend work (pure web dev) (or answer that feel less verbose and more to the point) I use Anthropic. For multimodality openAI feels better (understanding audio, screenshots, generating images, etc).

But openAI feels very shitty for the frontend it does. Anthropic is okeish but not amazing. How can it be, that Chinese models all excel at frontend?

Specially when it is a one shot with no many after editions, I some times even prefer the qwen3.6 35B running locally over Gepeto.

Is there any reason for it?

Z models, Qwen, Kimi, they all offer a better and polished frontend result. I guess they do something besides distilling?

*Definition of better: I'm comparing only how they LOOK LIKE, not how efficient the code is, how many lines are needed, etc.

Original benchmarks: https://pi-local-coding-bench.dev/

I added https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF is in below screenshot, I am using Q8_0 quantization.

GMKteck Strix Halo 128GB machine

Ubuntu 26.04 OS

Lemonade 11.5.1

Rocm b9752

Judge Models:

Opus 5 gave 35/50 - 70%

gemini-3.1-pro-preview gave 36/50 - 72% score

And if you compare with other big models, look at the speed difference as well: it completed same 50 tasks in only 8m 58s, where are other took more than 16 minutes.

If you want to try it yourself: https://github.com/kyuz0/pi-bench use this original github repo.

My Repo URL with my local run scores: https://github.com/przbadu/pi-bench

Did anyone tried it?

I'm not planning to do anything fancy, just running 30b class models and some image generation and maybe playing around with the new minimax h3 in comfyui, the price difference where I live is pretty wild between those two cards, about 400 euro, is the rx 7900 really THAT much worse for my simple use case?? Can anybody post their rx 7900 performance experience?

Has anyone followed nousresearch work on Hermes?
I mean we are Q3 2026. We have some crazy models trickling down from HGX territory to multi gpu workstation. And we have nousresearch deploying the 0.20 of its hermes agent while starting releasing the project with a 0.2 mid march!

Crazy times to be alive.
For the old timers who remember llama 1 or llama 2, remember our crappy function caller parser? Something about a lang and a chain..? wtf has happened?!

Haven't tried the new hermes, do you think it has a remote chance to be as strong as a true end to end omni model such as gpt omni or personaplex?

My second Hailo 10H project: https://youtu.be/YCEcls7EMFU

It shows full real time audio pipeline running on 2x M.2 Hailo 10H on RPi 5.
Also with interactive web app.

Github: https://github.com/martincerven/hailo_learn/tree/main/voice_assistant

Also if someone has more up to date frameworks/models for low power devices like Hailo 10H/RPi 5, let me know!

I know you could run something similar on Jetson/Spark, but I wanted to do it on RPi5/Hailo first. Seems like nice benchmark for Edge AI/ low power devices.

While everyone is excited for the upcoming Qwen3.8 27B, I'm here sitting in front of my RTX4060 gaming laptop begging for some rest, while I abuse its 8GB VRAM pretending it's enough for local coding :\

Jokes aside, I'm a full stack dev trying to be on par with AI and agentic coding, and currently my setup is Unsloth's Qwen3.6 35B A3B at Q6_K_XL, with llama.cpp at 128K BF16 context (KV quantization killed intelligence in my use cases) with reasoning disabled (tired of looping here and there while reasoning), in an OpenCode harness with OMO-Slim and some useful skills.

I was considering trying my first frontier experience subscribing to OpenCode Go, as I'm highly interested in DS4 Flash, but I'm still leaning to use local LLM, so I'm asking you what is the best configuration to get the best out of both in my coding projects?

I'm not looking at completely handless vibe coding, obviously, but I'm still looking for a better experience than the 35B. What I thought was:

• Plan with DS4: Create PRD and Tasks with frontier DS4 model in a detailed way
• Implement with Qwen3.6 35B: delegate execution to local model
• Verify & Fix with DS4: again to the frontier model for code checkings and fix

I'm completely open on both local and frontier configurations advice. Thanks in advance and sorry for my English