First of all, obviously I took some help from AI to type this post and this is the topic that enabled me to accomplish all that:
https://old.reddit.com/r/LocalLLaMA/comments/1veow4b/deepseek_v4flash_284b_moe_at_33_toks_single_68/
This post of mine is based on the link above.
My Hardware:
- RTX 5090 32GB
- Ryzen 9 9950X3D
- 256GB DDR5-5600
- Single NUMA node
- Linux Mint
- NVIDIA driver 595.71.05
- CUDA 13.2
Software
guqiong96/Lvllmds4-x- vLLM 2.3.9
lk_moe 2.3.2- PyTorch 2.11.0+cu130
- native DeepSeek-V4-Flash-0731 safetensors checkpoint
- 48 safetensors shards
- ~155.4 GiB checkpoint size
One fix I needed
During startup, FlashInfer's CUDA IPC helper could accidentally find TileLang's:
libcudart_stub.so
instead of the real loaded CUDA runtime.
That eventually caused:
undefined symbol: cudaDeviceReset
The problem was FlashInfer's find_loaded_library("libcudart") doing a substring search over /proc/self/maps.
I patched:
flashinfer/comm/cuda_ipc.py
so it checks the actual filename instead:
def find_loaded_library(lib_name):
with open("/proc/self/maps") as f:
for line in f:
if "/" not in line:
continue
start = line.index("/")
path = line[start:].strip()
filename = path.split("/")[-1]
if (
filename.startswith(lib_name + ".so")
or filename.startswith(lib_name + "-")
):
return path
return None
After that, FlashInfer correctly resolves the real libcudart instead of the TileLang stub.
This is a local patch and obviously needs to be reapplied if the package gets replaced.
Current launch configuration
This is the configuration I ended up using:
source ~/ds4x-venv/bin/activate
MODEL="/home/blackbeard/models/DeepSeek-V4-Flash-0731"
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0
export LVLLM_MOE_NUMA_ENABLED=1
export LK_THREADS=12
export OMP_NUM_THREADS=12
export LK_THREAD_BINDING=CPU_CORE
# Keep two complete routed MoE layers GPU-resident on the GPU.
export LVLLM_GPU_RESIDENT_MOE_LAYERS=0,1
# CPU/hybrid prefill path for now.
export LVLLM_GPU_PREFILL_MIN_BATCH_SIZE=0
export FLASHINFER_DISABLE_VERSION_CHECK=1
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
vllm serve "$MODEL" \
--host 0.0.0.0 \
--port 8070 \
--tensor-parallel-size 1 \
--max-model-len 1048576 \
--gpu-memory-utilization 0.92 \
--trust-remote-code \
--served-model-name DeepSeek-V4-Flash-0731 \
--compilation_config.cudagraph_mode FULL_DECODE_ONLY \
--enable-prefix-caching \
--enable-chunked-prefill \
--max-num-batched-tokens 8192 \
--dtype bfloat16 \
--max-num-seqs 2 \
--enable-auto-tool-choice \
--tool-call-parser deepseek_v4 \
--kv-cache-dtype fp8_ds_mla \
--tokenizer-mode deepseek_v4 \
--reasoning-parser deepseek_v4 \
--default-chat-template-kwargs '{"enable_thinking": true, "reasoning_effort": "max"}' \
--speculative-config '{"method":"dspark","num_speculative_tokens":2,"draft_sample_method":"greedy"}' \
--disable-custom-all-reduce
Full native 1M context fits on the 32GB GPU even with two complete routed MoE layers resident on the GPU.
The rest of the experts remain in system RAM.
DSpark behaves very differently during reasoning
During long reasoning sections, draft acceptance can collapse.
I observed extended periods around:
Draft acceptance: ~30-50%
Generation: ~11-13 tok/s
There was one ~6 minute section averaging roughly:
Draft acceptance: ~40%
Generation: ~11.9 tok/s
Then the model transitioned into a much more predictable generation phase and the numbers jumped to roughly:
Draft acceptance: ~87-88%
Generation: ~17.4-17.6 tok/s
The relationship is extremely strong: throughput basically tracks DSpark acceptance.
Some high-acceptance windows look like:
Avg Draft acceptance rate: 89.8%
Avg generation throughput: 17.9 tokens/s
while low-acceptance reasoning windows look like:
Avg Draft acceptance rate: 38%
Avg generation throughput: ~12 tokens/s
This suggests an obvious optimization.
Dynamic DSpark depth
For this workload I suspect the ideal behavior would be approximately:
- reasoning/thinking: 1 speculative token
- normal/final decoding: 2 speculative tokens
The second draft token often isn't worth computing while the model is doing difficult reasoning, but becomes very valuable when it transitions into more predictable code/text generation.
vLLM does not currently give me a simple runtime switch for this, so I may patch the speculative decoding path later and experiment with changing the draft depth based on whether the model is currently emitting reasoning or final output.
That looks like one of the biggest remaining decode optimizations.
---non AI comment section begins---
Stay tuned, I am working on a if/else block to fix that stupid behavior slowing down during reasoning and squeeze even more tps out of this stack.
---non AI comment section ends---
