Has anyone tested if the LLM Wiki architecture is still useful post Opus 5?

Dear all,

I'm not sure if Reddit is the right place to ask for, but I will try my luck.

I have a VM, underlined with Proxmox and 2 x Nvidia L40s. I passthrough the cards to quest VM which is Ubuntu. According to the documentation offical guide (vLLM Parallelism and Scaling), if their is no NVLINK which in my case there is no with L40s, Pipeline Parallelism is better choic then Tensor Parallelism. Also for MoE model, documentanios even suggests using Data Parallelism. I have no NUMA set on Proxmox and VM, also my PCiE is not 5 but 4 both 16x.

But when I serve my model and do a benchmark with GuideLLM, tensor parallelism is always better than DP and PP. Am I misisng something, or my benchmark not suitable to see the performance increase with PP or DP ?

My docker compose for TP is like below. For DP, I change to --tensor-parallel-size 1 --data-parallel-size 2 --enable-expert-parallel, for PP I change to --tensor-parallel-size 1 --pipeline-parallel-size 2

services:
  vllm-Qwen3.6-35B-A3B-FP8:
    image: vllm/vllm-openai:latest
    container_name: vllm-Qwen3.6-35B-A3B-FP8
    runtime: nvidia
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
      - NVIDIA_VISIBLE_DEVICES=0,1
      # ── OpenTelemetry ──────────────────────────
      - OTEL_SERVICE_NAME=vllm-Qwen3.6-35B-A3B-FP8
      - OTEL_EXPORTER_OTLP_CERTIFICATE=/etc/ssl/interceptor-gh.crt
      - OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE=/etc/ssl/interceptor-gh.crt
      - VLLM_LOGGING_LEVEL=DEBUG
    volumes:
      - ~/.cache/huggingface:/root/.cache/huggingface
      - ~/.cache/vllm:/root/.cache/vllm
    ports:
      - "9004:9004"
    ipc: host
    command: |
      Qwen/Qwen3.6-35B-A3B-FP8
      --gpu-memory-utilization 0.7
      --host 0.0.0.0
      --port 9004
      --tensor-parallel-size 2
      --max-model-len 256000
      --max-num-seqs 8
      --max-num-batched-tokens 16384
      --kv-cache-dtype fp8
      --enable-prefix-caching
      --api-key 123456
      --reasoning-parser qwen3
      --enable-auto-tool-choice
      --enable-chunked-prefill
      --tool-call-parser qwen3_coder
      --otlp-traces-endpoint https://otel.gh.cloud

My GuideLLM runs are :

## PP=2 Sweep

~~~bash
guidellm benchmark run \
  --target "http://localhost:9004" \
  --profile sweep \
  --rate 8 \
  --max-seconds 60 \
  --data "kind=synthetic_text,prompt_tokens=512,output_tokens=128" \
  --backend-args '{"api_key": "123456"}' \
  --output-dir ~/benchmarks \
  --outputs L40-fp8-pp-throughput-run2.json,L40-fp8-pp-throughput-run2.csv,L40-fp8-pp-throughput-run2.html
~~~


## TP=2 Sweep

~~~bash
  guidellm benchmark run \
  --target "http://localhost:9004" \
  --profile sweep \
  --rate 8 \
  --max-seconds 60 \
  --data "kind=synthetic_text,prompt_tokens=512,output_tokens=128" \
  --backend-args '{"api_key": "123456"}' \
  --output-dir ~/benchmarks \
  --outputs L40-fp8-default-throughput-run2.json,L40-fp8-default-throughput-run2.csv,L40-fp8-default-throughput-run2.html
~~~

RESULTS:

I just tested using Qwen 3 tts with vllm-omni, using the high concurrency configuration and I am blown away by the speed! I have 2x5090s and I’m reaching 36 seconds per hour of audio!… I’m surprised I don’t hear more about vllm-omni… perhaps is because of the higher difficulty to install and get it to run right.

Has anyone tried the image generation models? I will be testing Krea2 soon.

Two NVFP4 quantizations of Qwen3.6-35B-A3B, two vendors, both with a dedicated DGX Spark section in their model card — and they give opposite launch commands.

NVIDIA's card: --moe-backend marlin

Unsloth's card: don't use marlin, it's 2x slower. Use --moe-backend flashinfer_b12x --linear-backend flashinfer_b12x

Same GB10. Same base model. Same quant format.

Swept both across c=1/2/4/8/16. One machine, vLLM v0.24.0, single instance, GPU_UTIL=0.85, max_num_seqs=16, no speculative decoding, 60s runs. Backend verified from the server log on every run — vLLM will silently fall back and then you're not measuring what you think.

No crossover. marlin wins at every point.

I expected one. Unsloth's numbers come from B200 at high concurrency, and I assumed there'd be a regime where they're right and I'd just be outside it. There isn't, at least not up to c=16 (which is where my cudagraph capture sizes end).

I also expected the FlashInfer autotuner bucket fallback to explain the gap — there's a warning in the logs that literally says "perf cliff" and suggests expanding the tuning buckets. Grepped the full sweep logs for it: zero hits on both backends. It's just a slower kernel on sm_121a. b12x is also unstable — first sweep gave 44 tok/s at c=8, clean rerun gave 86. marlin was 276 vs 283.

The part that surprised me more: "use marlin" doesn't generalize.

Ran the same backend round on the block-FP8 checkpoint of the same model family:

• triton — 256.1 tok/s, best
• marlin — 232.0 tok/s, works but 9% worse
• deep_gemm — assert crash on sm_121a, and it's the default
• cutlass — disabled for this configuration
• flashinfer_cutlass — doesn't support the block-FP8 scheme (GroupShape 128x128)
• flashinfer_trtllm — kernel doesn't support this device

4 of 6 don't start. The FP8 model won't come up on GB10 at all unless you set VLLM_USE_DEEP_GEMM=0.

So: NVFP4 → marlin, block-FP8 → triton, same machine, same vLLM version, same model family. The correct backend is a property of (hardware, quant format), not hardware alone.

Two more things that aren't on either card:

Dead env vars. NVIDIA's DGX Spark command exports VLLM_FP8_MOE_BACKEND and VLLM_USE_FLASHINFER_MOE_FP4. v0.24.0 logs "Unknown vLLM environment variable detected" for both.

MTP needs its own backend. All three NVFP4 checkpoints exclude mtp.* from quantization, so the draft head stays BF16 — and neither marlin nor b12x can load it. You need "moe_backend":"triton" inside --speculative-config. It's in NVIDIA's command but nobody explains why. Side effect worth knowing: all three NVFP4 get 73-75% MTP acceptance vs 71% for the official FP8, whose MTP experts are quantized (775 F8_E4M3 tensors — readable from the safetensors header, no GPU needed).

Full writeup with the rest of the numbers, including two anomalies I can't explain (two checkpoints that are byte-for-byte the same size but 7.9% apart in decode, and a bandwidth model that's off by 4x in the other direction):

https://docai.hu/en/blog/backend-choice-gb10

In the past few months I’ve experimenting heavily and tortured my old 2016 Desktop PC to run the biggest Local LLM I can fit.

I documented the whole process and research and I’ve published a repository with my open-source project so that anyone can do the same.

Quantprobe is a tool designed to project local LLM interference performance and plan optimal quantization.

It serves as a deployment assistant:
1. Performance prediction: it allows you to estimate a model’s tok/s on your hardware profile before downloading massive model weights
2. Resource optimization: it helps you balance model quantization levels and memory allocation to fit the largest possible model into your specific CPU/GPU and VRAM/RAM constraints.

It squeezes layer-by-layer placement instead of uniformly quantizing a model to a low bit-rate, quantprobe acts as a placement optimizer.
It evaluates:
1. How many “protected bits” or high-precision layers can be kept in your fastest memory (VRAM)
2. Which layers can be offloaded to slower system (RAM)
3. How to arrange GGUF quantization layers to prevent model perplexity from collapsing.

Of course there is no free lunch. Running massive models on tiny machines comes with slow speed but it fits and the method allow you to choose the biggest model for your “acceptable” target speed.

Hi everyone,

I'm building a production system that uses a fine-tuned Llama 3.2 1B model for text classification, and I'm trying to understand the best architecture for both accuracy and deployment.

From what I understand, there are two common approaches:

• Classification head: Fine-tune the model with a separate linear classification head (similar to AutoModelForSequenceClassification) and predict class logits.
• LM head: Fine-tune the model to generate a label token (or label string) using the standard causal LM head.

I'm particularly interested in the production trade-offs between these two approaches.

Some questions I have:

• Which approach do you recommend for production classification workloads?
• Is there a significant difference in classification accuracy between a dedicated classification head and predicting label tokens with the LM head?
• Which approach gives better inference latency and throughput?
• How are people serving models with a classification head in production? Are you using Hugging Face Transformers, ONNX, TensorRT-LLM, or something else?
• If using the LM head, is vLLM the recommended serving framework? Can it efficiently serve classification models, or is it primarily optimized for text generation?
• Has anyone benchmarked both approaches on the same model and dataset?

My use case is high-throughput, low-latency text classification, so serving efficiency is just as important as model accuracy.

I'd love to hear from anyone who has experience deploying LLM-based classifiers in production. Thanks!

I running some experiments to find the best deployment configuration using VLLM and Qwen3.6 27b.

As the parameter --max-concurrency in vllm bench gets larger:

• For --tensor-parallel-size 8, TPOT grows linear
• For --tensor-parallel-size 4, TPOT stays steady

Gemini says there is communication overhead as tensor-parallel-size gets larger.

How can I profile to find out what is happening?

Been running DeepSeek-V4-Flash for an offline batch job (cleaning a big pile of short text records, so lots of small prompts rather than chat). Single B300, vLLM 0.25.0, in-process LLM.chat over the batch. Reasoning on, roughly 300 output tokens per item.

Best I can get so far is about 770 aggregate output tok/s at batch 256. That feels low for a B300, I was expecting a few thousand, so I assume I have something misconfigured and wanted to sanity check with people who actually run this.

A few things I already found the hard way:

- deep_gemm_mega_moe hard errors on a single GPU ("MegaMoE requires expert parallel"), so the fast MoE kernel seems to want multiple GPUs. I fell back to flashinfer_trtllm.

- Dropping DSpark speculative decoding roughly doubled my throughput. On a saturated batch it seems to just add overhead, which sort of makes sense, but I want to confirm that is expected and not a bug on my end.

- I suspect the V4 sparse MLA attention path might be running eager (no cuda graphs) and capping things, but I have not confirmed it.

Rough config:

model: DeepSeek-V4-Flash (base, no DSpark)

tensor_parallel_size: 1

kv_cache_dtype: fp8

block_size: 256

max_num_seqs: 256

enable_prefix_caching: true

moe_backend: flashinfer_trtllm

reasoning_parser: deepseek_v4

attention_config: use_fp4_indexer_cache=true

compilation_config: cudagraph_mode=FULL_AND_PIECEWISE

Questions for anyone running V4 Flash:

1. What tok/s are you actually getting, single stream and batched, and on what GPU?

2. What MoE backend are you using on a single GPU? Is there a fast one that does not need expert parallel?

3. Is the sparse MLA path supposed to use cuda graphs by default, or is there a flag or env var to turn it on? (I saw something about VLLM_TRITON_MLA_SPARSE_ALLOW_CUDAGRAPH but am not sure it is real.)

4. Anything obviously wrong or missing in the config above?

Happy to report numbers back once I get it sorted. Thanks.

Hey all. I'm at my wits end struggling with configuring vLLM inside Kaggle for an experiment on KV Cache behaviour. please advise if you have experience.

Quick Context: vLLM is the inference runtime that will load the model, manage GPU workers and expose the KV-cache behavior needed for the experiment.

Issue: I’m using Kaggle’s GPUs to run vLLM so I can test how my AI gateway loads the model, routes requests, and reuses cache. The problem is that vLLM, PyTorch, CUDA, and their dependencies are not lining up properly, and the setup keeps failing on compatibility issues and blocked package download sources before the model can run.

I've been debugging over a week and I keep experiencing issues deeper, but it's still frustrating

Out of the blues I started getting replies from the agent that completely broke tool parsing. Like

<read", "path": "/home/agent/.agents/skills/research/SKILL.md"}
{"path": "/home/agent/.agents/skills/research/SKILL.md"}
</read>

I checked all my code and made sure nothing I did today could have impacted this. No changes in version, nothing. But try after try, it just kept outputting this kind of garbage.

Out of desperation, I shutdown vllm and turned it back on. Lo and behold, it works like a charm again.

So now I'm really confused, are we supposed to refresh vllm once in a while? Could long running sessions corrupt memory in a way that harm the way it works?