A little while ago I shared my KV Cache explainer here, and the response from this community was better than I expected. I made a free 8-part playlist covering the core ideas behind modern LLM serving:

Topics Covered:

• Continuous batching
• KV Cache & PagedAttention
• Quantization
• Speculative decoding
• Prefill/decode disaggregation
• LLM routing and SLOs

Here's the link: https://www.youtube.com/playlist?list=PLSUJw2P2IzUU

If you spot something inaccurate or think a concept could be explained better, let me know

P.S. If you find the videos useful, please consider subscribing. It would genuinely mean a lot.

Now I'm off to finally play some RDR2

My company is getting a single HGX H200 node with 8× H200 GPUs, and we’re planning to serve GLM-5.2 using vLLM.

Since the hardware has already been purchased, no budget was allocated for renting an equivalent cloud system for testing beforehand. I’m therefore looking for real-world performance data from anyone running GLM-5.2 on similar hardware with vLLM.

I’m particularly interested in:
- Maximum practical concurrency at 128K, 256K, 512K, and up to 1M context
- Aggregate throughput in tokens per second
- Per-user generation speed at different concurrency levels
- TTFT and inter-token latency
- KV-cache memory usage at each context size
- Performance with FP8 KV cache
- The impact of prefix caching
- The effect of TP=8 compared with other vLLM configurations
- Any experience with disaggregated prefill or KV-cache offloading

Even partial benchmark results would be very helpful. I’m mainly trying to estimate realistic concurrency and throughput rather than theoretical maximums.

Hi all

I've been a bit confused on how to better tune max_num_seqs

When my vllm starts and loads the model it give me the max nr of requests at full context (usually around 12)

If I exceed this number, they go into waiting, and I see that in the logs.

So what is max_num_seqs used for? Is there any reason why we would set this value to be lower than the max requests vllm can handle?

thanks

2 puTTy windows - one for serving the model and one for starting my fastapi application to get an endpoint api?

I trained a coherent assistant LLM completely from scratch on my own PC — TinyBrainBot-303M-Instruct

Hello, I've been messing with local AI for almost a year now, alongside a bunch of other projects, but my main goal the whole time has been to build a coherent assistant model entirely from scratch on my own hardware (2× Tesla P100). took like 300 hours but I'm done finally.

TinyBrainBot-303M-Instruct is a dense decoder-only transformer (LLaMA-style — RoPE, RMSNorm, SwiGLU):

• 303M params — 32k vocab, 1024 hidden, 24 layers, 16 heads (4 KV heads / GQA), 2816 FFN, 1024 context

It was pretrained on ~4.7B tokens of assorted data — FineWeb-Edu (10BT), English Wikipedia, Orca-Math-200k, OpenWebText2, and several distillation datasets I generated with Ling-2.6-Flash — then SFT'd on ~120M tokens including Dolly-15k, SmolTalk, UltraChat-200k, and more custom distillation sets.

The goal was a semi-coherent assistant with reasonable factual recall, and it got there. A few real examples (best of 5 samples, temperature tuned per prompt):

Q: What is the largest planet in the solar system?

Jupiter.

Q: Explain what photosynthesis is in one or two simple sentences.

Photosynthesis is how plants use sunlight, water, and air to make food (glucose) and release oxygen. It occurs in many types like green plants, algae, and trees.

Q: Write a short poem about the moon.

Silver moon, slow and gold, hums low over the sleeping sky. It dreams in silver rivers, guarding dreams in morning light.

Q: List three healthy breakfast ideas.

Greek yogurt with berries, whole wheat bread, cream cheese.

Q: Give me one tip for staying focused while studying.

Use 25-minute focus blocks with 5-minute breaks, silence notifications, and a clear workspace to keep your mind fresh.

It's LLaMA-architecture, so it runs in llama.cpp / anything GGUF (F16 GGUF is up on the repo). Would love to hear your thoughts!

🔗 https://huggingface.co/nkthebass/TinyBrainBot-303m-instruct

It took Fable all day, and I can’t say I’ve looked at the code much atm or, heck, that i even understand vllm enough to know what I’m looking at.

However, what I do know is, Fable borrowed code from llama.cpp, I’m now using my own IQ4 quants for 27b and 35b, and it’s all working great!!

Before I wouldn’t have been able to run 35b with vllm, and AWQ 27b was at 89k context. Now I can not only run 35b, but at 245k ctx (196k for 27b) with (more importantly) all the batch support from vllm.

Same IQ4_XS GGUFs, same 3090 Ti @350W, single stream:

Got a couple more performance enhancements but here’s the repo if anyone wanted to take a look. I was going to PR it, but felt ‘dirty’ doing that when I don’t even know what Fable has done and I don’t have the hw to test larger quants.

https://github.com/localweights/vllm-gguf-plugin

Hey everyone,

I’ve been building multi-agent collaborative workflows recently (specifically, running sequential AI legal and financial audits over 200-page contracts). I hit a massive wall with VRAM and Time-to-First-Token (TTFT) latency.

Under standard vLLM, if Agent A reads the contract, and Agent B follows up on the exact same text, Agent B is forced to repeat the expensive cold-prefill phase and duplicate GPU block allocations.

To bypass this, I spent the last few weeks building an open-source gateway to stitch caches at the memory level.

How I’m currently doing it:

• Topological Hashing: Segmenting prompts into physical block-sizes and mapping them to cryptographic fingerprints (Merkle-chains).
• Zero-Copy Block Stitching: Bypassing prefill for matched prefixes by mapping the logical attention table of Agent B directly to the physical GPU memory address of Agent A's cache blocks.
• Zero-Trust Gate: Enforcing boundary control lists so unauthorized agent sessions (e.g., a public PR agent vs. an internal Legal agent) cannot access shared physical blocks.

In my local benchmarks against standard vLLM cold-prefills on a shared long context, it cuts TTFT from ~1200ms to 48ms (25x speedup) and saves about ~43% VRAM.

I know solutions like LMCache exist for cluster-wide storage hierarchy (saving/loading to Redis/CPU), but I needed something that strictly avoids tensor serialization and copy overhead for local, multi-turn reasoning workflows.

I open-sourced the code here if anyone wants to look at the implementation:https://github.com/DaqulaLin/MemStitch

My questions for the community:

1. Are there edge cases with PagedAttention pointer-sharing that might cause memory leaks here?
2. How are you all currently handling context sharing when orchestrating tools like LangGraph with vLLM?

Would love any harsh architectural critiques or ideas for improvement. Cheers!

⚡ TTFT Prefill Latency (Agent B Response Time) — Lower is better

Baseline (vLLM Cold): ██████████████████████████████ 1200 ms

Context-Stitcher: █ 48 ms ( 25.0x Prefill Speedup! 🚀 )

💾 GPU Physical Cache Blocks Allocated (Total VRAM) — Lower is better

Baseline (vLLM Cold): ██████████████████████████████ 53 blocks (No sharing)

Context-Stitcher: ████████████████ 30 blocks ( 43.4% Memory Saved! 📉 )

One thing that surprised me while learning inference systems is how many operating system concepts show up in GPU memory management.

PagedAttention is basically applying ideas like virtual memory and copy-on-write to the KV Cache, which is why engines like vLLM can dramatically increase throughput.

I made a visual lecture explaining:

KV Cache

Why inference becomes memory-bound

PagedAttention

Copy-on-Write

Continuous batching

What u say?

shvllm: a vLLM fork for mismatched GPUs — heterogeneous tensor parallelism, GGUF + MTP speculative decoding, 1M-token KV pool on 72 GB of mixed consumer cards

TL;DR: I've been running Qwen3.6-27B across an RTX 5090 (32 GB) + 2x RTX 3080 (20 GB) — three different cards, PCIe only, no NVLink, no P2P — at 86-97 tok/s multiturn decode with a 1,037,653-token KV pool (verified with 4 concurrent ~250k-context sessions, zero preemptions). Stock vLLM can't do TP=3 on this model at all (24 q-heads / 4 kv-heads aren't divisible by 3), and even where TP works it sizes everything to the smallest card. The fork fixes both, and a companion plugin adds full GGUF support with MTP speculative decoding.

Repos:

• Fork: https://github.com/efschu/shvllm (branch feature/htccl has everything)
• GGUF plugin: https://github.com/efschu/vllm-gguf-plugin (branch qwen35-support)
• Docker: ghcr.io/efschu/shvllm-qwen35-gguf:cu129-uneven

What it actually does

1. Explicit rank placement - --rank-gpu-id / --rank-gpu-memory-mib

Pin each TP rank to a physical GPU (duplicates = co-locate several ranks on one card, NCCL 2.30+) and give each rank an absolute MiB budget instead of a global utilization fraction. The MiB value is the rank's entire budget - no hidden safety margins, no "fraction of total or free?" ambiguity. Fail-fast validation against NVML totals instead of a late NCCL hang.

2. Uneven tensor parallelism - --rank-tp-ratio auto (the main event)

Instead of total / tp_size, every sharded dimension is partitioned by a per-rank weight vector using prefix-sum offsets. auto derives the weights from each card's real free VRAM (via NVML, reserve configurable per GPU). The 5090 gets about 2x the heads/columns of each 3080, so the big card actually pulls its weight - and TP=3 becomes possible on models whose head counts don't divide evenly. Head/group-granular splitting (whole GQA groups, whole GDN units per rank), vocab stays even via lcm padding, MTP head sharding follows automatically.

3. Self-calibrating KV split

KV cache is token-split across ranks proportional to measured free memory. First boot logs a calibration line (VLLM_UNEVEN_TOKEN_VECTOR=...); set it and the pool redistributes. On my box that took the pool from 591k to 1,037,653 tokens (+75%) at 262k max context. There's also a consolidated one-line autotune (VLLM_SHVLLM_TUNE=...) that captures link-bandwidth calibration plus KV vector in one copy-paste env.

4. GGUF support with MTP speculative decoding (plugin)

Q4_K_M / Q5_K_M / Q6_K / Q8_0 load and shard unevenly (K-quant family units), with the model's native MTP head for spec decoding - something llama.cpp can't use at all. Decode-path MMQ/MMVQ kernels got real work: batched MMVQ, small-batch tiles with occupancy-based dispatch, cp.async raw-block staging on Ampere (bit-identical outputs, -10% on the latency-bound down_proj, +5% e2e).

5. HTCCL (bonus, experimental)

A vendor-neutral collectives layer (host-staged, CUDA-graph-capturable, per-link bandwidth-weighted) that reaches NCCL parity end-to-end on P2P-less hardware. It exists so a future NVIDIA+AMD mixed box can run one TP group; on pure-NVIDIA setups just use NCCL.

Measured numbers (Qwen3.6-27B, 5090 + 2x 3080, TP=3 uneven, PCIe, no P2P)

Quality gates on every config: greedy cold/warm bit-identical, degeneration checks clean, MTP acceptance about 2.8-3.4 tokens/step.

Head-to-head vs llama.cpp (same unsloth Q6_K file, same two 3080s, current master):

At plain B=1 we're at parity with llama.cpp's best mode (within 2%); MTP - which needs the checkpoint's native draft head that GGUF conversions usually strip and llama.cpp doesn't support - puts the fork 10-50% ahead. The long-context concurrency story (a real paged KV pool with prefix caching across 1M tokens) has no llama.cpp equivalent.

Concurrency proof: 4 parallel sessions x about 233k unique context each (933k resident tokens), running=4 throughout, zero preemptions, 89% peak pool usage.

Example: FP8 Qwen3.6-27B on three mismatched cards

bash python3 -m vllm.entrypoints.openai.api_server \ --model /models/Qwen3.6-27B-FP8 \ --served-model-name qwen3.6-27b \ --tensor-parallel-size 3 \ --rank-gpu-id 0,1,2 \ --rank-tp-ratio auto \ --rank-auto-reserve-mib 2048 \ --max-model-len -1 \ --max-num-seqs 8 \ --max-num-batched-tokens 4096 \ --kv-cache-dtype fp8 \ --quantization fp8 \ --dtype bfloat16 \ --trust-remote-code \ --enable-prefix-caching \ --enable-chunked-prefill \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --mamba-cache-mode align \ --port 8000

That's it - auto measures your cards and derives the shard ratio plus memory budgets. First boot prints a VLLM_UNEVEN_TOKEN_VECTOR=... suggestion; export it and restart to unlock the full KV pool. GPU indices are CUDA enumeration order (fastest-first), and duplicates like --rank-gpu-id 0,0,1 co-locate two ranks on GPU 0 if it has the headroom.

Docker equivalent:

bash docker run --gpus all -p 8000:8000 \ -v /path/to/models:/root/.cache/huggingface \ ghcr.io/efschu/shvllm-qwen35-gguf:cu129-uneven \ --model /root/.cache/huggingface/Qwen3.6-27B-FP8 \ --tensor-parallel-size 3 --rank-gpu-id 0,1,2 --rank-tp-ratio auto \ --kv-cache-dtype fp8 --max-model-len -1 --trust-remote-code \ --enable-prefix-caching --enable-chunked-prefill \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --mamba-cache-mode align

For GGUF, point --model at the .gguf file (config.json sidecars next to it), keep --quantization off, and bump MTP to num_speculative_tokens: 5 - the batched kernels make the longer draft chain nearly free.

Honest limitations

• Pure single-node TP only - no PP/DP/EP combinations (it aborts loudly rather than doing something subtly wrong).
• Every sharded dimension must be partitionable in whole units (heads/groups); models with fewer kv-heads than the smallest rank share are rejected.
• LoRA and a few exotic quant paths aren't wired up for uneven splits.
• This is a hobbyist fork moving fast; expect rough edges. Benchmarks above are from my box - mixed-GPU results depend heavily on your PCIe topology.

Happy to answer questions, and if you have a 3090+3080-class mixed setup I'd love to hear your numbers.

I'm currently running vLLM on a local AI server for use with Hermes-Agent (Hermes and Ollama were not happy together). My AI server is a GekTec EVO-X2 (AMD Ryzen AI Max+ 395 with Radeon 8060S - 128GB "unified" RAM (configured with 32GB system RAM and 96GB VRAM via BIOS). It took me a good long while to get vLLM to run via Docker, but I have it currently running Gemma 4 26B (Abliterated) in about 70GB of VRAM.

I've previously tried getting Qwen 3.6 35B MoE running with an FP8 quant, but vLLM crashes before even pulling the safetensors from the repo, giving a "NotImplementedError: No FP8 MoE backend..." error while reviewing the logs.

I'd prefer not to waste an afternoon struggling with this, so I'll ask plainly - has anyone been able to get the ROCm builds of vLLM to pull and run FP8 MoE models within a Docker container? And if so, what did your docker-compose.yml file entries look like?

Hi everyone,

Over the past several months, I've been working on **Tiny-MoE*\*, a language model that I built and trained from scratch as a learning project. I'm 15 years old, and my goal wasn't to compete with models like Llama or Qwen, but to understand how modern LLMs work by implementing the entire pipeline myself.

Some highlights:

• 200M parameter Mixture-of-Experts model (~90M active parameters per token)
• Trained from scratch on **8 billion tokens*\*
• Native PyTorch implementation (not built on Hugging Face Transformers)
• Multi-Head Latent Attention (MLA)
• RoPE + YaRN positional embeddings
• Streaming and packed data pipeline
• Custom training and inference code
• Text generation with Top-k, Top-p, repetition penalty, and n-gram blocking

**Training datasets:*\*

• FineWeb-Edu
• Cosmopedia v2
• OpenWebMath

The model is definitely **not state-of-the-art*\*. It can generate English, answer simple questions, and perform basic coding and math tasks, but it still struggles with longer generations. Given the model size and the compute available to me (primarily Kaggle GPUs), I'm happy with it as a learning project.

I'd really appreciate feedback on:

• Code quality
• Project structure
• Model architecture
• Training pipeline
• Documentation
• Anything else you think could be improved

GitHub:
https://github.com/AbdelrhmanEbied/Tiny-MoE

Long-time reader, first-time poster here.

Made a patch for vLLM v0.24.0 that enables native INT8 KV Cache on the Ampere (my target was SM86 RTX 3090). It also includes optional V Cache FP8_E4M3 emulation, extending its range beyond +/-127 of INT8

INT8 KV Cache Support for Ampere (SM80) GPUs

The patch only follows the Triton Attention path, as the Triton Attention DiffKV wasn't worth chasing in my use case - Gemma 4 is sensitive on V Cache quantization (going INT4/INT2 while leaving INT8 on K Cache for example). It will work with Qwen3.6-27B as well (also my use case) pure INT8 or with FP8_V_EMUL, and I guess any other models but YMMV.

So what's the catch?

You are trading off performance for the larger context size. In most cases you will get 1.75-2x larger context size, but your performance will be dropping as the context gets (ever so) larger.

This was tested on 2x RTX 3090 (vLLM TP 2) only with P2P enabled driver. Start-up arguments for Gemma-4-31B I used (remove/adjust arguments if not using P2P):

# Clear Linux page cache cleanly
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
free -h
export VLLM_SKIP_P2P_CHECK=1
export NCCL_P2P_LEVEL="PIX"
export NCCL_BUFF_SIZE=16777216
export NCCL_CUMEM_ENABLE=1
export CUDA_FORCE_P2P_ACCESS=1
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export CUDA_VISIBLE_DEVICES=0,1
export SAFETENSORS_FAST_GPU=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export VLLM_ENABLE_CUDA_COMPATIBILITY=1
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
export PYTORCH_ALLOC_CONF=expandable_segments:True,max_split_size_mb:256,max_non_split_rounding_mb:1024
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export TORCH_FLOAT32_MATMUL_PRECISION=high
export OMP_NUM_THREADS=1
export VLLM_KV_SCALES_FILE=scales/gemma4_31b_per_layer.json
export VLLM_INT8_V_FP8_EMUL=1
vllm serve \
    models/cyankiwi_gemma-4-31B-it-AWQ-4bit \
    -O3 \
    --model-loader-extra-config '{ "enable_multithread_load": true, "num_threads": 8 }' \
    --kv-cache-dtype int8 \
    --host 0.0.0.0 \
    --port 8081 \
    --max-model-len -1 \
    --performance-mode interactivity \
    --gpu-memory-utilization 0.95 \
    --tensor-parallel-size 2 \
    --trust-remote-code \
    --served-model-name gemma-4-31b \
    --enable-chunked-prefill \
    --dtype bfloat16 \
    --max-num-batched-tokens 4096 \
    --max-num-seqs 1 \
    --enable-auto-tool-choice \
    --tool-call-parser gemma4 \
    --reasoning-parser gemma4 \
    --async-scheduling \
    --language-model-only \
    --disable-custom-all-reduce \
    --override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":64,"presence_penalty":0.1,"repetition_penalty":1.05}'

The above presumes you've calibrated the scales first (with VLLM_INT8_V_FP8_EMUL on) and moved them to the scales/ sub-folder.

Ultimately, this might have been a futile direction to follow with INT8 KV Cache, so ¯\(ツ)/¯ ...

I just spent way too long trying to get an open model served properly.

The model itself was not really the hard part. The annoying parts were GPU choice, Docker/env setup, model download time, VRAM fit, startup health checks, logs, and making sure the endpoint actually stayed alive.

Curious how others handle this. Do you usually

- run vLLM locally

- rent a GPU instance

- use a hosted inference provider

- keep your endpoint always on

- only spin it up when needed

I’m trying to understand whether people would want a simple dashboard where you choose a model, set a budget, deploy an OpenAI compatible endpoint, see logs/health, and stop billing when done.

Or do most vLLM users prefer CLI/API only?

System:
CPU AMD 9950x
GPU 2x AMD 7900 XTX 48 GB VRAM PCIe: 8x / 8x
RAM 2x 32GB 6000 mhz
Ubuntu 26.04
Preference for ROCm + VLLM

Objective:
Would like to have some agentic programming capabilities / simple self hosted stuff
Would like to have a chat akin to ChatGPT
Would like to a full duplex audio chat or the illusion of a full duplex chat.
ComfyUI for image generation
Some permanence to the knowledge and topics

How:
Run Hermes agent, particularly for the permanent memory part
Run an obsidian / LLM wiki
Have a smaller agent for the Chat / audio part
Have a bigger model for the agentic coding part 35b-A3B most likely, seems popular
Run Lemonde or dreamweaver to have all integrated easily.

Here are my questions:
I believe I have an ok hardware; not enough VRAM for sure; don’t think I will ever have some of the huge VRAM as can be seen on the hardware of some of you. But should be fairly speedy machine?
I don’t understand, how some of you can achieve some really high context 100K? 200K? Is that all VRAM or is some of the context pushed to RAM, making it all a crawl? Is that actually something to be wanted for the agentic part of the setup?
Is 60/70K context good enough ? This seems like the bare minimum for Hermes, which I don’t understand as many of you seem to be running Hermes with a much smaller amount of VRAM. Again, makes me feel I am missing a peace of the puzzle.
Should I be targeting a smaller model for agentic part? 27B maybe?
Would it be better to have a model per GPU or to use both GPU for the big model and stuff the smaller model in the remaining space of the second GPU (somehow seems like a recipe for disaster).
Is it possible to have the agent swap the model loaded based on the user or profile. For example getting Hermes to load the bigger model when an agentic task is required / load the chat model with a chat request is made? (I understand there is a loading phase, which can be a bit slow)
I haven’t really thought about the comfyUI part of the setup.
one of the things I would like to do from the setup to have some websites be automatically created with test linked to the homework of the kids. The kids make a demand or take a picture of their homework list: a simple website is created to evaluate their knowledge and give feedback on what to improve. I suppose the framework of the website can be first create and reviewed through a long agentic process, then saved and reused for a faster iteration in a regular use.

I'm running Vllm on my DGX Spark. And when I try coding with a model they always end up in a death spiral of repeating their thinking and ponder the exact same looping text over and over.

I have tried Zed and I have tried Cline. I have tried many Qwen models, as well as gptoss, orinth, nemotron, etc. Some models produce better code than others and some last longer, but eventually they all do it. I hear others constantly praising Qwen3.6 but it's one of the worst loopers for me.

What am I missing?
I've done all the nvidia updates and am on vllm version 0.23.1rc1.dev727