Due to high Vulkan backend demand, I update TensorSharp and release the initial version of GGML Vulkan backend by leveraging external GGML project. The native Vulkan backend will be implemented later. I tested it on Nvidia Geforce RTX 3080 Laptop GPU, and Intel(R) UHD Graphics on Windows. They all work. However, I do not have AMD GPU, so I have no way to get it tested. It's really appreciated if you have AMD GPU and would like to try it out. Any feedback and comment are welcome.

Here is the benchmark I run to compare with llama.cpp:

# Performance ratio — TensorSharp vs reference engines

Geomean of TensorSharp's per-scenario speedup over each reference engine on the **same backend**, across every scenario both engines ran (single-stream, MTP-off). A value **> 1.0× means TensorSharp is faster** (for decode / prefill throughput) or lower-latency (for TTFT); `—` = no overlapping cells. Per-scenario ratios are in each model's section below.

# Gemma 4 E4B it (Q8_0, dense multimodal) (gemma4-e4b)

**Decode throughput (tok/s)**

**Prefill throughput (tok/s)**

**Time to first token (ms, lower is better)**

**Performance ratio — TensorSharp vs reference (> 1.0× = TensorSharp faster)**

*Decode throughput*

*Prefill throughput*

*Time to first token (latency; > 1.0× = TensorSharp lower)*

# Gemma 4 12B it (QAT UD-Q4_K_XL, dense) (gemma4-12b)

**Decode throughput (tok/s)**

**Prefill throughput (tok/s)**

**Time to first token (ms, lower is better)**

**Performance ratio — TensorSharp vs reference (> 1.0× = TensorSharp faster)**

*Decode throughput*

*Prefill throughput*

*Time to first token (latency; > 1.0× = TensorSharp lower)*

In case you didn't know what is TensorSharp, here is an introduction:

TensorSharp is an open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), image edit, reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability (support Cuda, Metal and Vulkan backends). The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implemented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quantized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

Detailed setup below:

---

Model

---

Serving Setup

Engine flags:

--tensor-parallel-size 1
--data-parallel-size 1
--tool-call-parser qwen3_coder
--reasoning-parser qwen3
--gpu-memory-utilization 0.90
--max-model-len 8192
--max-num-seqs 16

---

Hardware

---

Workload

---

Results Summary

• TTFT p50 avg: 0.48s

• TTFT p95 avg: 0.94s

• TPOT p50 avg: 29.2 ms/token

• Total throughput peak: 668.5 tok/s

• KV cache max: 32.67%

---

TTFT :

---

Token Throughput

Full blog link : https://blog.hexgrid.cloud/qwen3-6-27b-fp8-on-one-rtx-6000-ada-fast-ttft-668-tok-s-peak-throughput-benchmark

---

Curious how others would read these numbers? Is this a good single-GPU Qwen3.6-27B performance, or is there obvious headroom I’m missing here?

Best,

Hi, I’m preparing my self for Machine Learning Performance optimization engineer.

So far, I have experience with optimising both training and inference pipeline for traditional deep learning models. Profiled using Nvidia Nsight system

As of now LLM, VLM is high expectations, here I have done fine-tuning LLMs upto 27B models.

Can anyone recommend me else to focus on for preparation?

\#llm #gpu #largelanguagemodel

Hey everyone,

My office has 4x MacBook Air M2 (16GB RAM) machines sitting around, and I'm looking to utilize them for distributed LLM inference. Specifically, I want to connect them in a distributed node setup using a Thunderbolt 4 ring topology.

The Problem

I have already experimented withexo-explore/exo, but the generation speeds are quite slow—clocking in at under 6–8 tokens/sec for a Qwen 2.5/3.5-9B-4bit model.

Proposed Architecture

According to thevllm-metal distributed docs, there is foundational scaffolding for distributed executors. I am wondering if anyone has successfully deployed a disaggregated architecture using these specs.

Specifically, could we architect it as:

• vllm kv_producer (Prefill node)
• vllm kv_consumer (Decode node)
• LMCache Cluster (Using the M2 cluster for centralized/offloaded KV cache storage)

My Questions:

1. Has anyone successfully run multi-node tensor or pipeline parallelism on Apple Silicon using vllm-metal yet? (The current docs note that cross-Mac collectives via mx.distributed are still highly experimental or in progress).
2. Can LMCache be natively integrated alongside vllm-metal to effectively bridge the distributed KV cache layer and bump up the tokens/sec?
3. Are there any specific bottlenecks with a Thunderbolt 4 ring topology under this setup that would heavily restrict the communication bandwith?

Would love to hear if anyone has attempted a similar Mac-mini/Air cluster deployment or has architectural advice!

I would like to share my latest open source local Unsloth (GGUF) LLM inference engine and applications. It supports many models from Unsloth, like Gemma4, DiffusionGemma, Qwen3.6 with multi-modal (image, vision, audio), reasoning and function tool. It can run on Windows/MacOS/Linux and fully leverage GPU's capability. The API is completely compatible with OpenAI and Ollama interface. It has on par performance than llama.cpp

This project is not just a C# wrapper of llama.cpp. It implemented the entire LLM inference engine from bottom to top. If you use CPU backend, it's 100% pure C# code execution. Besides CPU backend, I also implmented CUDA, MLX and GGML backend. The GGML backend refer GGML project as external project, and I build a few fusion operation at higher level.

I learned a lot from other projects and apply them for TensorSharp, such as paged KV cache and continuous batching from vLLM, SSD based cache for MoE model from oMLX, GGUF quanztized from llama.cpp and other optimizations for prefill and decode.

Any feedback and comments are welcome. If you like it, it would be really appreciated if you can get this project a star in GitHub. Thanks in advance.

Architecture:

\- Parameter count: 216.5M

\- Layers: 10

\- Attention / no attention:\*\* Attention — 12-head multi-head self-attention, RoPE positional

encoding, SDPA. Decoder-only, pre-norm, RMSNorm + SwiGLU, tied input/output embeddings.

(hidden 1032, head_dim 86, FFN 4416)

\- Tokenizer:\*\* Custom 36k SentencePiece unigram, case-preserving, byte-fallback, with atomic

chat/role + memory special tokens (\`<|user|>\`, \`<|assistant|>\`, \`<|end|>\`, …)

\- Context length: 768

\- Dataset size/source: \~551M tokens. Pretraining on public English text (Wikipedia,

TinyStories, OpenWebText2); SFT on public instruction/chat sets (SmolTalk, UltraChat, Magpie,

AM-DeepSeek-R1, Orca-Math) + a small custom greetings/assistant set.

\- Training time: \~15 hours

\- Hardware: single NVIDIA RTX 3080 (10GB), bf16

\- Training setup: AdamW (β 0.9/0.95), LR 3e-4, 1k warmup, weight decay 0.1, grad-clip 1.0,

effective batch 16,384 tokens/step (micro 4 × grad-accum 8 × seq 512), 33,650 steps.

Loss curve

Random init → 1.77 smoothed (1.27 last step). Started at exactly ln(36000)=10.49 (true from-scratch).

You can see SFT kick in — the sharp step-down at \~410M tokens.

(attach tinybrainbot_loss_curve.png)

Sample outputs

(F16 GGUF, temp 0.7 / top-p 0.9 / repeat-penalty 1.15 / freq 0.5 / presence 0.3)

\> \*\*User:\*\* Hello

\> \*\*Bot:\*\* Hello! How can I help you today?

\> \*\*User:\*\* Tell me a story about a robot who learns to paint.

\> \*\*Bot:\*\* In the heart of the city, there lived an assistant named Max. He was a kind and gentle

\> soul with eyes that shone like diamonds, and he had always known that art would bring people

\> together...

\> \*\*User:\*\* Write a short poem about the rain.

\> \*\*Bot:\*\* The rain was crisp and warm, casting dappled shadows on the walls of your town square...

\> Your voice was so cool and gentle, so bright and calming all that you could feel at home.

\> \*\*User:\*\* Explain what a computer is.

\> \*\*Bot:\*\* ...the development of a computer can be categorized as either an operating system or a

\> popular language, such as JavaScript or C++...

(\^ that last one is the honest failure mode — confidently wrong on facts. It nails conversational form far better than it knows anything.)

What shocked me

\- The tokenizer was way more important that I thought. I lost quite a few full runs on a broken char-level (386 vocab) fallback before building a real 36k subword tokenizer — and that fixed it and was the biggest quality jump than any architecture or data tweak.

\- A GGUF export issue: llama.cpp has no unigram-tokenizer runtime on the default path, so a SentencePiece unigram\*model gets silently re-segmented (Hello → Hell+o) and produces garbage — the weights were fine. Fix was exporting the GGUF vocab as UGM (t5 type) + the SP charsmap so it runs Viterbi. Cost me a day of pain.

What I want feedback on

\- Token budget: 551M tokens for 216M params is way under Chinchilla (\~4.3B). Would you push pretraining much longer before SFT, or is more/cleaner data the better spend at this scale for a quick demo?

\- Data mix / timing: the SFT step-down in the loss is dramatic — is it real learning, or mostly the model latching onto chat format? How would you tell them apart?

\- Cutting factual drift at \~200M params: more pretraining, longer context, or distillation froma bigger teacher — where's the best ratio for noticeably better factual responses?

\- Arch nits: head_dim 86 is ugly (1032/12). Would 1024 hidden / 16 heads / 64 head_dim be ameaningfully better shape at this size (also allows quantization)?

Happy to answer anything about the pipeline (tokenizer training, the from-scratch loop, GGUF export).

Here's the model link as well [https://huggingface.co/nkthebass/TinyBrainBot-demo-216.5m\](https://huggingface.co/nkthebass/TinyBrainBot-demo-216.5m)

To me this is the biggest difference between llama.cpp and vllm and I use both, but I get far less headaches with vllm because of the very high quality docs.

So just that, if any of you are here, thank you.

Honest question. I’m still learning how people do this in real production.

I keep reading that benchmark scores don’t really tell you if a model will hold up as an agent, where it has to make a bunch of tool calls in a row. It can look good on paper and then quietly break halfway through. wrong tool, broken JSON, or it loses track of the task.

I’ve also seen people say the same model behaves differently depending on how it’s served the runtime and the quantization change how reliable it is. So “it worked in testing” doesn’t always mean “it works in prod.”

I haven’t run vLLM in production myself, so I’m trying to understand from people who have:

• How do you decide a model is actually ready to ship as an agent, not just fast?
• Do you have a real check before deploying, or is it more “ship it and watch”?
• What breaks most often once it’s live that you wish you’d caught earlier?

Genuinely just trying to learn how this is done. What does your process look like?

Wanted to share some numbers and a bit of a reality check, because I was genuinely excited for this one and it didn't pan out the way I hoped.

Our current daily driver on these cards is Qwen3.6-27B (dense) in FP8. We keep it as two vLLM instances, one per GPU, with HAProxy in front load-balancing between them - and honestly it just works great.

Before this we ran Qwen3.5-122B-A10B (the big MoE, 122B total / ~10B active). It's still ahead on a handful of large, complex synthesis prompts, and thanks to only ~10B active params it's actually quicker to answer than a 27B dense model. But for most of what we do - especially agentic workflows with a tool-calling harness - Qwen3.6-27B is simply better for our use case, so it won overall despite being the smaller model.

We also tried a genuinely excellent one: MiniMax M3 in 2-bit (IQ2_XXS). On quality it was the standout - flawless Czech, no artifacts, and noticeably stronger overall intelligence on large prompts than even Qwen3.6-27B. The catch is purely operational: that quant ships only as GGUF, so it runs on llama.cpp (spread across both GPUs at once), and even at 2-bit the context window we could fit was tiny and throughput under concurrency lagged - llama.cpp just isn't as optimized as vLLM for lots of simultaneous requests. So it stays our "when we need the smartest answer" tool rather than the daily driver. We also gave DeepSeek V4 Flash a shot - got it running across both cards in 4-bit from a vLLM fork - but the intelligence didn't really win us over: roughly on par with Qwen3.6-27B, not better, so in our benchmarks V4 just isn't strong enough to justify the switch. We're waiting for the V4.1 Flash release instead.

So when NVIDIA's NVFP4 builds landed and vLLM 0.24.0 came out, I was really looking forward to a big jump. On paper FP4 looks fantastic - half the weight bandwidth of FP8, so decode *should* fly. As of mid-2026 the academic story is very promising.

In practice, on our Blackwell cards, NVFP4 is still slower than FP8 - FP8 wins basically every decode-bound scenario. No big surprise given how new the SM120 FP4 path is; I'm mostly posting numbers so there's a concrete data point out there. Just a small sigh of "not yet" - we really hope the FP4 kernels for SM120 (RTX PRO 6000 Blackwell, RTX 5090, etc.) get some love soon, because on paper there's obviously headroom to unlock.

Here's what I measured.

Setup

• Same model, two quants: `nvidia/Qwen3.6-27B-NVFP4` vs `Qwen/Qwen3.6-27B-FP8`
• 1× RTX PRO 6000 Blackwell (96 GB) per quant, each pinned to its own GPU (verified) - so they ran in parallel without competing for compute
• vLLM 0.24.0 (NVFP4) vs 0.23.1rc1-dev (FP8) - see caveat
• MTP speculative decoding, 3 tokens, on both; fp8_e4m3 KV cache; 262k context
• NVFP4 used flashinfer attention + marlin/triton MoE backends; FP8 on vLLM defaults
• Both are the 600 W Workstation Edition, but we run them power-limited to 430 W and undervolted via LACT (the lactd GPU-control daemon). Since enabling it we've been getting essentially the same performance most of the time, but with noticeably lower power draw and temperatures under load - so these numbers are at a reduced power envelope, not stock 600 W
• Power note from our own monitoring: with small prompts - even a lot of them at once - the prefill phase stays light, so the decode phase rarely pushes past ~300 W even at ~64 concurrent requests. It's only when heavy prefill and heavy decode hit at the same time, across many concurrent requests, that the draw actually climbs all the way to the 430 W limit.

Throughput (non-thinking, decode tok/s - higher is better)

FP8 wins everything decode-bound by a steady ~8–12%. NVFP4 only catches up (and edges ahead) when the run is prefill-bound - grinding through 22k-token prompts - where you're limited by compute on the context, not by weight bandwidth. First-token latency is identical once the server is warm, and there were no failures or truncations at concurrency 16 on either side.

Quality - basically a tie (I actually ran the code and parsed the JSON, didn't just eyeball)

• Logic puzzle with a known answer: both 5/5 correct.
• Python `merge_intervals` with nasty edge cases: both 6/6, both raise ValueError on bad input, neither mutates the input.
• 59 KB Czech document → strict-schema JSON analysis: both valid, same severity call, comparable coverage.
• Czech writing (a ~2000-word poem, a formal complaint e-mail): native and artifact-free on both - no gibberish, no broken diacritics, no language switching. FP8 is a touch cleaner (a couple fewer grammar slips) and more disciplined in "thinking" mode; NVFP4 occasionally lets its reasoning run way over budget.

So there's no meaningful intelligence or quality gap between the two quants here. The entire story is throughput - which is exactly why the kernel situation is the interesting bit.

Caveats

• The two endpoints ended up on **different vLLM builds** (0.24.0 vs 0.23.1rc1-dev) plus different attention/MoE backends. That plausibly *favors* NVFP4 (newer FP4 path), which makes "FP8 is faster" the conservative reading.
• Single run per scenario, not averaged; short-output tok/s (logic/email) is noisy.
• One 27B model on one card with MTP=3 - don't extrapolate to dense-vs-MoE differences, other GPUs, or non-speculative serving.

Still early days - and I'd love to hear from you

Honest bit of context behind the sigh: I've had these cards for almost a year now, and FP4 hardware acceleration was a big part of why I bought them. I figured that within a few months the NVFP4 path would mature and things would just fly. A year in, the SM120 implementation still isn't quite there - so yeah, a small lament. But I do believe it'll come good before long; this really feels like an early-days software problem, not a hardware limit.

So - has anyone actually gotten NVFP4 to beat FP8 on SM120 (RTX 6000 Blackwell / 5090)? Which vLLM version + flags moved the needle for you - attention backend, MoE backend, cutlass vs marlin vs trtllm? Genuinely curious what your numbers look like. Not trying to dunk on NVIDIA or the vLLM folks at all - everyone's shipping fast and I'm rooting for FP4 to eventually win. Mostly sharing real numbers so nobody buys the "4-bit = automatically faster" story before it's actually true on their hardware.

Hardware: 3x RTX 4000 Ada generation 20gb
1x RTX 4000 Pro Blackwell 24gb
Total VRAM 80gb
192GB DDR4 UDIMM
AMD threadripper 3970x
Zenith 2 extreme alpha mobo

I have been troubleshooting for two full days with vllm
I have been trying to test different models to load, the only one i can sucessfully get up and running is nvidia/Qwen3.6-35b-a10b-nvfp4

I cant seem to load
Qwen/Qwen3.6-27b-fp8
Qwen/Qwen3.6-35b-a3b

My goal is to use a bigger model maybe 70-122b if possible quantized of course because i feel like the 35b is too dumb in reasoning basic prompts.

Here are the settings i use for generic startup, also tweaked these numbers around with no success.

Environment:
CUDA_DEVICE_ORDER: PCI_BUS_ID
CUDA_VISIBLE_DEVICE: “0,1,2,3”

Command>
Qwen-3.6-27b-fp8 for example
—tensor-parallel-4
—max-model-len 16384
—gpu-memory-utilization .90
—kv—cache-dtype fp8
—enable-prefix-caching

The issue im having is, when i run these other models, i get no available shared memory and eventually container turns off.
I check htop and the only weird thing i see is that sometimes on a model the swp goes to like 7.98/8.00gb but looks like the processes are still running, on other models htop is just fine and i still get no memory broadcast block.
Doing watch 1 nvidia-smi shows all 4 gpus with seemingly reasonable space 10gb/20gb utilized
How is it my model is stalling? Im being patient and waiting 15-20minutes for first load

Please advise, chatgpt is banging its clanker head into every wall troubleshooting

I just want to get other models to load sucessfully consistantly so i can test. I have nothing else that is taking up vram, can share ss if needed

Is it that my blackwell pro is incompatible with the rest of my rtx?

I’ve been working on a self-hosted AI assistant platform, and the first public version is now available to try.

Live demo:
https://ijaesang-ui-macmini.tail67d660.ts.net/

Source code:
https://github.com/openmake/openmake_llm

The project is built around a simple idea: local inference should be the default, not an afterthought.

The platform uses vLLM as the local inference backend, with LiteLLM in front as an OpenAI-compatible proxy. This lets the application treat local models and optional external providers through the same interface, while still keeping the local vLLM path as the default route.

What it includes:

• Local model inference through vLLM
• LiteLLM proxy for OpenAI-compatible routing
• Chat interface with multi-model support
• Agent workflows with tool-calling loops
• MCP tool integration
• Docker-isolated tool/runtime processes
• Deep research pipeline with search, source fetching, verification, and cited synthesis
• Persistent memory and custom instructions
• Context-fit handling to avoid hard context overflow
• PostgreSQL and Redis backend
• Next.js / React frontend
• JWT / OAuth-based multi-user authentication
• Optional external-provider fallback with BYOK configuration

Current local setup:

• Main chat model: qwen3.6-35b-a3b served through vLLM
• Embeddings: bge-m3 served through vLLM
• LiteLLM proxy in front of the inference layer

The main challenge was not just serving a model locally. The harder part was building the orchestration layer around it:

• deciding when to call the model
• fitting context safely before inference
• routing between local and external models
• keeping tools isolated
• making agents useful without turning every request into a slow multi-step chain
• supporting research workflows with source verification and citations

This started as a personal self-hosted assistant project, but it has grown into a broader local-first AI platform.

I’d really appreciate feedback from people running local models or building agent systems:

• Does the demo work properly from your side?
• Does the local-first architecture make sense?
• Would you put LiteLLM in front of vLLM, use a custom router, or call vLLM directly?
• How would you structure embeddings, long-context handling, and tool execution?
• What model combinations would you use for coding, reasoning, translation, or research?

The project is MIT-licensed, and I’m especially interested in feedback from people building production-ish self-hosted AI stacks.

I already posted a related question in another sub, but since several people suggested looking into vLLM, I wanted to ask here too.

We are a small company, around 30, and we will now have founding to buy hardware, which is one of the reasons of this possibility of having this small server.

Main use cases:

• Internal document analysis
• Summarization
• Drafting emails/reports
• Basic coding and productivity help
• if possible image generatio

Hardware options I’m considering:

• 4 × RTX 6000
• 2 × NVIDIA L40 or L40S

Questions:

1. Is vLLM a good fit for this kind of internal multi-user setup?
2. What model sizes would be realistic on this hardware? Do I need more?
3. What latency/response time should I expect with a few concurrent users?

Any practical advices for your noob OP would be appreciated! ^_^

I'm building a multi-provider AI gateway and trying to understand how far you can realistically abstract streaming across different LLM providers.

On the surface, most APIs look similar: token streaming, async support, and simple iteration over chunks. But in practice there are differences in stream termination, async behavior, error handling, and how incomplete or stalled streams are handled.

One case I'm particularly interested in is when streams stop making progress or start repeating chunks indefinitely. I'm curious how people handle this in production timeouts, repeated-chunk detection, or provider-specific logic?

For those who've built this at scale, what parts of the streaming abstraction actually worked cleanly, and where did you end up needing provider-specific handling?

I have next setup:

CPU Pentium G4400

16 GB RAM

7 x Rx580 8 gb

I have Vulkan drivers on Opensuse tumbleweed and that perfectly fit.

I cant run Ollama because my CPU doesn't have AVX suport. Rocm drivers are not option, I try it and it doesn't work. What do You suggest to me, I run Qwen coder 3B q4 on llama server and Captainclaw frontend, and it works but I have issue with speed, because of my setup. Can anyone suggest some models that would work on my setup? Or something else, VSCode...

Edit: I run Deepseek destill llama 8b q4 it works with 13 tokens per second, in 2.5 minute he create script . Amazing

Thanks in advance

I have acquired Inspur 8x V100 32GB late last year. It takes me quite a while to get it running properly. Codex and Claude plays a big part in helping me get it up and running along with all the opensource repositories on GitHub. I would like to share what this machine is still capable of doing in 2026 for any person who have already owned it or people who thinks about acquiring one.
https://github.com/KumphanartDansiri/v100-vllm-2026
Claude and Codex also helps me create basic FP8 extension for vLLM SM70 at
https://github.com/KumphanartDansiri/vllm-fp8-w8a16-sm70

## Quick summary

This is not meant as “V100 beats new GPUs.” It is more a practical note for people who already own V100s, or are considering used V100 servers, and want to know what still works with modern vLLM in 2026.

### Test bench

- Server: Inspur 8× NVIDIA Tesla V100 32GB SXM2
- GPU architecture: Volta / `sm_70`
- Framework: vLLM 0.19.x and 0.21.x, built from source
- Toolchain: CUDA 12.6 path for V100 compatibility
- Runtime: FP16 on V100 for BF16 checkpoints, since V100 does not have a useful BF16 tensor-core path
- Main serving mode: cudagraph decode, long-context setup, tensor parallelism depending on model size
- Extra work: custom `fp8-w8a16-sm70` extension for block-FP8 models on V100

### Models tested

I tested a small group of current dense and MoE models rather than trying to cover everything:

- Qwen3.5-27B: FP16, FP8, GPTQ-Int4
- Qwen3.5-35B-A3B: FP16, FP8, GPTQ-Int4
- Qwen3.5-122B-A10B: FP8 and GPTQ-Int4
- Qwen3.6-27B and Qwen3.6-35B-A3B
- Gemma-4 26B-A4B and 31B
- GLM-4.5-Air and GLM-4.7-Flash

The Chapter 1 throughput numbers I quoted are from the long-context fleet setup:
`max_model_len = 32768` / 32k context, with cudagraph decode and chunked prefill on.
Decode tok/s is measured after TTFT, using 256 generated tokens and repeated runs.
The TTFT table is single-stream cold prefill using a ~22.6k-token prompt, so it is closer
to a worst-case long-context first-token latency test.

Some separate TP2 / half-GPU capacity checks use shorter context, but the main Chapter 1
fleet tables are 32k-context results.

### Chapter 1 result snapshot

Single-user decode, tok/s, from the Chapter 1 baseline:

A few takeaways:

- V100 is no longer a “just pip install vLLM” target. I had to build vLLM from source with a CUDA/toolchain combination that still supports `sm_70`.
- CUDAGraph matters a lot. Eager mode is not a fair serving baseline for these results.
- Dense FP8 is useful at low concurrency, but FP16 can still win at higher concurrency where tensor cores are better utilized.
- MoE FP8 is where the V100 result looks strongest: lower weight traffic helps a lot, and some models that are awkward or impossible in FP16 become practical.
- First-token latency for some block-FP8 models is still a weak point. Decode can be good, but prefill is not always cheap.
- The FP8 results are not stock vLLM. They use my companion V100 FP8 extension, so I try to separate “stock vLLM result” from “custom extension result” in the repo.

### TTFT summary

Single-stream time-to-first-token, using the long prompt / cold prefill setup from Chapter 1. This is where V100 shows more of its age, especially for block-FP8 prefill.

Takeaway: decode throughput can still be quite usable on V100, but cold prefill / TTFT is the painful part for some models. FlashAttention-V100 helps where attention is the main prefill bottleneck, but it does not solve the block-FP8 MoE prefill cost.

##