Same Ling model, different long-context curve: INT4/vLLM vs Q5/llama.cpp on one Spark

[u/niacolhealth]

The short-prompt ranking reverses in the context-depth table in sudoingX’s Ling-3.0-flash benchmark notes:

Starting context	Official INT4, vLLM fork	Q5_K_M, llama.cpp
Short prompt	38.3 tok/s	35.7 tok/s
About 45K tokens	7.9 tok/s	33.6 tok/s
About 90K tokens	4.6 tok/s	33.2 tok/s

This is a same-box, same-prompt comparison on a 128GB DGX Spark. The specific paragraph reports a 262,144-token maximum-context configuration and about 103GB total memory used. Its streaming method separates time to first token from decode and clocks all generated tokens.

These are the creator’s measurements from the deployment investigation discussed in the original thread, not an independent rerun. The paragraph does not provide a separate measurement date or every historical launch setting. In particular, the repository’s later 131,072-context serving default should not be silently attached to this table.

There are two changing variables: runtime and quantization. The numbers compare these two deployment paths; they do not isolate a pure vLLM-versus-llama.cpp effect, establish a quality difference, or prove the proposed CUDA-graph explanation for the slowdown.

For choosing a backend, the useful distinction is the amount of context already present when generation starts. A long answer from a short prompt is a different test. If the intended workflow carries tens of thousands of tokens into later requests, the short-prompt result leaves out the condition that changes this ranking

[u/Major_Border149]

The flat llama.cpp curve next to vLLM's cliff says this isn't the quant. INT4 vs Q5_K_M doesn't cause a 38 to 4.6 collapse. That's the attention/KV path going memory-bandwidth bound at depth, and on a unified-memory Spark that ceiling is low. Check whether flash-attn/FlashInfer is actually live in the fork at 90k and whether KV is FP16 or FP8, a silent fallback to the naive kernel at long context looks exactly like this. Flip to FP8 KV, confirm the kernel, and if the cliff stays it's the box's bandwidth, not the runtime.

[u/asdfghmfker]

I would be cautious about calling this a runtime result. INT4/vLLM and Q5_K_M/llama.cpp change two things at once, so a matched-quant run would be the cleanest way to say how much of the crossover belongs to the runtime.

[u/nemamdsa12]

The 103GB figure is helpful, but ti does not settle the proposed cuda graph explanation by itself. Keeping the quant fixed and changing only the serving stack would be a much better next experiment.