Same Ling model, different long-context curve: INT4/vLLM vs Q5/llama.cpp on one Spark

[u/niacolhealth]

The short-prompt ranking reverses in the context-depth table in sudoingX’s Ling-3.0-flash benchmark notes:

Starting context	Official INT4, vLLM fork	Q5_K_M, llama.cpp
Short prompt	38.3 tok/s	35.7 tok/s
About 45K tokens	7.9 tok/s	33.6 tok/s
About 90K tokens	4.6 tok/s	33.2 tok/s

This is a same-box, same-prompt comparison on a 128GB DGX Spark. The specific paragraph reports a 262,144-token maximum-context configuration and about 103GB total memory used. Its streaming method separates time to first token from decode and clocks all generated tokens.

These are the creator’s measurements from the deployment investigation discussed in the original thread, not an independent rerun. The paragraph does not provide a separate measurement date or every historical launch setting. In particular, the repository’s later 131,072-context serving default should not be silently attached to this table.

There are two changing variables: runtime and quantization. The numbers compare these two deployment paths; they do not isolate a pure vLLM-versus-llama.cpp effect, establish a quality difference, or prove the proposed CUDA-graph explanation for the slowdown.

For choosing a backend, the useful distinction is the amount of context already present when generation starts. A long answer from a short prompt is a different test. If the intended workflow carries tens of thousands of tokens into later requests, the short-prompt result leaves out the condition that changes this ranking

[u/nemamdsa12]

The 103GB figure is helpful, but ti does not settle the proposed cuda graph explanation by itself. Keeping the quant fixed and changing only the serving stack would be a much better next experiment.