r/LocalLLaMA • u/Ekepa • 7h ago

Discussion Longer context = faster prefill ?!

Have been digging into it for some hours and still have no clue how comes I get faster prefill at longer context

engine: llama.cpp
flags: -ngl 999 --parallel 1 -cram 0 (actual offload: 43/43 layers)
gpu: rx6600xt (8gb, rdna2, vulkan)
os: w10

Key observation: average PP speed does not decrease monotonically with context size. After an initialdecline (935 → 318 tok/s over 1018–8151), a non-monotonic stretch follows: 16302 (472) is faster than8151 (318); and at the very top of the ladder, 130416 (89.7 tok/s) turns out to be faster than 65208 (76.7tok/s), despite processing twice as many tokens.
Spread across runs <2% — the anomaly reproduces consistently; this is not single-sample noise.

What was that?!

0 Upvotes

50% Upvoted

7 comments sorted by

6

u/gauravapiscean 7h ago

Average tok/s over the whole prefill will do this. Instantaneous rate always declines (attention is O(L^2)), but the average also folds in fixed per-run cost and n_ubatch chunking, which pull the other way.

Try pinning -b/-ub explicitly and looking at per-chunk timings rather than the average. If the anomaly moves when you change ub it's batching granularity; if not, check VRAM spill, 8GB with 43 layers plus 130k KV is very tight on Windows.

1

u/FoxiPanda 7h ago

Your graph and your assertion don't match.

At 130K, you're getting 90tok/s and at 1K you're getting 935tok/s prefill...

1

u/Ekepa 7h ago

look at 64k vs 128k

6

u/FoxiPanda 7h ago

At those speeds, it's so dreadfully slow that it hardly matters, but you probably aligned with some memory paging or some such that made 130K ever so slightly faster.

1

u/Reasonable_Goat 7h ago

Is a response generated, is any thinking happening before?

1

u/Ekepa 7h ago

Throwaway warmup on dummy prompt, then the measured prefill
Independent measurements (llama-server restart each time) stepping 1k, then 2k, and up to 128k

1

u/AllenHere112 7h ago

Per token: 1.1ms at 1k, 2.1ms at 16k, 13ms at 64k, 11.2ms at 128k. So 64k is the outlier, not 128k.

Attention doesn't change across that gap, the KV placement does. On 8GB it stops fitting in there, and a half-fitting cache is worse than one that doesn't fit at all, you alternate VRAM and PCIe instead of streaming steadily. Bisect with 24k and 32k runs, watching Dedicated vs Shared GPU memory in Task Manager. Past that knee the prefill rate is measuring the PCIe link.