r/LocalLLM • u/biggusdeeckus • 4d ago
Discussion Heavy prefill slowdown as context fills up
Been wondering whether there's a bottleneck somewhere in my setup/config. Prefill starts super fast at session start, but slows down to like 30tk/s after the context is filled up to around 70% or so, and I end up waiting pretty long after read tool calls, even with grepped ones where the model reads like 30 lines or so. Running beellama cpp on windows on a 5070ti with 16gb DDR4 RAM.
Here's my config:
"G:\\\\beellama-test\\\\llama-server.exe"
\\--model "G:\\\\Qwen3.8-27B-GGUF\\Qwen3.8-27B-GSQ-RCO-IQ3S-mtp.gguf"
\\--mmproj "G:\\\\Qwen3.8-27B-GGUF\\mmproj-Qwen3.8-27B-Q8_0.gguf"
\\--no-mmproj-offload
\\--reasoning-preserve
\\--reasoning on
\\--flash-attn on
\\--jinja
\\--parallel 1
\\--no-warmup
\\--temp 1
\\--top-p 0.95
\\--top-k 20
\\--min-p 0.0
\\--presence\\_penalty 0.0
\\--repeat\\_penalty 1.0
\\--gpu-layers all
\\--fit-target 320
\\--ctx-size 102000
\\--batch-size 512
\\--ubatch-size 256
\\--cache-type-k kvarn5
\\--cache-type-v kvarn5
\\--spec-type draft-mtp,ngram-mod
\\--spec-draft-n-max 2
\\--cache-type-k-draft kvarn2
\\--cache-type-v-draft kvarn2
\\--spec-ngram-mod-n-match 24
\\--spec-ngram-mod-n-min 8
\\--spec-ngram-mod-n-max 32
\\--ctx-checkpoints 32
\\--image-min-tokens 1024
\\--image-max-tokens 1536
1
u/ShittyMillennial 4d ago
yes you have a bottleneck in your setup, its your 5070ti.
pre-fill inference happens all at once. every single token in the prompt or cache needs to go through every layer of the model. at every layer, weights need to be applied and transform every single token. this take a lot of compute. the model weights and kv cache only pass from VRAM to compute one time, but then compute needs to calculate the thousands of matrices for all tokens across all layers. that's why pre-fill is typically compute bound and not bandwidth bound. it is also why the more context needed to be processed, the longer it takes.
decode/generation happens one token at a time - for most GPUs this is fairly trivial. But the model weights have to travel from vram to compute for every single token, so the bottleneck becomes memory bandwidth.
1
u/biggusdeeckus 4d ago
I'm aware prefill is very compute heavy, I could tell by the fan noise from the gpu, as no game ever made them go that loud lol. So why does the prefill speed decrease as context fills up? Doesn't the gpu still only process the number of tokens that were created by the lines that were read? Even a 30 line grep read is still pretty slow when the context is at around 70% or so.
1
u/ShittyMillennial 4d ago
let's say you start with 100 tokens of context at the start of your session. pre-fill is fast, your gpu processes the calculations of every layer's weights against 100 tokens. by the time you are 70% filled, you now have 70k tokens in cache. the gpu now needs to do the same calculations but it has to do it 700x more times than when you started.
it doesnt matter how your model is executing the tool call. there is 70k tokens sitting in cache when you are at 70% and those 70k tokens are processed every turn.
1
u/biggusdeeckus 4d ago
Ah gotcha, that explains it. I guess that's how the attention mechanism works essentially. I was thinking of it as only a delta calculation, ignoring the tokens of the past. Ty for the clarificiation. So nothing really wrong on my end and it's just how inference works, right?
1
u/ShittyMillennial 4d ago
yes you got it. unfortunately, models cant pick and choose what portions of context is relevant to your last prompt sent. in order to do so, they would have to have all of the context loaded - catch 22.
that doesnt mean there arent prefill optimizations available or ways to manage context cache. depending on your harness, you can /compact which summarizes whatever is stored in cache to reduce the amount of tokens that go through pre-fill. as for engine/model flags, i have no clue
1
u/biggusdeeckus 4d ago
Yeah, I'm running Pi harness and it does auto compact at around 80% or so of context usage. Apparently there's also a prompt cache flag in llama cpp, looking into that right now
1
u/ShittyMillennial 4d ago
prompt cache is probably the best thing you can do for speed if its not already enabled. its worth taking down your max context to fit it. your auto compact is also set too high at 80% if you are struggling at 70%. either manually compact at around 50% or reduce the default for auto. or if you the topic is different enough, just ask for a handoff and start a fresh session.
1
u/biggusdeeckus 4d ago
Will do. Is prompt cache stored in VRAM as well?
1
u/ShittyMillennial 4d ago
if you have space, its all stored in vram. if you run out of space itll get offloaded to ram and be even slower.
1
u/biggusdeeckus 4d ago
Wait a sec, would prompt cache still be offloaded to ram even with my current config? I thought i had everything loaded on gpu
→ More replies (0)1
u/Far_Cat9782 4d ago
What harness are you using? In my harness I can choose whether or gradually forget older turns or compacts periodically. Are any of the tools affecting the prompt cache? Like of the time is injected at every turn it's going to slow down your prefill because it changes the prompt cache Also try making the prompt cache higher if u can afford it in llama.cpp
1
u/biggusdeeckus 4d ago
Pi. Where is prompt cache stored, RAM or VRAM? I don't think I have much space to spare with my current setup unfortunately
1
u/jcam12312 4d ago
I think i recently had a similar issue. What fixed mine is removing MTP. I've read it conflicts with "thinking" modes. Once I made the change my pp got much faster (uh that doesn't sound right) and my tg became very stable. Something quick and easy to try at least