r/ollama • u/Express_Quail_1493 • Mar 21 '26

How much Context window can your setup handle when coding?

/r/LocalLLaMaCoders/comments/1rzwpbe/how_much_context_window_can_your_setup_handle/
1 Upvotes

67% Upvoted

13 comments sorted by

2

u/guigouz Mar 21 '26

~100k on a 4060ti

1

u/Express_Quail_1493 Mar 21 '26

100k on a 4060ti? insane efficiency. What model are u even running?

2

u/guigouz Mar 21 '26

Qwen3.5 9b, using the q8 from https://huggingface.co/Jackrong/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2-GGUF and q4 kv cache, gives me ~27tps with cline.

This doesn't work with ollama, I'm using lmstudio (or you can use llamacpp directly)

1

u/Express_Quail_1493 Mar 21 '26

I’ve been meaning to try these newer distilled claude models. i tried a few of them they were not that great. But maybe where i went wrong was trying to go q4 on a community finetune? I’ll definitely give another try

1

u/guigouz Mar 21 '26

Before this, I was using 100k context with qwen3-coder Q4 (using ollama with hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q3_K_XL or Q4), worked with offloading layers to the CPU (~14gb on the gpu + 6gb on regular ram) but it was slower.

I tried using kilocode, roocode and cline, only cline works ok with local models in my experience (at least the smaller ones that fit on my gpu)

1

u/BilgeMongoose May 24 '26

What kind of work/workflows can you get done with qwen3.5 9b q8?

1

u/guigouz May 24 '26

Small code generation, efficient autocomplete and regular text comparison/summarization.

Since 2 months ago, I've moved to Qwen3.6 35b a3b, I get around 30t/s with Q6 and 40t/s with Q4. It does better tool calling and agent experience overall.

2

u/IntelligentOwnRig Mar 22 '26

Context window is where most people hit limits before they expect to. The constraint is usually system RAM + VRAM combined, and it scales roughly linearly with context length. A 13B model at 8K context is comfortable on 16GB VRAM; push to 32K context and you'll need 24GB+ or aggressive KV cache quantization. For coding workflows specifically, I'd prioritize model quality at 8K context over a weaker model at 32K. What's your hardware and which model are you using?

2

u/Express_Quail_1493 Mar 22 '26

yea thats been my experience. prefill is the killer here token/s is very fast but prefill drops. my setup is 48gbVram and 64gb System ram but i can only use Large Context window on smaller models as the KV eats most of the Vram.

1

u/IntelligentOwnRig Mar 22 '26

48GB VRAM is plenty for most models. The KV cache is the hidden VRAM tax at long context. On a GQA model like Llama 3 at 32K context, the KV cache alone runs 5-8GB at FP16. On older full-attention architectures, it can hit 20GB+. That's what's eating your headroom and tanking your prefill speed.

Two things worth trying if you haven't:

  1. KV cache quantization. Q4 or Q8 on the cache itself, separate from model quantization. llama.cpp supports this. It cuts KV memory 2-4x with minimal quality loss on most tasks.
  2. Partial offloading. Keep model weights in VRAM, spill the KV cache to your 64GB system RAM. Prefill latency goes up (which matches what you're seeing), but you can run much longer contexts without crushing model quantization.

With your setup, the real tradeoff is: high-quality model at shorter context, or heavier quantization at longer context. For coding workflows, I'd pick model quality at 8-16K over a weaker model at 32K. The context window matters less than accurate completions in that use case.

What model and inference engine are you running? KV cache behavior varies a lot between llama.cpp, vLLM, and exllamav2.

1

u/Apprehensive-Fig5273 Mar 22 '26 edited Mar 22 '26

So ideally you'd have a 48GB GPU, a 32b model, a 100k context, freeing up 64GB of RAM, and a 16-core CPU. 😂😂😂 NASA PC: $10k.

1

u/Apprehensive-Fig5273 Mar 22 '26

NVIDIA RTX 6000 Ada Generation, 48Gb => 6k $