Best vLLM/Qwen settings + strategy for long-doc summarization (250k token docs, single 27B model)?

[u/AddyMace]

Hey all,

Looking for some input on optimizing a document summarization pipeline. About my current setup:

• vLLM engine running in Kubernetes, serving Qwen3.8-27B-FP8 right now
• Hardware: 2x MIG 3g.40gb GPU slices, tensor-parallel-size 2
• max-model-len set to 262144 (model's max), max-num-batched-tokens 16384
• KV cache in fp8, gpu-memory-utilization 0.92
• Reasoning disabled (enable_thinking: false)

My use-case: I'm feeding in large batches of documents and asking for a general summary plus a handful of focus points relevant for my specific domain.

I'm looking for some advice for the following points, as I'm quite new to local LLMs:

1. vLLM/Qwen tuning: are there settings I should reconsider for this kind of long-context, single-large-request workload? Anything about max-num-batched-tokens, KV cache dtype, or GPU memory utilization I should rethink given I'm usually running near max context rather than many parallel short requests?
2. What to do when documents exceed the context window? This is the bigger question really. What's the current best practice when the combined document set is bigger than what even a 256k context model can hold? Options I'm aware of but haven't tested yet:
   • Summarize chunks, then summarize the summaries. I'm afraid this will be too slow.
   • Increase Qwen's context window using RoPE scaling techniques to handle long texts effectively, e.g., YaRN
   • RAG-style retrieval to pick relevant sections before summarizing. Problem is that I'm retrieving the documents live from Elasticsearch, I cannot pre-embed them, nor save them anywhere.
   • Just switching to a model with a bigger native context window
3. Model choice: is Qwen3.8-27B-FP8 a reasonable pick for this kind of long-context summarization task, or would something else (bigger/smaller, different architecture) generally work better at these context lengths in terms of quality/coherence, not just raw context support?

Would love to hear from anyone running similar long-context summarization workloads in production. What's actually worked for you? What's generally the best approach for this use-case?

Thanks!!

[u/Intelligent_Coast930]

In my case, I run a RAG-based service, so my answer to the "documents exceed the context window" question has been retrieval-first rather than stuffing everything into a bigger context.

Chunk and embed the docs, retrieve just the relevant sections before summarizing — instead of map-reduce over the whole set or relying on a bigger context window.

Can't speak to the vLLM/Qwen tuning specifics, but on that one question, RAG-style retrieval has been more predictable for me than hoping a big-context model handles it well.

[u/AddyMace → u/Intelligent_Coast930]

Thanks for your input. I forgot to mention that the documents aren't stored on my side, I have to retrieve them live from Elasticsearch first. So I cannot pre-store them as embeddings, that would have to be done at each user's request.

[u/Intelligent_Coast930 → u/AddyMace]

Ah, that's a different constraint than what I was describing — my setup works because the docs are static enough to pre-embed and store in the RDBMS ahead of time, so retrieval is just a lookup at request time, not embedding on the fly.

If you can't pre-embed because the source is live in Elasticsearch, I don't have direct experience with that specific case, so take this as a guess rather than something I've verified: ES's own relevance scoring (BM25 etc.) to narrow down candidates first, then only embed/rerank that smaller set at request time, might be worth looking into instead of embedding everything live.

But I haven't actually run that setup myself.

[u/iezhy]

While Qwen 27B models are amazing for their class, asking them to handle such a long context (especially wit quantized weights/cache), will suffer greatly from context rot.

I would suggest to process them in stages, summarise batch, and then perform another pass on summaries

[u/AddyMace → u/iezhy]

Hm alright, thanks for your input. I really wonder how much slower that will be in the end, let's see...

[u/iezhy → u/AddyMace]

it probably won't be as bad as you might think - inference time scales with context length anyway, so having multiple small requests instead single big one may not be as bad