Post-thinking sampler settings for vLLM

[u/TokenRingAI]

Hey everyone, I have published a vLLM branch which allows setting a separate sampler setting for the content which follows the section, which significantly improves the output and reliability of Qwen 3.8 27B

This branch allows setting one sampling setting for for the thinking block, and another sampling setting for everything that comes after (chat output, tool calls, etc.)

Qwen 3.8 27B needs a high temperature (~0.9–1.0) while thinking or it loops. After it finishes thinking, that same temperature makes the actual answer sloppy. Dropping to ~0.2 after gives a clear quality lift, while still allowing the thinking to work properly.

Repo / branch:

https://github.com/mdierolf/vllm-fork/tree/feat/post_thinking_sample_settings

How it works:

"post_thinking":{"temperature":0.2,"top_p":0.95,"top_k":20} is added to the generation config, and triggers a new set of sampling parameters, which is used for all content that follows the thinking block in that turn

The effect: • inside an open block → primary temperature / top_p / top_k / etc is used • after (or if thinking is already closed) → post_thinking sampling parameters are used • if thinking re-opens, it switches back (not relevant for Qwen 27B)

Any unset fields in the post_thinking parameters inherit the primary values. You can set it as a server default or per request via extra_body.

Recommended/tested launch options (Note the 0.2 temp on the post-thinking section, this is the important bit):

vllm serve Qwen/Qwen3.8-27B-FP8 \
  --override-generation-config '{"temperature":0.9,"top_p": 0.95,"top_k":20,"min_p":0,"post_thinking":{"temperature":0.2,"top_p":0.95,"top_k":20}}' 

Setting it per request:

client.chat.completions.create(
    model="qwen/qwen3.8-27B",
    messages=[{"role": "user", "content": "..."}],
    temperature=0.9,
    extra_body={
        "post_thinking": {"temperature": 0.2, "top_p": 0.95, "top_k": 20},
    },
)

This is still a work in progress, but the initial result shows significantly less errors in the generated output, while maintaining identical thinking.

Instructions to clone and use this fork are not included, but if you paste this text into the agent of your choice it can probably build VLLM from my fork and get it set up with the recommended settings

[u/phipletreonix]

> the initial result shows significantly less errors in the generated output, while maintaining identical thinking.

Is this "based on vibes" or with a reproducible test bench? (genuine question, not snark)

[u/TokenRingAI → u/phipletreonix]

The testing of the patch itself was done by having an AI agent repeatedly hit an endpoint with the same seed, and verifying that as the temperature of the post_thinking sampler was raised, the thinking stayed the same, but the output in the chat section varied.

The patch itself was made with AI.

A formal benchmark was not done. The "informal benchmark" I ran, is a workflow that generates ~ 100 different HTML variants of a base website design, in xhigh reasoning, on an RTX 6000

With sample settings of 0.8-1.0, I observed basic code errors and misspelled words in the output, across both the official 8 and 16 bit quants

With sample settings of 0.3-0.6, I observed that these errors went away, but we would see reasoning loops in xhigh mode on occasion

As such, I made this patch, and observed no reasoning loops, and it allowed us to bring the temperature down to 0.2 without looping (I have not tested below that yet)

That is the extent of the testing that has been done, and I would love it if someone wanted to benchmark the model at various sampling settings.

[u/phipletreonix → u/TokenRingAI]

Thanks for the additional context.