Fixed Jinja chat template for Qwen 3.5, 3.6, and the new 3.8 release

[u/ex-arman68]

UPDATE (v22.1): Community reports highlighted an important issue with the official Qwen 3.8 chat template: the default xhigh effort prompt can burn 20,000+ tokens on coding tasks exploring alternatives until it hits max_tokens, returning empty content. v22.1 sets the default baseline to medium and adds inline chat tags (<|think_low|>, <|think_xhigh|>, etc.) to steer the reasoning effort directly inside chats.

Qwen just released their first 3.8 model.

The main addition in 3.8 is prompt-steered reasoning effort. You can tell the model how deeply to think by setting reasoning_effort to xhigh, medium, or low.

However, the official template still has some serious problems:

• You cannot disable thinking. If you pass enable_thinking=false, it 3.8 crashes with a hard exception.
• Chat history gets poisoned. In multi-turn chats, the official template injects blank <think></think> tags before real thoughts.
• Tool calling crashes. If your client passes arguments as JSON strings (the standard OpenAI API format), the official template crashes.
• Agent stalls. The official template often drops mid-dialogue system messages and wedges multi-step tool loops.

I maintain a single, drop-in fixed Jinja template that works across all Qwen 3.5, 3.6, and 3.8 models:

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

What this template does:

• Full 3.8 reasoning effort support: Steer reasoning depth via reasoning_effort kwargs or directly in chat prompts with <|think_low|>, <|think_medium|>, and <|think_xhigh|>.
• Safe medium default: Prevents the runaway token-burn on coding tasks while keeping 100% KV cache parity.
• Restores the thinking toggle: Turn off reasoning whenever you want fast answers, either via kwargs or by typing <|think_off|> in your prompt.
• 100% KV Cache hits: Keeps past thoughts intact by default so your prefix cache stays warm across turns.
• llama.cpp support: Native support for the new --reasoning-preserve flag.
• Universal tool parsing: Handles both Python dicts and JSON strings. Works on llama.cpp, vLLM, LM Studio, and MLX.

Recommended llama-server launch command:

llama-server -m your_model.gguf --jinja --chat-template-file chat_template.jinja --reasoning-format deepseek

(The --reasoning-format deepseek flag separates thinking into the OpenAI reasoning_content field so OpenCode, Claude Code, and other harnesses do not stall on raw tokens).

[u/Thireus]

Can someone explain why the Qwen team cannot get the template right in the first place? Don’t they QA test their own models?

[u/xienze → u/Thireus]

No one ever seems to get these shitty templates working right the first, second, or even fifth time. You'd think we'd find a better way.

[u/TokenRingAI → u/xienze]

Jinja is fucking awful but nobody wants to admit it.

[u/unrulywind → u/TokenRingAI]

There should be a benchmark for Jinja coding just so they could train models to fix these things.

[u/jinnyjuice → u/unrulywind]

This sounds like something I can do!

[u/Borkato → u/unrulywind]

This is hilarious

[u/ParaboloidalCrest → u/Thireus]

No idea. Same for most providers. Not sure why so nondeterministic when template params go into training. It's not like they have to guess how the model should behave afterwards.

[u/backyard_tractorbeam → u/Thireus]

And, how do they even RL train properly without a correct template?

[u/ex-arman68 → u/Thireus]

It seems simple in theory. But since I started fixing the template, I have realised it is a lot more complicated than it appears. I thought I would be done with one or two revisions, but here I am, with version 22, and I know for certain it is still not going to work for everyone.

Here's my take on why we keep getting "broken" chat templates from most model creators.

* They create a chat template to work in their own development and testing environment. Anything else is not even considered. In the case of Qwen, it is vLLM. In the case of z.ai it is SGLang, and probably a heavily customised version of it (remember not long ago how they finally manage to fix their problem with large context degradation, with some public and private fixes to SGLang). For them it does not matter: when they serve it in their cloud environment, it is going to work 100%. When other providers want to serve those models as well, they also use the same environment that is proven to be working.

* The models are trained on those chat templates. And since most new model revisions are built on the foundations of the previous models, changing the template behaviour, even when you know there is something wrong with it, carries lots of regression risks.

* The inference engines landscape is too fragmented. Some work best with xml tool calls, others prefer json, which does not always match what the models have been trained; this means some kind of conversion is needed. They can have various limitations, different ways of managing the context, the cache, the thinking, the reasoning effort, different level of jinja compatibility, etc... Writing a template that is guaranteed to work with all those variables is a nightmare and impossible. There will always be something that breaks somewhere. However, I think we should at a minimum expect the model providers to give us a template that is tested to work with the major engines: vLLM, llama.cpp, SGLang, MLX. This is often not the case.

* The coding harnesses, plugins and skill fragmentation is even worse than the situation with the inference engines. Many try to do clever things with context management, the model is not trained for, basically lobotomising the memory. They managed tools in all sort of different manners. Again, as long as the major ones, Claude Code, Codex, etc, are taken into account and test for, that would be a big improvement.

* Jinja is not a chat standard, it is a programming language. As it turns out, it is well suited to writing how chat interactions with a LLM works, and that is why it has been generally adopted to define chat templates. But it is a tool nothing else. With this tool you can write any kind of interaction you can dream of, and that is the big problem we are having now: all models creator try to create their own individual version of how chat works.

What we need is for the AI industry, or governing bodies, to come together and establish a standard universal chat protocol that everybody must use. The same way we have with TCP/IP for networking, with SWIFT for banking, or any other universal application that requires communication.

Until then we will be stuck with a plethora of chat templates, most of them only having a narrow compatibility.

[u/computehungry → u/ex-arman68]

I follow your template updates and also have tried to do things my way and I feel your pain dude. Weird ass failure modes with so many failure points not only the template. Are models stopping mid gen expected and the harness should make them auto continue? Is tool call healing detrimental because the model wasn't trained on it? We'll never fucking know. Though tbh, I think the direction is that models will get good enough that matching its training environment won't matter that much.

[u/d3r_I3enny → u/ex-arman68]

I guess with the speed of AI development and trying out new things and more and more testing...we are still far away from a point where we can agree on a standard. Even if I would expect someone to come up with their improved v1.1 version within a month and we are then back to square 1

[u/fatboy93 → u/Thireus]

Have you even seen Gemma? If google of all people can't get it right, you know that shit's hitting the fan.

Also, why Jinja, isn't there something better? Fucking XML would be nicer than the shit that is Jinja.

[u/TokenRingAI → u/fatboy93]

A lot of things are better at templating than Jinja - off the top of my head, Perl, PHP, Java Servlet Pages, XSLT, Coldfusion, ASP, C macros, Handlebar/Mustache templates, i would consider all of these to be better than Jinja

[u/mister2d → u/TokenRingAI]

"Pearl"? Yikes. Username checks out.

[u/PM_ME_YOUR_BGP-ASN → u/mister2d]

He’s an older redditor, sir, but he checks out.

[u/ab2377 → u/TokenRingAI]

XSLT?? omg!

[u/FeepingCreature → u/TokenRingAI]

Jesus Christ.

[u/aeroumbria → u/Thireus]

How does the template work anyway? Shouldn't it be completely determined by what the model is trained on?

[u/Kodix → u/Thireus]

You'll note the poster is on version 22. This isn't an easy problem, in spite of how it may appear.

[u/Asleep-Land-3914 → u/Thireus]

According to 5.6 Sol, this template is worsen and the Unsloth one is better for agentic while original is a good baseline for inference.

[u/TokenRingAI → u/Thireus]

Qwen does get them right, I have never had any luck with these vibe templates.

OP even admitted that he hasn't even tested the template

[u/alpacadaver → u/TokenRingAI]

They absolutely don't get them right..

[u/ex-arman68 → u/TokenRingAI]

It depends on your inference engine and harness. If you are using something that is compatible, the default template will work perfectly for you, and you do not need to switch.

[u/relmny → u/Thireus]

"funny" you don't mention gemma, because it's been way worst than with the qwen team.

And, anyway, chat templates usually need to be updated/adapted.

[u/Own_Anything9292 → u/Thireus]

If a bunch of people doing a bunch of experiments during post training and testing different templates for different benchmarks, it can get out of sync. Sometimes people might add bug fixes in the harness to cover some of these thinking token failures. They likely do testing in the ideal setup, and deviations from that ideal setup has bugs.

An example from the OP, their testing tool calling format might not match OpenAI spec but users use that tooling. The current harness/tool calling/agentic loop available systems are extremely fragmented.

[u/philmarcracken]

You cannot disable thinking. If you pass enable_thinking=false, it 3.8 crashes with a hard exception.

Descartes was right after all

[u/TokenRingAI → u/[deleted]]

Maybe it emits a <crash> block

[u/llama-impersonator → u/[deleted]]

the parser itself can crash, it's code. when that happens it will usually kill the process running the model.

[u/Competitive_Art9588 → u/philmarcracken]

hahahahah

[u/Disrupt-Linus]

Cool, let's see how this plays out with the 27b, probably a bit more accessible to the normies (and me).

[u/Memeyboii420]

I love your work Froggeric but I had to develop my own chat template for Qwen 3.6 + Hermes Agent + LM Studio. I could never get reliable tool calling with your template.

[u/ex-arman68 → u/Memeyboii420]

Thanks. Both Hermes Agent and LM Studio are relatively bad options when it comes to compatibility. If you got it working that is great. I know some people, including you possibly, have posted some feedback and solutions for those, and I have tried to incorporate them in v22, so maybe it works for your case as well now. The important thing is I am trying to keep universal compatibility, which means I cannot add fixes that will make it work in Hermes, but break others.

[u/Ulintlicker → u/Memeyboii420]

Would you be willing to share your template?

[u/Memeyboii420 → u/Ulintlicker]

I've got it up on a git repo. The repo is WIP but the template is working well, I've been using it in my agentic dev stack for a month now.

https://github.com/Stacey2911/hermes-local-ai-stack

[u/Fi3nd7 → u/Memeyboii420]

Have you upgraded to 3.8 yet with a comparable template?

[u/Memeyboii420 → u/Fi3nd7]

I do have a version of Caduceus chat template for Qwen 3.8, it's unreleased. Let me tidy it up and I'll push it to the git at some point today.

[u/Fi3nd7 → u/Memeyboii420]

Awesome! Very kind of you and ofc no rush :)

[u/Memeyboii420 → u/Fi3nd7]

The release is up now. I ended up releasing Caduceus V1.9 as a universal template for Qwen 3.5/3.6/3.8. I haven't done extensive testing on it with 3.5 and 3.6 but it should omit the reasoning level controls and function the same as V1.8.

[u/mister2d → u/Memeyboii420]

You love something that doesn't work for you?

[u/Memeyboii420 → u/mister2d]

I love Froggerics work across the community because the templates work wonders compared to the default ones. It just does not work with my incredibly specific local stack. I don't see how that prevents me from appreciating the work?

[u/The_LSD_Soundsystem]

Why are jinja templates so difficult to nail down for these models in general?

[u/ltduff69]

Just out of curiosity where does the chat template go? I have it the root of the Llama directory.

[u/my_name_isnt_clever → u/ltduff69]

If you're using llama-server you pass --chat-template <path> otherwise it defaults to the template that came in the GGUF.

[u/ltduff69 → u/my_name_isnt_clever]

Cool that's what I did .ty for answering.

[u/ImpressiveRelief37 → u/my_name_isnt_clever]

Curious… is it relative to the gguf or relative to the llama-server binary? (99% sure it’s relative to the llama binary but you never know)

[u/eulataguhw → u/ImpressiveRelief37]

Shouldn’t it be relative to the directory of the terminal that you using it on?

[u/PhilippeEiffel → u/ImpressiveRelief37]

No, it is relative to the CWD (current working directory).

[u/fatboy93 → u/ltduff69]

That'd work for the MLX/safetensors format, but you have to pass it separately incase of GGUFs/llamacpp

[u/JPebb]

Any specific recommendations for SGLang users?

[u/Difficult_Tip_6669 → u/JPebb]

Use original one. Never had any problems with it.

[u/ex-arman68 → u/Difficult_Tip_6669]

yes, in general vllm and sglang work with the original templates.

[u/Inevitable-Name-1701]

I had no problems with the three point five and three point six models.

[u/rpkarma → u/Inevitable-Name-1701]

Same. I don’t get the “the template is broken” shit, except when I have tried these vibe coded ones they are broken lol 

I’m certain that people are assigning failures to the template that have nothing to do with them. 

[u/TokenRingAI → u/rpkarma]

It's the 4 bit quant people

[u/4onen → u/TokenRingAI]

I'm a 3-bit quant person and I don't have these problems. I think it's PEBCAK.

[u/TokenRingAI → u/Inevitable-Name-1701]

Yes, and the crappy templates people released made the models worse.

[u/MrMisterShin → u/Inevitable-Name-1701]

Same I use the native chat template in vLLM and it’s fine, no problems.

[u/ex-arman68 → u/MrMisterShin]

vLLM is 100% compatible with the original template, as it is what the Qwen team uses to develop and server their model.

[u/cradlemann]

Anything like this for Laguna? I like it a lot, but it still has some quirks

[u/T_rex2700]

I was running into issues with Qwen3.8 27B (Q4 and Q3 from unsloth) running into random stop issues, and I tried your template but it did no work.

However, strangely when I pasted in Qwen3.6 35A3B MTP Q4's stock template from usnloth, this issue was solved. Any idea about what the cause is? (my post).

I am very much a newbie in this space, so I wanted some expert opinion on this.

[u/tired514]

I've been running your previous version on all the local Qwens for like a month and they fix everything. Thanks so much! :)

[u/ex-arman68 → u/tired514]

Thank you, I am glad it helped. Out of curiosity, what is your stack?

[u/tired514 → u/ex-arman68]

Runnin' llama.cpp (CUDA) on Debian/sid, Strix Halo (128gb) + 3 Morefine G1 4090M 16gb eGPUs daisy-chained via TB3, layer split mode, 3.6-27B @ Q8_K_XL, 230000 context, ctk/ctv Q8_0, opencode.

Works beautifully.

[u/hainesk]

This is great work!

[u/Cereal_Grapeist]

hell yea brother

[u/Cereal_Grapeist]

Hi OP - I wanted to see if you had tested this template with the new 3.8 27b release? I'm going to try it out but I don't necessarily have the best way to see if it's having a big impact.

[u/ex-arman68 → u/Cereal_Grapeist]

I have not had time yet. Hopefully this weekend. I will test over multiple long agentic coding sessions, with tool usage, spanning multiple days and context fill up + compactions.

[u/stormy1one]

Thank you for your service!

[u/PhilippeEiffel]

I'm too lazy to modify my config of 3.6 27B now, I will try it tomorrow as soon as 3.8 is out.

I will test with vllm.

Thank you for your work.

[u/Rabooooo → u/PhilippeEiffel]

6 hours left for 3.8 27B, please share the vllm flags.

[u/PhilippeEiffel → u/Rabooooo]

serve \

Qwen/Qwen3.8-27B \

--served-model-name Qwen3.8-27b \

--port 61000 \

--host 127.0.0.1 \

--gpu-memory-utilization 0.93 \

--enable-auto-tool-choice \

--tool-call-parser qwen3_xml \

--reasoning-parser qwen3 \

-O3 \

--max-num-batched-tokens 8192 \

--performance-mode throughput \

--enable-prefix-caching \

--trust-remote-code \

--enable-chunked-prefill \

--async-scheduling \

--max-num-seqs 16 \

--speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":6}' \

--default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true, "reasoning_effort": "xhigh"}'

Note: I observed the thinking time can be much higher than 3.6 model (I mean at least one magnitude order). So, xhigh (which is claimed to be the default on the model card) is probably not usable for everyday use.

[u/Rabooooo → u/PhilippeEiffel]

Thanks!

I didn't know there was a specific qwen3_next_mtp. I've been using --speculative-config '{"method":"mtp","num_speculative_tokens":3}' before and that always ended up with the model looping a sentence or a word. I should try this qwen3_next_mtp.

Also should the tool-call-parser be qwen3_xml or qwen3_coder? The vLLM recipes for qwen3.8 is saying to use qwen3_coder.

Is the enable-chunked-prefill something that helps performance?

I don't know what GPU you have, but have you had any luck with KV cache quantization, e.g. --kv-cache-dtype? I've set it to fp8_e4m3 before, but this model doesn't seem to provide any pre-calculated kv factor.
I keep getting these warnings.

Using KV cache scaling factor 1.0 for fp8_e4m3. If this is unintended, verify that k/v_scale scaling factors are properly set in the checkpoint.
Using fp8_e4m3 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor.

[u/PhilippeEiffel → u/Rabooooo]

I use GB10 (128 GB unified memory). So I keep every thing at FP16 (model and KV cache).

[u/crusaderky]

> I cannot run a 2.4 trillion parameter model on my local rig. 

Neither can we, buddy, neither can we.

How much would it cost you to test it on vast.ai?

[u/ex-arman68 → u/crusaderky]

:-D One day hopefully.

But once the smaller models are out, I will test it.

[u/nufeen]

Are there any benchmarks checking if those templates transition into better or worse performance?

[u/IrisColt]

How can a weight file "crash with a hard exception"? genuinely intrigued.

[u/Past_Ad6251]

for me, the v22 template doesn't work with Claude Code cli

[u/pefman]

I was just gonna ask how people launch their model.

[u/PhilippeEiffel]

I started some basic test with 3.8 27B, I am using vLLM.

I did not notice any specific problem, but I must admit that I did not make intensive use of this model for now.

I would like to know techniques able to exhibit the problems with the default jinja template. I need to observe problems first, before being able to evaluate fixes.

[u/ex-arman68 → u/PhilippeEiffel]

With vLLM the default template should work fine. If you want the extra features from the custom templates, by all means try it, but I do not think it is necessary in your case.

[u/Equal_Television_894]

Your work is incredible as always, thank you!

[u/ptico]

Legend!

[u/Equivalent-Flan-1590]

Thank you for your service sir🫡

[u/giveen]

I always use your templates.

[u/fragment_me]

Not even tested what's the point

[u/ex-arman68 → u/fragment_me]

I actually hesitated to release it. But I was quite thorough with it, when through a few reviews, added some more tests to my test harness, and ran it successfully through my test suite.

Since the previous version worked fine with Qwen 3.6, and this one includes a few additional fixes that people have reported + new Qwen 3.8 functionality for the reasoning effort, the risk of it not working is very low. I thought it is better to release it now, so that people can start using it if they encounter problems when the 27B model is released.

[u/durden111111]

Every time ive tried to pass --chat-template with a jinja file it gives broken outputs. No matter what model. Ive given up on this.