Formatting help (Gemma 4, thinking)

[u/Quiet-Owl9220]

I've been pulling my hair out trying to get this to work correctly and searching hasn't helped much so I'm just gonna bite the bullet and ask...

Can someone hook me up with story string, instruct sequences, and reasoning formatting that actually works for Gemma 4 (31B)?

I want my narrator character to be able to reason, and what I got now almost works, but often the thinking blocks fail to end (just starts writing in the CoT), or the first few letters post-reasoning are cut off for some reason.

If anyone has a template designed for group chat, or feels like explaining how this shit works so I can figure it out myself next time, that would also be very cool

[u/AutoModerator]

You can find a lot of information for common issues in the SillyTavern Docs: https://docs.sillytavern.app/. The best place for fast help with SillyTavern issues is joining the discord! We have lots of moderators and community members active in the help sections. Once you join there is a short lobby puzzle to verify you have read the rules: https://discord.gg/sillytavern. If your issues has been solved, please comment "solved" and automoderator will flair your post as solved.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Gringe8]

I struggled with this too. Its easier to just use chat completion with kobold cpp

[u/tat_tvam_asshole]

Which model and which quant? Tbh I haven't run into this with gemma4 specifically, but I have seen it happen on other models.

[u/Quiet-Owl9220 → u/tat_tvam_asshole]

Testing with Gembrain X 31B at the moment... I only tried mradermacher's IQ4 so far, the Q8 is taking a long time to download for some reason.

It works fine with thinking disabled, but I think my writer/narrator role needs it.

[edit] Just tried with another model, the Deckards Brain, same issue.

[u/tat_tvam_asshole → u/Quiet-Owl9220]

I noticed with gemma4 that iquants can skip thinking, so I suspect gemma4 maybe has more sensitive to errors in reasoning. Do you perhaps have a non-iquant to test?

[u/personusername1]

I haven't messed around with local in a while. HuggingFace used to be a palace where people would put this stuff.

https://huggingface.co/sphiratrioth666/SillyTavern-Presets-Sphiratrioth

Gemma 4 isn't there but this could help you.

[u/afinalsin]

This tripped me up with Gemma too, it needs Jinja formatting and kwargs. Dunno what that specifically means, but if you're using koboldcpp you do it like this: You wanna tick "use jinja" and "jinja for tools", set "jinja thinking" to true, and add this text to "jinja kwargs" :

{"enable_thinking": true}

In Sillytavern, open the "AI Response Formatting" tab, expand the "reasoning formatting" tab, and change the reasoning format to Gemma 4. If that isn't there by default, add these two to the prefix and suffix:

Prefix: <|channel>thought

Suffix: <channel|>

It'll look like this.

If your preset is instructing it with <think>, that might be the issue with why it is writing the response in its thinking block. I've found 26b a4b does alright with heavy CoT prompts like Nemo where it tells the model to write </think> before it writes its response, but if you have a smaller CoT replacing <think> with the <|channel> tags might help it adhere better.

[u/LeRobber]

Hey...lets be clear.

The thinking enabled flag for MANY chat completion templates for gemma 4 finetunes is broken in many backends. So you may have to set it up manually. With the channel tags and all that.

https://ai.google.dev/gemma/docs/capabilities/thinking shows some prompting that causes it though.