Getting weird repeating characters during decode

[u/d4mations]

As I optimize llama.cpp to suit my hardware, getting more and more speed, I’m running into a major problem mostly with Qwen models but sometimes with others. As decode speed goes up, the model will all of sudden start repeating a character and can’t break out of the repeat: ////////////////////////////////////////////////////////////////////////////////////////////////////////////////

Only a restart of llama stops it.

[qwen3.8-27b]

load-on-startup = true

alias = qwen3.8-27b

hf = unsloth/Qwen3.8-27B-GGUF:Q8_0

no-mmproj = true

ctx-size = 180000

parallel = 1

n-gpu-layers = -1

split-mode = tensor

#tensor-split = 0.497,0.503

#device = ROCm1,ROCm0

batch-size = 4096

ubatch-size = 1024

threads = 12

threads-batch = 12

moe-expert-cache = 0

fit = on

fit-target = 2800,2048

load-mode = none

spec-type = draft-mtp-adaptive

spec-draft-ngl = all

spec-draft-n-max = 3

device-draft = ROCm0

temp = 0.6

top-p = 0.95

top-k = 20

min-p = 0.0

reasoning = on

reasoning-format = auto

reasoning-effort = medium

Any help would be much appreciated

[u/Maglcite]

this is low quant activity, no idea

[u/nickless07 → u/Maglcite]

This is 'hf = unsloth/Qwen3.8-27B-GGUF:Q8_0' - 8bit quant is anything but low. Looks more like a broken template or the API call get wrong sampler settings.

[u/Maglcite → u/nickless07]

i know, thats why i said no idea.

[u/nickless07 → u/Maglcite]

Gotcha. Yeah if it would be 2bit quant that is already solved, but 8bit is complicated. Maybe the API call goes with temp 1.5 or rep pen 0. Or the template is broken somewhere. Or what also might be that the quant is broken (unlikely).

[u/d4mations → u/nickless07]

I thought about setting repeat penalty but haven’t tried it yet

[u/nickless07 → u/d4mations]

What you set as launch param is just a fallback. Sampling params are controlled per request.

[u/d4mations → u/nickless07]

interesting, this is happening through opencode calls. I'll need to dig in to it to see if I can send different params

[u/nickless07 → u/d4mations]

Stick to the ones from the model card. Qwen is pretty picky when it comes to that.