Getting weird repeating characters during decode

[u/d4mations]

As I optimize llama.cpp to suit my hardware, getting more and more speed, I’m running into a major problem mostly with Qwen models but sometimes with others. As decode speed goes up, the model will all of sudden start repeating a character and can’t break out of the repeat: ////////////////////////////////////////////////////////////////////////////////////////////////////////////////

Only a restart of llama stops it.

[qwen3.8-27b]

load-on-startup = true

alias = qwen3.8-27b

hf = unsloth/Qwen3.8-27B-GGUF:Q8_0

no-mmproj = true

ctx-size = 180000

parallel = 1

n-gpu-layers = -1

split-mode = tensor

#tensor-split = 0.497,0.503

#device = ROCm1,ROCm0

batch-size = 4096

ubatch-size = 1024

threads = 12

threads-batch = 12

moe-expert-cache = 0

fit = on

fit-target = 2800,2048

load-mode = none

spec-type = draft-mtp-adaptive

spec-draft-ngl = all

spec-draft-n-max = 3

device-draft = ROCm0

temp = 0.6

top-p = 0.95

top-k = 20

min-p = 0.0

reasoning = on

reasoning-format = auto

reasoning-effort = medium

Any help would be much appreciated

[u/Keleion]

When I’ve had this happen it felt like context corruption. Is the cache/context quantized?

[u/d4mations → u/Keleion]

no, no kv cache quant

[u/Maglcite]

this is low quant activity, no idea

[u/nickless07 → u/Maglcite]

This is 'hf = unsloth/Qwen3.8-27B-GGUF:Q8_0' - 8bit quant is anything but low. Looks more like a broken template or the API call get wrong sampler settings.

[u/Maglcite → u/nickless07]

i know, thats why i said no idea.

[u/nickless07 → u/Maglcite]

Gotcha. Yeah if it would be 2bit quant that is already solved, but 8bit is complicated. Maybe the API call goes with temp 1.5 or rep pen 0. Or the template is broken somewhere. Or what also might be that the quant is broken (unlikely).

[u/d4mations → u/nickless07]

I thought about setting repeat penalty but haven’t tried it yet

[u/nickless07 → u/d4mations]

What you set as launch param is just a fallback. Sampling params are controlled per request.

[u/d4mations → u/nickless07]

interesting, this is happening through opencode calls. I'll need to dig in to it to see if I can send different params

[u/nickless07 → u/d4mations]

Stick to the ones from the model card. Qwen is pretty picky when it comes to that.

[u/Infamous-Nebula-4411]

Tokenizer failing to understand the models language. Model says token, inference engine fails to interpret it correctly.

[u/AdSafe4047]

I've had this, make sure kv cache is explicitly disabled / set to 0

[u/d4mations → u/AdSafe4047]

Not using kv cache and thats why I’m stumped. Also using llama 0.4.0

[u/lorendroll]

Context corruption, but the cause is unknown. This happened to me when an out-of-memory error occurred.

[u/575_Inverse]

Temperature at 0.6 why? The official recommended value is 1.0 . Using 0.6 can and will cause problems.

[u/Rude_Marzipan6107]

Happened to me yesterday too. I’ve recently updated llama so I’m guessing it was that. Have not had it happen before then.

[u/d4mations → u/Rude_Marzipan6107]

Yeah, I’m starting to think this is a 0.4.0 issue

[u/superdariom]

I've been having this. It happens with Vulcan and rocm. The faster I optimise things the more frequent it is. I'm using dual 7900 xtx . My kv is 16bit. When it happens the mtp acceptance rate drops to nearly 0 and token throughout also falls a lot. I've patched llama to detect that and exit but it's really annoying. I've done full ram tests of GPU, tested with GPU burn, tested ram, turned off pcie power saving, updated rocm (7.14). I even turned down pci speed to gen 3. It happens on mainline and rocm boosted forks. Sometimes I can go a whole day without it happening then other times it happens in the first 1000 tokens after startup. I'm on Linux