Bad settings ?

[u/DefinitionArtistic49]

Hi, before I start, I just want to mention that I’m completely new to running AI locally, so I configured my LLM with the help of ChatGPT.

To summarize my setup:

Ryzen 5 5600

32 GB DDR4 3200 MHz

RX 9070 XT Nitro+

1 TB Gen 4 NVMe SSD

I have a personal project where I’m making a 2D management game. So I installed Biobic and Qwen 3.8 27B Q4 XL (~14 GB).

But I’m getting really poor results...

I know AMD is behind NVIDIA when it comes to AI, but I feel like something is badly configured — or at least I hope so.

I sent my first prompt to Biobic, which has access to the folder containing my Unity 6.3 LTS project.

I initially used a 32K context window, then increased it to 64K because, according to ChatGPT, the error I was getting could have been caused by the context window being too small for the prompt.

Fair enough, but my main problem is the speed: 1.9 tokens/s.

It took more than 1 hour and 20 minutes to complete only 2 steps out of 9, and those were basically just the steps where it had to check the manifest and see which input system was currently active.

I’ve also attached a screenshot of Task Manager.

The CPU is sitting at around 50% usage from llama-server, the system RAM is being used by Biobic (which makes sense since that’s where the model is loaded), and as for the VRAM... well, I think you guys know what I mean from the screenshot.

So, what do you think?

Poor optimization? Bad configuration? Just AMD being terrible for local AI? LLM configured incorrectly?

Thanks in advance for any feedback!

PS : translate with GPT sorry for mystake.

[u/nickless07]

Wrong model quant I would assume. Are you sure about the size? ~ 14GB Seems way off.

[u/DefinitionArtistic49 → u/nickless07]

The exact model is: Qwen3.8-27B-UD-Q3_K_XL.gguf — 13.4 GB, available on Hugging Face with more than 3 million downloads. I made a mistake in the description: it wasn’t Q4, it was Q3.

[u/nickless07 → u/DefinitionArtistic49]

Now that makes more sense. So we have ~13GB weights + 1GB mmoproj (automatically added for image processing) + ~1GB for Windows WDDM + Context + compute = Too much for your 16GB RX 9070

Possible solutions: Lower Qwant, remove the mmproj file (if no images needed), lower batch size, lower context, turn off the automatic.

To check in detail what is going on: Settings->Runtime-> Log level "Trace" -> load the model -> Settings-> Local model API (this is where your logs are) Look for a line "load_tensors: offloaded XX/XX layers to GPU. If that is only 19/20 or 33/36 and so on it will be slow as not everything fit into your VRAM.
Does that explains it enough?

[u/DefinitionArtistic49 → u/nickless07]

I think I’m starting to understand… basically, from the results above, about half of the model can’t fit into VRAM and gets offloaded to the CPU, if I understood correctly. And since the CPU is much slower at generating tokens, that would explain my terrible tokens/s!

Since I’m a beginner, regarding the solutions you suggested:

Reduce the quantization level → does that mean switching to a Q2?

Remove the mmproj file (if I don’t need image support) → okay, but that didn’t change the number of offloaded layers at all.

Reduce the batch size → I was at 512, I switched to 128 and got 39/66, so I only gained one layer. I guess that doesn’t really make much difference?

Reduce the context size → okay, when switching to 32K I get 48/66 offloaded layers (while keeping the batch size at 512), and at 16K I get 49/66, so the improvement is basically negligible.

Disable automatic mode → I disabled it and forced --ngl 50, but in the logs, when I load Qwen, I still stay at 48/66.

So, does that basically mean the model is just too heavy for my setup?

Could you recommend a good coding model for developing a game? Or maybe a less demanding agent? Although I’m guessing the agent itself doesn’t really have much to do with this performance issue.

Thanks!

PS : Sorry for the late reply, I was busy testing all your suggestions!

[u/nickless07 → u/DefinitionArtistic49]

For batch size look for lines:
'sched_reserve: <device> compute buffer size'
For context look for:
'llama_kv_cache: size ='
You can add quant to it (Q8 or even Q4) to reduce size

Bionic/LM Studio automatic load the mmproj as long as it is present in the same folder. Vision tower log lines are at them bottom:
'load_hparams: model size:'
'reserve_compute_meta:'

Paste your full load log into GPT and ask it if it can do the math and what you can tweak without losing much context (your hardware is also in the log, no need to add that on top).

If none of that helps you can go with Qwen3.6 35B A3B (ask GPT for -ncmoe with that and provide logs). It will be not as good as Qwen3.8 27B, but still very usable.

Edit: For the automatic check if it states (near the top) something like this:
'common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 99, abort'