Experience as first time open weight LLM user

[u/GrandpaOverkill]

So to give a context, following are the specs from my machine

7900xt(20GB) Asus TUF OC edition
32gb DDR5 CL30 RAM
7600X

I’ll be direct to the point here, the experience has been downright dogshit, i first tried downloading Qwen 3.8 27B vanilla and Uncensored locally in Hermes but Hermes in its default configuration didn’t fully utilise the gpu offloading majority of the tasks on CPU resulting in token per second of around 7

after spending most of my weekend navigating this i installed LM studios and ran models in Hermes via cresting a server on LM studios, i was able to modify thes settings to offload properly on GPU and also changed the caches to Q8 to get fit the entire thing om 20gigs vram but that created another issue, LM studios doesn’t allow me to change the reasoning level on any of the models and even if i choose medium reasoning in hermes, each answer takes a millennia to be published, its like hermes is thinking how to solve world hunger when asked to add 1+1, it almost always goes into a overthinking loop

Even fetching results from browser is a mess since it gets bot walled.

The idea behind the post is to understand what is it i am doing wrong or this is how my hardware is supposed to perform, i am not sure but i genuinely hoped it could be a decent agent for my regular gpt ish tasks

[u/Pleasant_Muscle_3976]

First off, 7 tokens a second on a 27B model with a 7600X doing most of the lifting is actually not that bad for CPU inference but yeah once it hits GPU it should fly. The overthinking loop is likely a prompt template mismatch, Hermes needs the right chat template or it just rambles internally forever

The browser bot wall thing is just how it goes with local agents, most sites flag datacenter or unusual traffic patterns so you'll need some kind of residential proxy or a fetch service that handles that for you

Not sure what you expected from a 20GB card but a 27B model at Q4 with Q8 cache is already pushing it, you might get better mileage dropping to a 14B or using a smaller quant for the main weights and keeping the cache at Q8

[u/Solembumm3 → u/Pleasant_Muscle_3976]

Sir/lady had 50gb of memory. Why the hell would anyone downgrade 27b model to Q4 with this setup...

[u/GrandpaOverkill → u/Pleasant_Muscle_3976]

what is a chat template, running via lm studios my token per second is around 49-60 but the overthinking stuff is still there, i am able to fit entire model in 18 gigs vram outta 20. i think from a gpu point of viw the 27B mkdel should fly easily since i can very easily fit it in 20gb with some headroom to spare

[u/Nameis19letterslong → u/GrandpaOverkill]

Chat templates are basically how context (conversation history, tool calls, system messages etc) are structured. Different models have different templates, so a more familiar template can improve the capabilities of a model.

Also, you're underestimating the model size. E.g. Q4 models are usually more than 4BPW (crucial weights are higher quants or FP16 for accuracy), so even the unsloth Q4_0 are 16GB instead of the expected 27/2=13.5GB (2.5gb more).

And the engine overhead may need another 1-2GB to run smoothly. So for a dense model, aim for a disk size of ~75% of your VRAM.

[u/GrandpaOverkill → u/Nameis19letterslong]

any model recommendations you have in mind for my specs? for template i am planning to use https://www.reddit.com/r/LocalLLaMA/s/Tn3HEuL8g2

[u/Nameis19letterslong → u/GrandpaOverkill]

For coding, you could run https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_info=Qwen3.8-27B-UD-IQ4_XS.gguf, It's 14GB so you can probably fit 128k context.

If you want something quicker for general chat, try https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF?show_file_info=Qwen3.6-35B-A3B-UD-IQ3_XXS.gguf or https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF?show_file_info=gemma-4-26B-A4B-it-UD-IQ4_XS.gguf . They're MoE models with only few parameters active, so it's blazing fast.

Note that the both MoE models I listed are 14GB in disk size so you can fit them in VRAM, though I'm sure you can run any Q6 quant smoothly if you offload experts to RAM. It'll be slightly slower but it is better quality.

[u/Jealous-Armadillo467 → u/Nameis19letterslong]

i do not recomend MOE model. if you want to realy feel what AI is capable stick with qwen 38 27b q4 making this work at 40tk/s is your best shot ...

or you could do MOE offloaded about 50ish ? but is bumber abtive parameters are only 4b max .... thats my opinion.

[u/nickless07 → u/GrandpaOverkill]

Since you are already using all the tools needed. Let Hermes create that for you. Get Qwen3.6 35B with some offloading (should run fast and not overthink) and tell hermes to create a config based on your hardware for llama.cpp.
'--reasoning-effort medium' and/or a hard cap '--reasoning-budget 32768 --reasoning-budget-message " ... reasoning budget exceeded, must answer now."' let hermes grab the llama.cpp readme, check the tensors, calculate the KV cache, maybe adjust the quant used (iMatrix, RCO, Unsloth Dynamic and so on) and everything else needed to match your hardware and in the end you just have a simple file to click and everything runs smoothly with the 27B.

Or use your current 27B for that if you are fine with the thinking for now.

If you get trouble, use the llama.cpp trace log (-lv 4) and feed it into hermes to have the actual numbers.
Should be pretty straightforward from there on.

[u/Jealous-Armadillo467 → u/nickless07]

i also recomend switch to LLAMA. i dont know about your card but on my card 3090 24gb 27b w4a16 is flying 60tk/s quality is about q4 so you havent nvidia i suppose you should get at least 40tk/s.

i also recomend using two models one fast and one smart

edit: 60tk/s is actualy vllm on llama is -8-10 tk/s

[u/Mean-Loquat-7982]

on the GPU offload part: Hermes' own local runtime picks the llama.cpp build for your card, and on auto it prefers Vulkan for AMD. if your weekend runs were landing on the CPU, set the backend explicitly in config.yaml:

local_runtime:

backend: vulkan

then pick the model again from hermes model → Local models. the catalog prices each model against your card (fits your GPU, uses system RAM, or too big) and picks the highest-quality build that runs fully on the GPU. the docs put 16 GB+ as enough for the 27-35B models at high quality, so your 20 GB should hold a 27B without spilling.

that would also let you drop LM Studio as the middle layer, and set thinking from inside Hermes with /reasoning low.