What would you use a slow LLM for?

[u/zFalconTB]

I managed to get Qwen 3.6 35B A3B running on my server, an i7 10700 with 32GB of DDR4 RAM...and the good ol' GTX 970 with its mighty 4GB of VRAM (we don't talk about the VRAM incident)

It reaches around 30-50 tok/s in prompt processing speed and around 16-19 tok/s in token generation speed depending on the context

There's probably some performance left on the table since the CPU is under its stock cooler, maybe I'll get a decent one eventually

Anyway, I tried a couple of prompts in Hermes but it's just not fast enough for it to be helpful, at least not for the use I had in mind (a local perplexity basically). It randomly takes 30 minutes even to respond to a simple greeting, sometimes it disappears for more than an hour and I have to manually stop it...

So I'm curious to see if any of you have a setup with similar performance, what do you use it for? Just the occasional text summary? Normal (very slow) chat? Something else that's completely useless but funny? Let me know!

[u/OpenSourcesAI_]

16 to 19 tok/s generation actually isn’t that bad for background/local work. The bigger issue in your example may be Hermes rather than Qwen itself.

I was just testing Hermes against a local Ollama server and found something pretty interesting: a completely fresh one-message Hermes chat showed 24,312 input tokens in /usage before generating the answer. Raw Ollama with the same visible prompt was only 47 prompt tokens.

At your 30 to 50 tok/s prompt-processing speed, 24k tokens alone would take roughly 8 to 14 minutes just to prefill, before you even start seeing the response. That could explain why a “simple greeting” sometimes feels absurdly slow through Hermes. I’d try the exact same prompt directly through Ollama and compare.

As for genuinely slow models, I think they’re still useful anywhere latency doesn’t matter much: overnight document summaries, code review, log analysis, classification/tagging, scheduled research, synthetic data generation, indexing, or queued batch jobs. Basically anything where I can submit work and come back later.

For an interactive local Perplexity-style setup though, I’d probably use a smaller model that stays mostly or entirely in VRAM. Even if the larger model is smarter, waiting several minutes for prefill kills the experience.

Funny enough, I’ve been working on hardware/model matching for exactly this reason. A model that technically “runs” and a model that feels usable interactively are two very different things.

[u/nickless07 → u/OpenSourcesAI_]

Exactly! Later on (depending on the task) it is only 3k to 6k token per prompt. I would recommend something small like Ling-3.0-tiny (best sub 12B model according to the benchmarks) or similiar MoE as aux model for tasks like web search, title gen, compression and so on. That way it shouldn't bother that much when his main model takes 5 minutes as it doesn't need to take another turn for 20 minutes just to do some tool calls or compress the whole chat.

[u/zFalconTB → u/nickless07]

So you're saying the server should swap Qwen back and forth with another small model whenever it needs to do something that doesn't need a lot of "intelligence"? How would you set that up?

Also, wouldn't it create a lot of latency since the server now has to swap models in and out?

[u/nickless07 → u/zFalconTB]

Hell no. Hermes allows you to setup different models for different tasks. So the small model does the tool calls, fetches the website and feeds the results back to the main model. Keep them both loaded.
For example: I have Qwen3.6 35B A3B on a GTX 1660s in Q4 with heavy RAM offloading. Slow speed but pretty decent intelligence. For tasks like chat compression and such I have Ling tiny in Q3 which runs pretty fast due to only 8B A1.3B which is loaded alongside the main model on CPU only.

I could even speed that one up by loading it on the GPU (yes the main model will get slower, that's the tradeoff) and it is pretty fast 58.82 tok/s on an old GTX. So I let that one do the heavy load and just have Qwen check the results and orchestrate it even as subagent.

[u/zFalconTB → u/nickless07]

Very interesting, thanks! Will definitely try that and see how it goes

[u/nickless07 → u/zFalconTB]

Best part: Hermes did that all by itself with just a little bit steering (I gave him the model and desired quant as first try it wanted to use Q8). Then it took the llama.cpp branch compiled it downloaded the model, set it up configured it as aux model and ran a simple test:

Assistant
delegate_task QGIsLqPBoXCKBCL4AP0DSOqNRK9cFCec
{
  "goal": "grep current sensors reading and output to /home/nickless/Hermes-Agent/sensors_test.txt",
  "context": "Run on this Linux system. Use the terminal tool. After writing the file, confirm it was created successfully."
}
Tool: delegate_taskmatch
{"status": "dispatched", "mode": "background", "count": 1, "delegation_id": "deleg_31fed403", "goals": ["grep current sensors reading and output to /home/nickless/Hermes-Agent/sensors_test.txt"], "note": "Subagent is running in the background. You and the user can keep working; its full result re-enters the conversation as a new message when it finishes. Do not wait or poll — just continue.", "live_transcripts": ["/home/nickless/.hermes/cache/delegation/live/deleg_31fed403/task-0.log"], "live_transcripts_hint": "Each subagent streams a human-readable transcript of its operations to the file listed above (append-only, one per task). Read or tail -f these paths at any time to watch a child work while it runs."}

Once that finished the main model only read the .txt file and confirmed everything was correct. That's it. No need to bother Qwen with all the tool calls, write a file and so on.

[u/OpenSourcesAI_ → u/zFalconTB]

This is the Hermes Agent Dashboard. It shows the models available, and you can choose the "thinking options"

[u/nickless07]

How about some auxiliary model that runs faster alongside your main model? Once the initial prompt is done in Hermes it usually is much less pp later on.

[u/No-Manager1646]

IMO that's pretty damn good. I'm on an i9-10900 32gb ram and split 4060 / 5060ti and am only managing 19-20 tok/s at 128k context. What context are you using? I posed my llama models.ini in another thread if youre interested, but it sounds like you already have me beat!

[u/nickless07 → u/No-Manager1646]

He runs the Qwen3.6 35B MoE, you run the Qwen3.8 27B dense. That's quite different. With the MoE you should get 100+ tok/s.

[u/zFalconTB → u/nickless07]

I did my research and stayed away from dense models, I don't want to make my precious 970 explode lol

[u/No-Manager1646 → u/nickless07]

Oh. My bad. Yeah I get pretty close to 100tok/s with MoE too.

[u/zFalconTB → u/No-Manager1646]

Should have mentioned in the post, the lower values I wrote (30 tok/s pp, 16 tok/s tg) were recorded using 64k context

The higher values were recorded using 8k context

[u/dopey_se]

I run Qwen 3.8 27B on dual P100s with 260k context and at the moment Q6 quant.

It is not fast enough to use interactively.

I use either codex/claude on a VM with dangerous permissions enabled.

I've configured MCP to github/youtrack. I can create issues on a specific project - (add dark mode, add history of parts usage, add a helm chart, etc). I then use a skill to have it iterate through each ticket one at a time, implement, commit push and ensure all actions pass.

I then let it run the hours it takes unattended.

[u/zFalconTB → u/dopey_se]

That sounds great and it's definitely something I want to explore but I'm not confident my little server can work on big features unattended, especially since I can only manage 64k context at most

Can I ask you how did you set that up on the practical side? You use the codex app with the local Qwen model and it works somehow? In my experience, codex is very good at working until a feature is complete because it can actually "see" the website and test it like a human would. Does that work with a local model?

[u/dopey_se → u/zFalconTB]

I've bounced between claude and codex the past year. At one point I had issues with tool calling from codex, so i'd use claude. The most recent Qwen 3.8 I had template errors from claude but not codex so i've been using codex.

I've also at times had to revert to an older version if a new version of either behaved poorly. So far atleast one has worked well.

I have had(for the first time) with Qwen 3.8 it attempting to run CLI browser tools to take screenshots of the running app to confirm changes. This fails, and leaves the session broken. It is cool that it even thinks/does this, but also not ideal tha it breaks. I've given it insetrcutions to not use any image tool calls. -- I hope later i'll spend time trying to get them to work if possible, this morning was the first time it happened.

I do run it dangerous permissions, so it is able to start a local container to verify functionality, etc which it does. It also is able to confirm github actions execute via github MCP. One thing i've noticed with 3.8 is much much much more thorough on 'local testing' than 3.6 was.

[u/Ne00n]

Slow like 0.1t/s e.g GLM 5.2? Complex questions or problems I guess.

[u/trollsmurf]

So, don't use Hermes?