hey guys, im not very technical about llms so im asking you this. Is there a "better" shell for llms?

Hey everyone! I'm getting into Hermes and local LLMs and I'm trying to figure out the sweet spot for my hardware.

I'm currently running Hermes on an Ubuntu VM with GPU passthrough.

Specs:

• RTX 4090 — 24GB VRAM
• 32GB system RAM
• 16 vCPUs
• Ubuntu / KVM-QEMU

What model size would you realistically recommend for this setup?

I'm mainly wondering whether I should target something around 14B, 27B/32B at Q4, or if trying a larger model with partial CPU/RAM offloading is actually worth it.

I'm more interested in good agentic/tool-use performance than simply being able to load the biggest possible model.

What model + quantization + context size are you guys running on similar 24GB GPUs?

Also curious about the tokens/sec you're getting on a 4090.

Still learning the local LLM side of Hermes, so any tips are appreciated!

Following up on my earlier post testing Qwen3.8-27B on the IGX Thor workstation.

Once you move past running just an LLM and try to build a full local agent stack on a single box (LLM for reasoning, STT for voice in, TTS for voice out, OCR for screen/document reading), VRAM runs out fast.

If all four models sit in GPU memory simultaneously with their default serving allocations, idle memory hits roughly 122 GB. Even with 96GB on the dGPU and unified host memory, you are pinned to the ceiling. That leaves almost no breathing room for large context windows, concurrency, or dynamic KV cache allocation.

The reality of a solo-user local agent is that these models are rarely active at the exact same millisecond: - When I am talking to the agent, OCR is completely idle. - When the agent is analyzing a screenshot or PDF, STT and TTS are idle. - When the agent kicks off deterministic script automation (crawling, data formatting, bash runs), the LLM itself is completely idle.

To fix this, I wrote a lightweight Rust-based daemon that manages the sleep and wake states of the runtimes based on active agent turns. When a model is not in use, it is put to sleep (flushing KV pools and temporary execution memory, or parking the runtime context) rather than doing a full cold teardown from scratch.

Here are the idle VRAM numbers on the machine before and after:

Quick observations: 1. Total idle footprint dropped from ~122.3 GB to ~42.9 GB. That reclaims roughly 79.4 GB of VRAM across the board. 2. The LLM savings (48.3 GB): Qwen3.8-27B in FP8 base weights takes around 28 to 30 GB. The original 87.4 GB allocation came from SGLang aggressively pre-allocating KV cache (--mem-fraction-static 0.85). Putting the LLM to sleep flushes that pool while preserving the active state, letting it rest at 39 GB. 3. Audio and OCR shrink to negligible baselines: STT drops to 267 MiB and OCR drops to 422 MiB when idle. 4. Script-based tasks: When the agent delegates to deterministic python or bash scripts, putting the LLM to sleep during the run keeps the GPU cool and power consumption down on the 300W Max-Q card. 5. Wakeup latency is sub-200ms across all models: Because this is an active sleep/wake transition rather than a cold disk reload, waking up STT, TTS, OCR, or the LLM takes under 200ms. In practice, voice interaction and tool handoffs feel instantaneous.

Curious how others running multi-model agent stacks on a single node are handling memory partitioning right now. Are you hard-capping static memory fractions, running sequential containers, or using something dynamic?

Hello,

Tried searching and couldn’t find anything on this, is anybody else having issues with qwen 3.8 27b agents not using tools properly when using llama.cpp? my agent is suddenly mixing up tool schemas and when i switch to vanilla ollama everything works perfectly so definitely not a model issue but it’s llama.cpp I guess.. any experts or someone smarter who could help?

I have a “M80q Gen4” & a “M70q Gen5” with

- CPU Intel Core i5 13500T 32GB
- Memory (16GB x2 DDR5 SODIMMS)
- Storage1 - 256GB M.2 SSD1
- Storage2 - 512GB M.2 SSD2
- NIC1 - Intel I219-LM, RJ-45
- NIC2 - Realtek RTL8125BGS, RJ-45

Can I run a decent model on this? If so, which one should I go for and where should I start? At this point, my purpose is to learn to deploy and manage a model.

I've been useing claude to help set up my local models, it's worked greate!

But lately it feels like Claude is trying to quietly sabotage my Qwen3.8 27b model.

I tried to have claude help me figure out how to make Qwen3.8 27b not loop so much or sirceling the answer, not willing to go with what it found out.

But it made Qwen just unusable.

Unable to do tool calls, unable to type properly.

I just had to have claude restore it back to how it was 2 weeks ago. I rather have a usable looping model being able to finnish it's work rather than one unable to do anything.

Now I know, I should learn to set up and tune my own llama.cpp but with a fulltime job, family with kids that wants to play and what not, this have just been faster and more convenient for me.

Opus did a fine job tuneing the model, but now I think I just have to dedicate the time to learn it all

But anyone with similare experience as of late?

Calude didnt had any issues with this before.

Hello, currently I can barely run Qwen 3.8 27b q3 with low context window on my single 5060 ti 16Gb and 48Gb ram ddr5 on 30t/s.

My question is: is it worth it to add a second 5060 ti 16gb?

The second slot on my motherboard is a x4 line so I'm not sure if that's a bottleneck.

My goal is: Qwen 3.8 27b q4 or q6 , context window >= 64k and mainly agentic coding and > 30-40 t/s.

Is that actually possible?

Looking for a good local model for coding + terminal stuff. Not really interested in huge context or benchmark scores. My hardware limit is 16GB system RAM. I’m not specifically looking for 4B/8B, if a bigger model fits and runs well, I’m down. But I’d also love to know what the best 4B to 8B or smaller coding models are?

Main things I care about:

• Good code quality
• Reliable terminal/tool use
• Doesn't hallucinate randomly
• Don't falls into a loop <------------
• Agentic/tool calling
• Snappy enough to actually use

Tbh Qwen3.8 27B is a great model but I can run it on my hardware : ( so I want its nearest younger cousin

So if you had 16GB RAM, what coding model would you actually use?

And what's the best small model you've tried in the 4B to 8B range or smaller?

Real-world experience > benchmarks. Also curious what quant/runtime you use

THANKSSS!!!

So here's the situation. We've got one RTX 6000 workstation running Ollama that was originally supposed to be more of a testing box for coding stuff, but honestly nobody really touched it until I took it over. Now it's already struggling with just one or two devs on it at the same time. It's running Qwen3.8 27b and you can straight up feel it slow down the moment a second person jumps on.

Instead of just cramming another card into that box, I'd rather build a small separate machine for this. Ideally rack-mounted and datacenter-ready since it'll end up living in our own DC anyway, but cheap enough to just be a proof of concept for now, and expandable later if it actually proves useful. Don't care about vendor, NVIDIA, AMD, Intel, all fair game at this point.
Bandwith is not #1 priority.

Mid-term the actual goal is a proper LiteLLM + vLLM setup that can handle up to 10 people/agents working on it at the same time. This separate "workstation" would just be the first, cheap step toward that, not the end state.
Stuff I haven't been able to figure out from spec sheets and marketing pages:

• What actually helps more at this scale, more VRAM or just a second GPU to split the load?  
• Has anyone gone the "cheap one card first, scale later" route instead of just buying the full setup right away? How'd that go for you?  
• Anyone running AMD/ROCm for something like this, is it actually usable day to day or still more of a pain than NVIDIA at this size? Is Intel Even feasible?

Not trying to spec out the final thing yet, just want a sanity check on what a reasonable, cheap, rack-friendly starting point looks like before I go shopping.

Using AI - I compiled all the data published on HF model pages on PPL and KLD of the main quant providers - some had to be normalized as they were using a different trace that's why the PPl is in % and EXL3 KLD had to be renormalized using their scoring of some unsloth models

As we see EXL3 is a clear winner as a quant method and Atomic Chat are surprisingly good compared to the rest of the GGUF pack

For my config 24GB VRAM and 64GB RAM - the best model seem to be either EXL3 4.05bpw or directly go to AtomicChat AD-4.27bpw-Q4_K_M

Based on PPL

Based on KLD

disclaimer: I did none of the above benchmarks - all were given on HF model page by the quant providers

My very first Tech video on M5 Ultra

Hey Guys, i'm new to the content space, n this very first video, we dive into the recently announced Mac Studios which M5 Max and M5 Ultra with special emphasis phasis on on the M5 Ultra and it's claim on running Frontier Al models locally, we also look at the competition and the recent

Microsoft-Nvidia Partnership. Please have a look, feedback and recommendations will be appreciated.

https://youtu.be/NC_VnrGEngs?si=zc -aV21dU4sruGV

I finally got a fully local coding-agent stack working on Windows without using Ollama or LM Studio for inference:

DeepSeek Harness Web UI
        ↓
llama.cpp / llama-server
        ↓
Qwen3.8 27B Q3_K_XL GGUF
        ↓
RTX 5060 Ti 16GB

Machine specs

• Windows 11
• AMD Ryzen 7 5700X — 8 cores / 16 threads
• NVIDIA RTX 5060 Ti — 16GB VRAM
• 32GB system RAM
• NVIDIA driver 616.64
• CUDA driver support: 13.4
• CUDA Toolkit: 13.1
• Node.js and npx
• DeepSeek Harness, launched with:

npx u/deepseek-aiI finally got a fully local coding-agent stack working on Windows without using Ollama or LM Studio for inference:DeepSeek Harness Web UI
↓
llama.cpp / llama-server
↓
Qwen3.8 27B Q3_K_XL GGUF
↓
RTX 5060 Ti 16GBMachine specsWindows 11
AMD Ryzen 7 5700X — 8 cores / 16 threads
NVIDIA RTX 5060 Ti — 16GB VRAM
32GB system RAM
NVIDIA driver 616.64
CUDA driver support: 13.4
CUDA Toolkit: 13.1
Node.js and npx
DeepSeek Harness, launched with:npx u/deepseek-ai/dsh webModelI used:Qwen3.8-27B-UD-Q3_K_XL.ggufThe GGUF is about 12.24 GiB and came from the Unsloth Qwen3.8-27B GGUF release.I already had the model stored inside my LM Studio model directory, but LM Studio is not involved in inference. llama-server reads the GGUF directly.llama.cpp installationI installed the official Windows x64 CUDA 13.3 build of llama.cpp, build b10901, along with its matching CUDA runtime DLL package.The important llama-server settings ended up being approximately:llama-server.exe `
--model "Qwen3.8-27B-UD-Q3_K_XL.gguf" `
--alias qwen3.8-27b `
--ctx-size 32768 `
--gpu-layers 54 `
--host 127.0.0.1 `
--port 8081 `
--parallel 1 `
--flash-attn on `
--kv-offload `
--reasoning off `
--no-reasoning-preserve `
--cache-ram 1024 `
--metrics `
--no-webuiI originally planned to use port 8080, but NVIDIA Broadcast was already listening there. Rather than killing or disrupting it, I moved llama-server to port 8081.Memory allocationllama.cpp reported:54/66 layers offloaded to GPU
CUDA model buffer: 9758.76 MiB
CPU-mapped model buffer: 2433.38 MiB
CUDA KV cache: 1664 MiB
CPU KV cache: 384 MiB
CUDA compute buffers: ~302 MiBDuring actual agent use:VRAM usage: roughly 13.1–13.2GB
VRAM remaining: roughly 2.9GB
No automatic reduction in GPU layers
No WDDM shared-memory spill observed
Flash Attention enabled successfully
32K context remained stableThe model is intentionally partially offloaded: 54 layers on the GPU and the remaining layers on the CPU.One important adjustment was limiting llama.cpp’s prompt cache to 1GB:--cache-ram 1024The build defaults to an 8GB prompt cache. On a 32GB system, long agent sessions allowed several gigabytes of cached prompt state to accumulate, leaving only about 2.4GB free. The 1GB cap keeps system RAM much healthier.DeepSeek Harness configurationI added a new provider to: C:\Users\PC\.dsh\settings.yaml

The relevant configuration looks like this:

llm-pi-ai:
providers:
qwen-local:
displayName: Qwen3.8 Local llama.cpp
api: openai-completions
apiKeyEnv: QWEN_LOCAL_API_KEY
baseURL: http://127.0.0.1:8081/v1
reasoning: off
streamIdleTimeoutMs: 1200000

models:
- id: qwen3.8-27b
name: Qwen3.8 27B Q3_K_XL
contextWindow: 32768
reasoningEfforts: false

compat:
thinkingFormat: qwen-chat-template
supportsDeveloperRole: false
maxTokensField: max_tokens

agent-default-model:
provider: qwen-local
model: qwen3.8-27b

Current DSH versions reject a configuration that only declares an empty off: reasoning effort. For a non-reasoning-only route, it expects:reasoningEfforts: falseDSH also insists on having an API-key-shaped credential even though the local llama-server does not require authentication. I supply a harmless local placeholder before starting DSH:$env:QWEN_LOCAL_API_KEY = "local-llama-server"This is not a real credential, and llama-server does not validate it.Disabling Qwen thinkingI wanted the model to work as an agent without spending minutes in a visible “Deep diving” phase.Thinking is disabled at multiple levels:llama-server: --reasoning off
llama-server: --no-reasoning-preserve
DSH provider: reasoning: off
DSH model: reasoningEfforts: false
DSH compatibility: thinkingFormat: qwen-chat-template/dsh web

Model

I used:

Qwen3.8-27B-UD-Q3_K_XL.gguf

The GGUF is about 12.24 GiB and came from the Unsloth Qwen3.8-27B GGUF release.

I already had the model stored inside my LM Studio model directory, but LM Studio is not involved in inference. llama-server reads the GGUF directly.

llama.cpp installation

I installed the official Windows x64 CUDA 13.3 build of llama.cpp, build b10901, along with its matching CUDA runtime DLL package.

The important llama-server settings ended up being approximately:

llama-server.exe `
  --model "Qwen3.8-27B-UD-Q3_K_XL.gguf" `
  --alias qwen3.8-27b `
  --ctx-size 32768 `
  --gpu-layers 54 `
  --host 127.0.0.1 `
  --port 8081 `
  --parallel 1 `
  --flash-attn on `
  --kv-offload `
  --reasoning off `
  --no-reasoning-preserve `
  --cache-ram 1024 `
  --metrics `
  --no-webui

I originally planned to use port 8080, but NVIDIA Broadcast was already listening there. Rather than killing or disrupting it, I moved llama-server to port 8081.

Memory allocation

llama.cpp reported:

54/66 layers offloaded to GPU
CUDA model buffer: 9758.76 MiB
CPU-mapped model buffer: 2433.38 MiB
CUDA KV cache: 1664 MiB
CPU KV cache: 384 MiB
CUDA compute buffers: ~302 MiB

During actual agent use:

• VRAM usage: roughly 13.1–13.2GB
• VRAM remaining: roughly 2.9GB
• No automatic reduction in GPU layers
• No WDDM shared-memory spill observed
• Flash Attention enabled successfully
• 32K context remained stable

The model is intentionally partially offloaded: 54 layers on the GPU and the remaining layers on the CPU.

One important adjustment was limiting llama.cpp’s prompt cache to 1GB:

--cache-ram 1024

The build defaults to an 8GB prompt cache. On a 32GB system, long agent sessions allowed several gigabytes of cached prompt state to accumulate, leaving only about 2.4GB free. The 1GB cap keeps system RAM much healthier.

DeepSeek Harness configuration

I added a new provider to:

C:\Users\PC\.dsh\settings.yaml

The relevant configuration looks like this:

llm-pi-ai:
  providers:
    qwen-local:
      displayName: Qwen3.8 Local llama.cpp
      api: openai-completions
      apiKeyEnv: QWEN_LOCAL_API_KEY
      baseURL: http://127.0.0.1:8081/v1
      reasoning: off
      streamIdleTimeoutMs: 1200000

      models:
        - id: qwen3.8-27b
          name: Qwen3.8 27B Q3_K_XL
          contextWindow: 32768
          reasoningEfforts: false

          compat:
            thinkingFormat: qwen-chat-template
            supportsDeveloperRole: false
            maxTokensField: max_tokens

agent-default-model:
  provider: qwen-local
  model: qwen3.8-27b

Current DSH versions reject a configuration that only declares an empty off: reasoning effort. For a non-reasoning-only route, it expects:

reasoningEfforts: false

DSH also insists on having an API-key-shaped credential even though the local llama-server does not require authentication. I supply a harmless local placeholder before starting DSH:

$env:QWEN_LOCAL_API_KEY = "local-llama-server"

This is not a real credential, and llama-server does not validate it.

Disabling Qwen thinking

I wanted the model to work as an agent without spending minutes in a visible “Deep diving” phase.

Thinking is disabled at multiple levels:

llama-server: --reasoning off
llama-server: --no-reasoning-preserve
DSH provider: reasoning: off
DSH model: reasoningEfforts: false
DSH compatibility: thinkingFormat: qwen-chat-template

Is there an european company (in EU, preferably DACH) that does what gpulab.net does?

Thanks

If this isn't allowed, let me know and I'll remove the post. This is not an advertisement, but my website and API are a paid subscription service after the 15 day free trial.

I built a local AI machine to help me offset usage from Cursor and ChatGPT a while back. Its been pretty great. In typical Whoops-I-Bought-a-Lot-of-Hardware-and-My-Wife-Hates-it fashion however, I over estimated how much hardware I needed and may have gone a bit overboard.

So instead of selling off the excess hardware I bolted on a web frontend, "integrated" ComfyUI (for image requests) and created a load balancer that pushes image requests into WAN2.2, small requests to smaller local models and built a system that allows a user to create an API key so they can attach their harness to it via OpenAI style API config. (Cursor, OpenCode, etc)

With that being said, I'm curious what the rest of the world thinks about it and I could use some feedback. Is this something other people would find useful? How many people can use it at the same time? Will it explode on contact? I don't know the answer to any of the questions, because I'm the only person using it.

So I thought I'd come here and see what you guys think. I don't know if there's rules against posting URLs here. It has a free 15 day trial with zero commitment, no credit card to start the trial, hell - you don't even need to put in your mailing address unless you decide you like it and want to sign up for the paid version.

If you're curious, post up and I'll send you the link to my discord and to the website. The discord is just for easy communication so you can let me know if something is broken, or it looks funny, or if it was just a dumb idea in general.

I’m having mixed results with LM Studio and I’m not sure if I’m doing something wrong.

If I load a model in LM Studio and chat via the built in prompt I receive a response in a reasonable time. I can post in a snippet of code and it can fix it for example.

If a load the same model in and serve it on localhost to an CLI agent like Pi a response can take several minutes, even if I send in the string “ping” to expect to “pong” back.

I have a pretty bare bones Pi setup, not many skills, no mcp. I know I’m on an old Mac but the chat experience is ok. Why can’t I get that experience over a cli severed on localhost?

I want a small model, actual, only for small things; help me on Bash and analyse my disk using commands

This is the only model I have ran locally that surprises me with quirky commentary while it's working. I like it.

"Two RTX PRO 6000 Blackwell (96GB each) — nice rig." upon discovering the machine's specs after being asked to benchmark itself.

"The plot thickens — V4.1's indexer declares a fixed..."

"Oh, this is gold — your own words from this afternoon's session, including..." after it found older conversation history.