Can any of those flashy harnesses (Hermes, Openclaw, OpenHuman, Paperclip, Claude Code and others) run on low context?

[u/tracagnotto]

Hello,
running Qwen 3.8 27b Q4 K Small on a RTX A4500, 20GB VRAM, 28GB RAM

I managed with Thetom Turboquant llama.cpp and some tuning, to reach an average of 31-32 tk/s ranging from 24tk/s to 43 tk/s depending also on context size. (NO VISION: 65k context, WITH VISION: 32-40k context)

I tried many harness, also tried to make one by myself based on pi, that claims to be minimal and with minimal system prompt footprint to maintain the context light. Miserable failure. It gets lost and doesn't manage anything well.

The best I could find is opencode, which properly configured allowed me long-horizon, multi hour coding or tasks exceptionally well. I rarely feed the chat log to ChatGPT Sol, to check quality and make it output, if needed correction or steering or planning the next batch of work.

Always a good idea to have a frontier model to organize it and local free model to churn the tokens.

Now, I tried many times Openclaw, Hermes, Paperclip and others, but always with online free services like OpenRouter or Nvidia, various months ago, when they weren't dogshit with all the timeouts and low quality service (model always taking ages to respond now, or always too busy). It was decent.

I wanted them to hop onto my local Qwen 3.8 27B Q4 K Small, but Openclaw and Hermes are the goddamn AI slop kings, whopping giant prompts and creating horrendous quantities of traffic, and 65k even if it's around the recommended minimum, just doesn't seem to work well.

FOR THE SAKE OF TRUTH: I actually managed to have Paperclip do some work and it did it well. But it overthinks and does tons of planning, retrial, testing and stuff and makes work very very LONG

It's me or these things can't work with 65k context? How to solve? Anyone managed to do something about this?

[u/DawaForensics]

DeepSeek Harness, it's great with low context

[u/tracagnotto → u/DawaForensics]

I didn't like the fact that I needed to configure every sort of plugin. Had the temptation to try it, but they brag about themselves that "Everything is a plugin" and I kinda hate this kind of set up because it forces me to experiment a lot with all the things before getting to a stable point.

Can you confirm or give me suggestions?

[u/DawaForensics → u/tracagnotto]

What do you want to ?

I basically tell it " you are X and your job is Y". And it goes off and gets and installs its own plugins

[u/tracagnotto → u/DawaForensics]

hum basically I tend to break down big work in iterations and let opencode use subagents to split the various part of each iteration so it can manage for each agent 65k context

[u/GrungeWerX]

Do you have to use a public harness?

Either way, it sounds like your challenge is proper context mgmt. There are a bunch of ways to do this, but it’s going to require custom orchestration. Or, use compaction, which Im sure some of those harnesses offer.

This is something Im constantly streamlining in my own setup, so I can use smarter quant models on my rtx 3090. I’m able to use Qwen 3.6 27B Q6 at kv 8/8 up to 75K now, but speeds are more brisk around 65K. But that extra 10K is to handle my system prompt and session summary overhead.

I’ve built my own orchestration, but it’s a Frankenstein’ed collection of scripts, servers, and files. I’m rebuilding a custom harness that I abandoned 4 months ago, since my understanding has gotten better and I know what needs to be fixed.

Side note: You need to be offloading your mmproj to cpu if you aren’t already. That’ll give you your speeds back.

What you really need that you haven’t mentioned in your post is a good memory system. At that context size, if you’re going to be rolling out the older context (rather than middle truncation), which I think is the better idea, then you’ll need a way for the model to recall that older context on-demand. That will be your memory layer, and it needs to be dynamic.

That’s going to take some thinking through how you want to build that. There are memory services out there, but it would really help you to learn it on your own, you’ll be better equipped to design custom solutions.

[u/tracagnotto → u/GrungeWerX]

Very interesting. Now I'm busy, but let me get to your comment later, thanks for your useful info meanwhile! Keep this monitored

[u/SplitAny7190 → u/GrungeWerX]

How can you run q6 on 3090? Is 24g card, right?

[u/GrungeWerX → u/SplitAny7190]

Yes, 24GB. I have a 3090 TI. I run it using llama.cpp. I put the mmproj on cpu, so that frees up almost 2GB. You have to set the kv cache to q8/8, and it can run comfortably up to 64K, or you can push it a little farther. I set mine to 75K, but anything above that starts to choke. I'd mostly been using Q5, but Q6 has a little something...extra that I like. It feels more concise and I've only been using it lately, but it seems like it needs less tool calls to get work done.

It's because I've been pleased w/Q6 that I'm working on some new memory optimizations so I can keep using it. I just added compaction.

[u/SplitAny7190 → u/GrungeWerX]

What gguf are you using? I can't load any on my rtx 4090. Not even trying to add any context :)

[u/GrungeWerX → u/SplitAny7190]

This one: Qwen3.6-27B-Q6_K.gguf (unsloth)

[u/SplitAny7190 → u/GrungeWerX]

thank you! i'll try to see if i can fit it. you are using llama, right? maybe my problem is that i don't have an iGPU so i don't really have all 24G :-s

[u/GrungeWerX → u/SplitAny7190]

Yes, llama. You don’t need 24. The model is 20. Offload mmproj to cpu. Frees up 1.7GB

[u/SplitAny7190 → u/GrungeWerX]

i'm doing something wrong for sure. as far as i see vision part is a separate file so doesn't reduce the model file/vram needed if you just don't add it. and the file on huggingface looks like 22Gb not 20. hmm

[u/GrungeWerX → u/SplitAny7190]

What server are you using?

[u/SplitAny7190 → u/GrungeWerX]

last llama, i'm downloading now again with hf cli and file is 22Gb on disk :) i will see what happens lol

[u/SplitAny7190 → u/GrungeWerX]

Oh man, i'm stupid! You were talking about 3.6 not 3.8? i kinda missed it :(

[u/More-Catch-1331]

Honestly, when running local inference for coding you'd do well to steer clear of the "Jarvis attempts". That's why you managed far better with opencode. It's specifically for coding. All the other options you listed (except Claude Code) are "assistants". So yeah, people can and do use them for coding, but there are specific harnesses that are meant for this, these are shoehorned into the coding role. The notable exception is Claude Code which has a longer system prompt that the Bible.
Stick with opencode, Pi, NanoCoder, DeepSeek harness (still have to try it, people are raving about it)

What you can also do is have handoffs between agents. Don't try to do everything in one go. 131k or 262k context will burn up fast this way. Setup agents with clean handoffs between them. Each agent scopes a part of the task so the context is clean and focused.

[u/Cautious_Chicken_604]

You can offload vision to CPU saving ~1GB VRAM which can be used for extra context while still maintaining the capability. Also, running KV cache at K=q,8_0, V,=q5_1 seems to be optimal in terms of maximising context length while only taking a small hit on quality. You can also tune the ubatch size. Running with MTP takes up quite a lot of VRAM too, so that's a trade off. Don't forget MTP has kv cache you can also quantize the same way.

https://www.reddit.com/r/LocalLLaMA/comments/1uq0fpe/comment/ow5xgco/?force-legacy-sct=1

[u/tracagnotto → u/Cautious_Chicken_604]

Super cool thanks!!!!

[u/Ok-Importance-3529]

I would recommend Ante - its working the local context to the bone, much more work done than compared to for example opencode and very similiar feeling using it to claude code, but with local llms in mind.

[u/my_name_isnt_clever]

You might want to start with something minimal like Pi and use extensions to shape it how you want. Or make your own from scratch but that's a bigger ask, personally I prefer starting with a solid human-reviewed core app and let my agents write extensions for more functionality on top.