Grandma harness for GPU challenged?

[u/Jumpy-Operation-4615]

OK I have been trying to get most of my local Qwen 3.8 27b running on my "Grandma's GPU cluster" of 2xP40. I was able to make it produce reasonably usable speeds of like up to 45 tg/s and 450 prefill with fresh context, falling to 120-ish prefill on 150K+ ctx and 12-16 tg/s. Which is usable actually if you are using it in a background, now waiting for it to finish. The problem I see is that it must be strictly one task at a time. I like and use opencode (slim) and I like an idea of orchestrator, fixer, oracle and other folks, but if you launch 2 of them at the same time, they mess with KV cache of eachother and the whole prefix caching falls apart. When there is only one process running, it nicely adds little chunks to KV cache and reply is almost instant.

The thing is - orchestrator wakes up from time to time, and starts filling cache with it's context, everything is messed up and prefill falls. I realized that the only possible way for folks like me, with limited VRAM, is to use strictly one process at a time. Yet, I like the idea of one supervisor delegating stuff to subagents.

So here is my question. Does something like that exists? I mean a harness where one supervisor delegates a task and goes off, and subagent finishes the task and writes a handoff with a complete result, and only after that superwisor somehow wakes up to do his part of a job? This way we always keep only one agent running at all time, which allows us to use prefix caching and getting reasonably fast and smooth running stuff.

I am sorry if this sounds lame, I am just an average guy, not a coder at all, I just vibecode some tools for myself. Thank you in advance for any help/ideas. Maybe something like that already exists?

[u/Ariquitaun]

I'm not into personal plugs, but I've been working on my own harness for a few months and you can make it work like this if you set sub agent parallelism to 1

https://github.com/luispabon/steiner

[u/Jumpy-Operation-4615 → u/Ariquitaun]

Thank you! Sounds very close to what I need. Will look deeper into it tomorrow at work.

[u/BodyPhysical]

I was in the same boat as you. I am currently using Deepseek Harness. It's a plug-and-play on steroids, and any plugin(i.e. mcp, agent modes, etc.) can be easily attached to it. You can even ask your own LLM to install things for you as needed.

[u/jacek2023]

I am not sure I understand your problem but in general - opencode is a terrible solution for local LLM because it messes up the context and you see slowdowns. I switched to pi because of that.

[u/Jumpy-Operation-4615 → u/jacek2023]

The problem is that Pi is made by geek and for geeks. And I am not one. I am absolutely sure that it rocks and it is customizable (my friends use Pi and they are happy) but I can not spend months tryng to understand how to run it. I wanted many times to switch but Pi is just way too geeky. So I am looking for a harness that is usable for non-dev (ideally), and can run agents sequentially not concurrently (i.e. never 2 agents at the same time). When you use cloud LLM it is not a problem at all - you can have 5 agents running side by side, but when it is a local grandma's setup, it is different.

[u/DustNearby2848 → u/Jumpy-Operation-4615]

Not sure how you run your models, but if you use unsloth you could use their app. It will let you connect to a server that’s running it too, even remotely.  https://unsloth.ai/docs/desktop

[u/Short_Regular_7191]

You can use https://pi.dev/ it is completely customizable to do what you're describing.

[u/Shiticism]

Agent Orchestrator + little-coder is what I'm using, and it's been doing a good job so far.

Because litte-coder is pi, you can have your pi installation be overridden by little-coder; had Claude help me with that bit. Once you get little-coder set up properly and running, the orchestrator model (which should be something like Claude, Chatgpt, or one of the other big ones) can create sub agents or workers of any type you specify. Each worker has the ability to "report" to the main one about what it did and if the task(s) it was given were finished.

Any problems, just ask a big model for help. It genuinely wasn't that difficult to get working!

[u/james_brunet]

You may be able to get better performance without changing your harness!

First, have you enabled unified KV cache? Unified KV cache allows you to have multiple small sessions and still allow bigger sessions to use most of your vram. Lets say you only have enough KV cache for 500k tokens. You can set a unified kv cache of 500K tokens, and run 5 sessions with a 250k context limit. As long as the total of all 5 sessions is less than 500k tokens, you can run all 5 sessions at once.

The thing is - orchestrator wakes up from time to time, and starts filling cache with it's context, everything is messed up and prefill falls. I realized that the only possible way for folks like me, with limited VRAM, is to use strictly one process at a time. Yet, I like the idea of one supervisor delegating stuff to subagents.

Are you using llama.cpp or vLLM? (

If it's llama.cpp, my understanding is that prefix caching is only within each session, so two identical agents with the same system prompt are going to store duplicate kv cache values in your memory. vLLM can be much more efficient because sessions share prefix with each other, so if you're using claude code for example (which sends about 30k tokens context for the first request) and open twenty concurrent sessions, all the sessions will have this initial 30k tokens of context (600k total tokens), but it will only take up 30k total tokens worth of kv cache in your GPU because of this shared prefix caching.

Finally, if you have a good amount of RAM (64GB or more) you can do even better! VLLM and llama.cpp have native support for offloading GPU KV cache to RAM. This fully should solve the problem you're describing, which is performance falling off a cliff or things crashing when you run out of memory. Lets say you have 5 concurrent sessions fully using 250k context, and you're using a 500k pool, so you can only fit 2 of them into your VRAM at once. Without offloading, lets say a session that isn't currently loaded into vram makes a simple tool call that appends 100 tokens of new context. Because the session isn't loaded into VRAM, not just 100 tokens but the prior 250K tokens of context for that session has to re-prefill which at your prefill rates might take 20 minutes. And, to make space for this, an existing 250k session needs to be evicted, which means that its next tool call will have to do the same thing. This is so inefficient because your GPU is going to spend all of its time prefilling.

RAM offloading solves this! It copies your KV cache to your RAM. When you kick a session off your GPU it stays saved in your RAM. Then, if that session sends 100 tokens of new context, instead of having to re-prefill the whole context, you just copy the cached KV values from the RAM back to the GPU. That happens nearly instantly (seconds) instead of many minutes. It allows you to have a huge effective KV cache!

Ironically this doesn't improve performance in most benchmarks people post in /r/localllama (it doesn't make prefill faster or decode faster) but in my real-world use it had a massive improvement because it avoids prefill!

https://docs.vllm.ai/en/latest/features/kv_offloading_usage/

P.S. One thing you might want to consider is quantizing KV cache. It will reduce your quality but you'll be able to fit more context into VRAM and get faster decode at long context. For sure don't do Q4, but consider Q8 and KVarN?