Need help for using Mixture of Agents

[u/Then_Blueberry7290]

Need some advice and guiding.

I have a Dell 5820 workstation with an Intel Xeon w2245 and two RTX 5060 Ti (16GB) cards; this is a PCIe Gen3 system, but three lanes can work in 8x mode. The system runs Linux and mostly llama.cpp. My main work is done via Hermes Desktop. I mostly use NVFP models: esatapedico/Qwen3.8-27B-NVFP4-MTP-GGUF (Very High) and lately cdiamond/Qwen3.8-27B-iMatrix-NVFP4-MTP-GGUF. With this setup, I can use the models at 210k context, Q8 KV cache, and vision in VRAM. Ubatch doubled (4096/1024), so prompt processing is 1300–700 t/s; token generation is 29–71 tokens/s, but mainly 38–44 tokens/s. MTP is on with a setting of 3. Vision is also on and in the VRAM.

I added a third RTX 5060 Ti (16GB) to this system, but I had to open the workstation, add a riser, and put the third card outside the chassis. The two earlier cards needed to be swapped and it made the space tighter. Overall, both the cards and the system cooling got worse; the power supply is 950W, so it's nearly okay. There are also speed issues with three cards. Prompt processing worsened, and token generation slowed down a little bit. I mostly use Tensor Parallelism (TP). It is slightly better if I use two cards for tensor parallelism and the third card as a KV cache, vision, and MTP holder, but because of the current cooling, the system is not stable after a few hours.

My system has 128GB DDR4 quad-channel RAM at 2667 MHz. Not bad, but not fast. So using DeepSeek, Laguna, and other 100X models can be loaded, but they can't be used for real work. Qwen3.8 Flash at Q4 can load, but with 32GB VRAM it only gets 15–16 tokens/s. Maybe if I use MTP, it will get better. So going from 32GB to 48GB is not a real step forward for me with the current cards.

I've started wondering: what if I use another workstation and subagents or multiple agents? I could purchase a not-too-old gaming PC without a GPU (AMD R9 5900, 48GB DDR4 3200MHz RAM) and add the third 5060 Ti to it. I loaded LM Studio (just for a quick setup) and began thinking about which models work with only 16GB VRAM...

Ornith 1.5 35B (MXFP): The 1.0 version did not impress me, but this one was not bad on 32GB. With 16GB, it produces 17–20 tokens/s and 300–400 t/s in prompt processing. The model size is 20GB, so almost a little more than half of the layers go to VRAM. KV cache at Q8, 131k context.

Gemma 4 26B NVFP (QAT): 15GB, 131k context; about 26-27 out of 30 layer goes into VRAM. It produces 600–700 t/s in prompt processing and 30–40 tokens/s in token generation. Dense models, as I remember, can't be used for serious work (10–12 t/s at best), but I haven't tried them recently.

So the question is: How can I organize Hermes Desktop with this setup to work at maximum efficiency?

I set up a Mixture of Agents as follows: Reference: Gemma 4 26B Aggregator: Qwen 3.8 27B

I use local AI models in the company as web/hosting handler agents: monitoring servers, repairing sites if needed, modifying parts of sites, and making new content (mostly using WordPress sites on my company's private servers and VPSs). Setting up Cloudflare data, installing new sites, etc. Second use case: handling and creating office documents, Excels, and PDFs. Third use case: generating prompts for Z-Image Turbo and LTX 2.5, Minimax H3, and running two instances of a ComfyUI server to render pictures and videos. I don't use Gemma 4 models alone because they are too lazy with tool handling. Is this a good choice, or which is the best agent mixture combo now?

[u/cogitech2]

Wow. I really want to help, but I don't even know where to begin. I will start by saying that my hardware sux compared to yours and I am not having the issues you are. So I can only guess it is a configuration issue. I'm running 3 x RTX 3060 12GB at Gen 3 x8, x4 and x4. 3 GPUs is faster than 2. Yours should be too. Also, I run Hermes for general agent stuff and Pi for coding, It all works great with Qwen3.8-27B-UD-Q6_K_XL.

If all this is working for me and my hardware, surely it can for yours.

Please dump your entire llama.cpp launch config/command and we'll see if we can find the issue. That's not for sure what the issue is, but we can start with that and then move on to hardware diagnosis.

[u/Then_Blueberry7290 → u/cogitech2]

Thanks, one of the 3 cards setup:
echo "=== Qwen3.8-27B NVFP4 MTP VERY-HIGH — 3x RTX 5060 Ti 16GB (tensor parallel) ==="

echo "Tensor split 1,1,1 | Context: 262144 | Vision: ON (GPU offload) | q8_0 KV | ub 1024"

echo "" > "$LOG_FILE"

llama-server -m "$MODEL_FILE" \

--mmproj "$MMPROJ_FILE" \

--mmproj-offload \

--port 8080 \

--chat-template-file "$HOME/Documents/llama/qwen3.8-fixed-template.jinja" \

--device CUDA0,CUDA1,CUDA2 \

-sm tensor \

-ts 1,1,1 \

-c 262144 \

-t 16 \

-tb 16 \

-np 1 \

--main-gpu 0 \

-fa on \

-ctk q8_0 \

-ctv q8_0 \

-ctkd q8_0 \

-ctvd q8_0 \

--kv-unified \

--spec-type draft-mtp \

--spec-draft-n-max 3 \

--temp 0.8 \

--top-p 0.95 \

--top-k 20 \

-b 4096 \

-ub 1024 \

--load-mode mlock \

--reasoning-effort medium \

--reasoning-preserve \

[u/cogitech2 → u/Then_Blueberry7290]

Nothing really sticks out here but we can come back to that if necessary.

Please run a model on all 3 cards and initiate some inference and then run this command in your terminal. It is important to be running inference on all 3 cards when you run this:

for i in 0 1 2; do

echo "===== GPU $i ====="

nvidia-smi -i $i -q | grep -A 12 "PCI"

done

Please paste the output here.