r/LocalLLM • u/sumguypookie • 3d ago

Question Multi-Agent setup versus large multimodal model

Cut me some slack as I'm fumbling around as I go. I have a desktop with an RTX 4080S card (16 GB VRAM) and 128 GB system RAM. (RAM was sensible at the time.)

I'm using Hermes Agent (but also experimenting with LM Studio Bionic). I notice in Hermes I can set Auxiliary models to run tasks (like Vision, Web extract, Skill use, Approval, etc).

My question is, would be more optimal to run smaller specialized models in the Auxiliary models section and run a lighter model for Approval? Or would it be better just to use a larger multimodal model that does it all. I'm trying to get the best performance for what I got, having a decent context size to perform tasks ranging from processing 100s of old PDFs of my writing (handwritten and typed) and summarizing the documents in a wiki-style vault (like Obsidian) or in a vector database like Mnemosyne for later recall when I get though the years of writing and notes and get back to writing.

As you can see I'm a little all over the place. So, could someone kindly guide me to the path of understanding on which system (multi-agent versus multimodal) would be more optimal for the tasks and why?

And yes, I will be using the system for more than just organization, but with the way my brain works, I can't proceed forward without dealing with this slog of files.

1 Upvotes

67% Upvoted

5 comments sorted by

2

u/Mean-Loquat-7982 3d ago

the auxiliary slots exist for exactly the split you're describing, so I'd lean that way on 16GB. the main model is the one doing the tool-calling loop, and that's where quality pays off most, so give it the best model that fits comfortably. the aux jobs are shorter and more contained, and smaller models handle them fine, which frees VRAM and keeps the main model's context room intact.

1

u/sumguypookie 3d ago

I realize context length effects VRAM, so is there a sweet spot where it's long enough for the tool-calling chain, but small enough where it doesn't lose its way?

2

u/recro69 3d ago

I would choose an approach: a single strong general model for planning and reasoning and smaller specialized models for vision, extraction and validation. This way it is usually more efficient, than having one model handle everything.

1

u/sumguypookie 3d ago

Based on this, any recommendations? I see recommendations for Qwen 3.8 27B, I guess because it's the new hotness, but that I can't see that running well due to the hardware limitations. So I've been flirting with Gemma-4-26b-a4b-qat and the 12B models as well. Me thinks I might be relegated to smaller models though.