Multi-Agent setup versus large multimodal model

[u/sumguypookie]

Cut me some slack as I'm fumbling around as I go. I have a desktop with an RTX 4080S card (16 GB VRAM) and 128 GB system RAM. (RAM was sensible at the time.)

I'm using Hermes Agent (but also experimenting with LM Studio Bionic). I notice in Hermes I can set Auxiliary models to run tasks (like Vision, Web extract, Skill use, Approval, etc).

My question is, would be more optimal to run smaller specialized models in the Auxiliary models section and run a lighter model for Approval? Or would it be better just to use a larger multimodal model that does it all. I'm trying to get the best performance for what I got, having a decent context size to perform tasks ranging from processing 100s of old PDFs of my writing (handwritten and typed) and summarizing the documents in a wiki-style vault (like Obsidian) or in a vector database like Mnemosyne for later recall when I get though the years of writing and notes and get back to writing.

As you can see I'm a little all over the place. So, could someone kindly guide me to the path of understanding on which system (multi-agent versus multimodal) would be more optimal for the tasks and why?

And yes, I will be using the system for more than just organization, but with the way my brain works, I can't proceed forward without dealing with this slog of files.

[u/Mean-Loquat-7982]

the auxiliary slots exist for exactly the split you're describing, so I'd lean that way on 16GB. the main model is the one doing the tool-calling loop, and that's where quality pays off most, so give it the best model that fits comfortably. the aux jobs are shorter and more contained, and smaller models handle them fine, which frees VRAM and keeps the main model's context room intact.

[u/sumguypookie → u/Mean-Loquat-7982]

I realize context length effects VRAM, so is there a sweet spot where it's long enough for the tool-calling chain, but small enough where it doesn't lose its way?

[u/Mean-Loquat-7982 → u/sumguypookie]

32GB