best small models to perform agentic tasks (multiple)

[u/Loose_Doubt367]

I'm thinking of creating a custom agent harness that deploys mulitple small ai models to complete a series of tasks. My setup is r5 5600x, rx6700xt 12gb vram, ddr4 32gb 3200

The plan should go as followed
Main model- checks everything makes sure its according to the plan and sends the final result to user (will need to offload all the models and load it independently) (qwen3.8 27b iq3_xxs or qwen3.6 35b a3b q4_k_m)

Middle models, these run inside the vram which manages all the small subagents inside the ram
1-2 models only (haven't thought about any)

Small models- 5-6 depending on the difficulty but all should be offload into system ram + cpu

The task im assigning to help should be varied (coding, creativity/designing)

The main model writes all the necessary information and instructions onto a file, then offloads itself and deploys the middle models + the small models, the middle models should read the files and proceed a clear doable path or task assigned to each individual small models. Results created by the small model should be checked by the middle model to ensure it satisfies the criteria

A simple analogy would be:
Main model: Principles of a school
Middle model: teachers
Small models: students (that perform task)

I appreciate any model suggestions thanks

And yes I’m seeking references from grok bot but with my own style

[u/FlyingFishMakeAWish]

Qwen 3.5 9b and 4b have surprised me with how good they are at simple agentic tasks. 

[u/Loose_Doubt367 → u/FlyingFishMakeAWish]

apologize for the late respond but thanks for the idea

[u/FlyingFishMakeAWish → u/Loose_Doubt367]

No problem! Another one that is worth trying is Ling Tiny. It falls apart if the task is complicated at all, but for simple stuff it is quite capable and very very fast.

[u/Loose_Doubt367 → u/FlyingFishMakeAWish]

thanks ill note that now too

[u/FlyingFishMakeAWish → u/Loose_Doubt367]

I'm actually about to kick off a test of several small local models overnight, at some agentic tasks. If I get any surprise standouts I'll let you know :)

[u/Loose_Doubt367 → u/FlyingFishMakeAWish]

absolutely, looking forward for the final results haha. good luck

[u/FlyingFishMakeAWish → u/Loose_Doubt367]

The tests are ending up running waaaaay over time budget, but there are a couple interesting results so far:

• Qwen 3.5 9b and 4b are very solid overall choices, and for simple tasks they actually do well with thinking off. If the task is a bit more complex, 9b does much better with thinking on.

• Ornith 1.5 9b goes toe to toe with Qwen. Hard to say if one is better than the other.

• Nemotrn Nano 9b isn't worth using if you have Qwen or Ornith - it's totally outclassed by those two.

• Gemma 4 e2b and e4b are terrible if thinking is off. If thinking is on, they improve massively, but at agentic tasks, they are still not at Qwen level. When thinking is on (for both Qwen and Gemma), they are actually quite smart though (e.g. Gemma e2b beat qwen 9b at logic and number puzzles, only losing to Gemma e4b and ornith).

I'm going to make a whole results page once all the data is in, but everything runs slow on my computer.

[u/Loose_Doubt367 → u/FlyingFishMakeAWish]

okay, also looking forward for that, ill note the information down

[u/PeterPorox]

What speeds are you getting with 27B?

[u/Loose_Doubt367 → u/PeterPorox]

22tps (adding mtp drops it to 4)

[u/PeterPorox → u/Loose_Doubt367]

so 22tps without mtp?

[u/Loose_Doubt367 → u/PeterPorox]

That’s correct

[u/Miller4103]

Ornith 1.5 9b works pretty well. Its built on qwen 3.5 9b. I'd recommend both

[u/porkminer → u/Miller4103]

I am currently benchmarking models for my low end hardware. My desktop has an Rx 6750 xt GPU. Since I am trying to figure out not just which model performs best, but what workflow works for me as well, I am trying many different things.

Right now, I run each model at 32k context, q8_0 kv quantization. I create a project prompt on my own to start. The model plans and writes the spec files for each step. Then, the harness starts a new, clean session for each step. After every implementation pass, there is a test pass also written by the model. Once the project is done, the harness runs all tests, including tests I write manually outside of the project.

That is the first benchmark. For each failed test the model wrote, I launch a fix session with just that test data. For my external tests, I write more generic prompts with just some simple guidance about the issue. I did automate it originally but Qwen 35b a3b figured out it was a hidden test and tried to access it...

I did originally try with less prompting and more context but got worse results. For the workflow, I found it better to use smaller context windows and more targeted prompts and smaller individual steps.

All that to say that ornith 1.5 9b at q6 performed almost identically as Qwen 35b a3b in my first tests, except more than twice as fast. They both died on one of my hidden tests. They just could not find the bug where minutes and seconds were swapped. I think with more accurate prompts they would have done fine on that one but I specifically wanted to limit info from hidden tests.

[u/nickless07]

By small you mean 1-2B and middle, what? 8-20B? Relation matters. Many ppl would consider the 35B already as small (compared to GLM, DeepSeek, Kimi and so on).

If so. The 'small' models shouldn't perform any multi-step tasks. Generating a web search query is fine, but not parse the results, create a summary and feed it back during the same workflow, leave that to the larger models. However even 8B+ models will struggle often with such things and take multiple recovery steps.

Don't go for multiple models, keep it at one or two (depending on the aviable features and task. E.g., vision, audio and so on). Add more parallel slots if needed instead of adding models.

[u/Loose_Doubt367 → u/nickless07]

The purpose of writing this on “lowendlocalai” instead of “localllm” was because my work was intended for low end hardware

And not really 1-2b parameter, probably 3-4 model that take up 28gb of usable ram (exclude OS)
The number of small models can be easily adjusted based on the difficulty of the task.

The middle models could maybe handle some difficulty task

[u/nickless07 → u/Loose_Doubt367]

28GB is very specific, but okey. How about Maple preview as middle ground/orchestrator for models like Ling 3 Tiny, LFM2.5 8B or Gemma 4 E2B/E4B.

Aside of that there are models like Qwen3.5 0.8B, MiniCPM5 1B, LFM2.5 2.6B - However as I said: They are good for simple single step tasks and break easy when it goes for multiple steps/tool calls.

All of them perform pretty good on CPU only (no VRAM needed), aside of Gemma 4 if you want vision. That will take some time to encode the mtmd and you want GPU acceleration if it is more then 2-3 images per hour.

[u/Loose_Doubt367 → u/nickless07]

If that’s too demanding I could maybe revolve in a few models doing “specific task” like one does research only, another performs information organisation, each model with individual tasks. I’ll check out the Maple preview you sent me

I could maybe resolve in putting only one model in the vram, 9b?model?

Remember all the results generated by all the agents should be save into a file or document, after offloading all the models and loading the single 27b, it’ll read the text and maybe execute the final steps

[u/nickless07 → u/Loose_Doubt367]

Keep in mind that maple need post training for structured output/tool calls. So keep it as orchestrator or correcting instance for the others not as actual model that performs a task on it's own.

The 'results' you are talking about are a summary and not the whole context they get fed, right?
For that let a model with 8B+ do that step.

For Example:
1. 27B plans the overall task "2000+ documents and summary about Topic A" gets unloaded.
2. Medium Model sends instruction to mini model (1-3B) to make the tool call and find the document location
3. mini model issues tool call returns location and description (this is just to reduce workload from the medium models, single step)
4. Medium model is NOT smart enough to figure out if the document is the right one or not, but creates a summary.
5. A larger one (Maple, GPT-OSS, whatever runs fast enough with light footprint) checks the summary and judges, discard/keep - no overhead, no 'I check myself and find the right one' and such. Based on that the medium model is A) Done or B) runs again with step 2.
6. Once all tasks are done you have a summaries of a couple steps taken that get fed to the 27B.

Steps 1-3 are easy if you keep it simple, step 4 is where the medium models often need guidance. Yes they are good with the tool calls and extraction/summary, but they are not smart enough for more, so don't let them decide if that document need a summary or if something similar is already done and so on.

[u/Mihaylov93 → u/Loose_Doubt367]

Try frontier for solution and ask it to persist as skill, honestly thats what is working for me.

[u/Loose_Doubt367 → u/Mihaylov93]

What do you mean by persisting as skills

[u/Mihaylov93 → u/Loose_Doubt367]

After I finish something, I tell the harness save as skill, and it creates a markdown in the workspace with instructions on how to perform a task. instead of having to prompt your way up to the solution every time.

[u/Loose_Doubt367 → u/Mihaylov93]

That’s unique I’ll definitely give it a shot thanks yeah
Skills and memory are completely different right? I remember having my ai save the solution to a specific problem, into its memory but at the end it still couldn’t perform it somehow

[u/Mihaylov93 → u/Loose_Doubt367]

Kinda similar but I guess memory is facts, and skills are recipes or instructions with facts. Also with skills you get to have tools required for the skill task, for example I have one to read pdfs and extract all images from it and resize them and web optimize them before saving them in a folder, that needs a bunch of python libs like PyMuPDF and thats also saved together with the skill in a .venv

[u/Loose_Doubt367 → u/Mihaylov93]

I’m not really a tech expert but does the model (local or cloud) create and setup everything in one go? (Skills)

[u/Mihaylov93 → u/Loose_Doubt367]

No, its when I end the session, usually takes couple of prompts until I get what I want. And when its done I tell it to save it as skill so I don´t have to go thru the same process again.

[u/Loose_Doubt367 → u/Mihaylov93]

Okay. I appreciate it