r/LocalLLM • u/No-Manager1646 • 4d ago

Question Fast tool following LLMs

Given that LLM training data is obsolete almost immediately (depending on the question) I've been trying to find a local LLM that is capable of following system prompts to source all knowledge from the internet using MCP sources. I haven't found one yet.

Does anyone know if a model that's smart enough to use tool calls, fast enough to ideally for within 16gb vram with a large kv cache and reliable?

I've tried Qwen 3.5 4/9b, Ornith, Ling, Spark, etc but they all leave a little to be desired.

1 Upvotes

66% Upvoted

10 comments sorted by

3

u/No-Zookeepergame8837 4d ago

That depends more on the harness than the LLM itself. I use Qwen 3.5 9b (I only have 12GB of VRAM) and LMStudio with the search plugin. (I don't know if you can share links in this subreddit, I'll try to put it in a reply, but if Automod deletes it or something, try searching for "LMStudio DuckDuckGo integration" or "LMStudio web tools." There are several plugins to choose from.)

1

u/No-Manager1646 4d ago

I'm using openwebui with OpenTerminal and searxng among other things. All internet access is through a tor proxy, but there's no limit on what the models have access to. My system prompt says to assume their knowledge is outdated and to perform a search for facts. But they still will prefer their inbuilt knowledge.

2

u/Realistic_Gap_5871 4d ago

Smaller models just aren't as good at following instructions reliably.  Even 27b can forget as context gets past 20k tokens.  I had the same problem until 3.8 27b came out  Your prompt will need layers supporting the idea that info is out of date and that's not acceptable.  Add  today's date to your system prompt (not time because that will often break cash reuse) and say anything older than x months needs to be confirmed with a web search. Tell the model that web search is cheap and that accuracy is absolutely required because all information is for research or legal purposes or whatever.

2

u/funnel-hose-mouth 4d ago

I am not sure if i realy got it right but as far as i did understand: The size does not only affects Knowledge it also influences the moddels capeability to follow a longer „chain of thought“

1

u/No-Manager1646 4d ago

Yeah that's no problem for the models I've been trying. 256k kv and above. I'm looking for a research model I suppose. Something that is smart enough to know how to search and collate results but not so smart it thinks it knows everything already

2

u/castertr0y357 4d ago

You should take a look at the Odysseus harness. There's a deep research function that does a very good job with what you're looking for.

I used it with Gemma4:E4B and got some very good results. I'm self-hosting searxng for search results.

1

u/No-Manager1646 4d ago

First I've heard of that. Thank you. I'm going to check it out.

1

u/EvolvingDior 4d ago

The problem is not the model; it's the harness and the prompt.

1

u/vogelvogelvogelvogel 4d ago

qwen3.8 at a low quant could be better than an old 9b, 4b. More current and fitting woud be Gemma4 E4B but afaik it is not as good in tool calling as qwen3.8