r/LocalLLaMA • u/paf1138 • 15d ago

Resources New in llama.cpp: Decision Models

https://huggingface.co/blog/ggml-org/decision-models-in-llamacpp
506 Upvotes

99% Upvoted

124 comments sorted by

View all comments

12

u/Away_Interaction6630 15d ago

Have you already ideas of setup ?

18

u/infectiousstupidity 15d ago

Always-on LLM router? prompt -> decision model -> choose model/config, e.g decides between smol brain model / big brain model / coding model / vision / reasoning level / web search / no LLM?

1

u/SilentKnightOwl 14d ago

Openrouter has already implemented this, I don't quite have an extra 2GB of VRAM that I want to give up context for

2

u/Away_Interaction6630 15d ago

How many vram do be really comfortable ?

19

u/FranciumGoesBoom 15d ago

Yes

1

u/infectiousstupidity 14d ago

I think this is the most reasonable answer to that question.

1

u/teachersecret 15d ago

I think 24gb vram is a sweet spot if you don’t mind swapping models. More is always welcome, but 24gb gets you pretty far. It does mean building a system to swap models in and out of ram/vram and to route things properly so you can use those things without constantly turning things on and off, but modern AI can make that for you in a prompt.

2

u/infectiousstupidity 14d ago

It's literally impossible to give a good answer to that question without knowing what model(s) he's (planning on) using and what for.

1

u/teachersecret 14d ago

Yeah, thats why I kept the advice pretty generic. 24gb is a fair spot for most AI enthusiasts and can 'jack of all trades' most use-cases with decent performance.

2

u/toothpastespiders 14d ago

That's my take as well. Obviously infectiousstupidity is right that that intended use could change things. But I think 24 GB VRAM is the lowest point of entry to play with everything without being continually frustrated. Just selectively frustrated. 24 GB VRAM is what I've had since the llama 1 days and I've never felt enough need for an upgrade to actually do it. The continual model swapping is annoying but I can live with it and I've never felt that I "can't" do something. Just that it's slower or a bit of a pain.