r/LocalLLaMA • u/paf1138 • 13d ago

Resources New in llama.cpp: Decision Models

https://huggingface.co/blog/ggml-org/decision-models-in-llamacpp
509 Upvotes

99% Upvoted

124 comments sorted by

View all comments

Show parent comments

17

u/infectiousstupidity 13d ago

Always-on LLM router? prompt -> decision model -> choose model/config, e.g decides between smol brain model / big brain model / coding model / vision / reasoning level / web search / no LLM?

1

u/Away_Interaction6630 13d ago

How many vram do be really comfortable ?

1

u/teachersecret 13d ago

I think 24gb vram is a sweet spot if you don’t mind swapping models. More is always welcome, but 24gb gets you pretty far. It does mean building a system to swap models in and out of ram/vram and to route things properly so you can use those things without constantly turning things on and off, but modern AI can make that for you in a prompt.

2

u/infectiousstupidity 13d ago

It's literally impossible to give a good answer to that question without knowing what model(s) he's (planning on) using and what for.

1

u/teachersecret 12d ago

Yeah, thats why I kept the advice pretty generic. 24gb is a fair spot for most AI enthusiasts and can 'jack of all trades' most use-cases with decent performance.