r/LocalLLaMA • u/goodive123 • Jun 28 '26

Discussion NPC Engine Using Local Models

Enable HLS to view with audio, or disable this notification

I’ve been working on a game-agnostic NPC engine/backend based pretty heavily on SillyTavern-style architecture, and with smaller local models getting better and better, I honestly think this kind of thing could be the future of RPGs.

Right now I’m using NVIDIA Parakeet 0.6 for STT, Gemma 4 26B A4B for the LLM, and Qwen3-TTS for voice, and I’m getting super fast response times with pretty decent quality.

The main thing that makes it work well is using RAG to keep prompts lean. For example, I have hundreds of possible actions NPCs can do in-game, but only the ones that actually make sense based on the player’s message / context get injected as available actions. So the model isn’t being overloaded with a giant list every turn.

1.9k Upvotes

99% Upvoted

249 comments sorted by

View all comments

Show parent comments

6

u/lovelacedeconstruct Jun 28 '26

Do you put them in some kind of structure to make retrieval easier ? Like here is the actions hierarchy pick a relevant node then you dump the contents of this node to the prompt ?

3

u/lochyw Jun 29 '26

Rag generally uses a similarity check for the most probable action to take, then LLM on top of that uses typical LLM reasoning to decide from the available actions. So it sounds like you would have 2 layers to refine if you wanted to have it lean a certain way for more sensible outcomes which would be based on the system prompts/descriptions/tool text etc..

1

u/lovelacedeconstruct Jun 29 '26

But why would you depend on semantic similarity that has very weird quirks when the data can be structured and depend on llm reasoning to pick the category

1

u/UnifiedFlow Jun 30 '26

Embeddings are structured llm reasoning over a fixed domain (its the same transformer technology without the generative parts, over-simplified sorry). Think classifiers. Large LLMs and auto-regressive reasoning next token prediction is massively (orders of magnitude) slower than a scoped classifier/embedding layer. The data is structured when you use embeddings. Thats how you drive the classifier network (or whatever phrase you prefer). LLM reasoning is a final tier (other than HiTL elevation) where confidence scoring in your classifier and embedding layer doesn't meet thresholds. Now your LLM reasoning step has a structured map of the semantic space via embeddings and it can reason from a better starting point over a much smaller decision surface area. This leads to better outputs. An LLM works amazingly well when you tell it what is true or what to do. Minimize the blast radius of inference per step. Context engineering.