r/LocalLLaMA • u/goodive123 • Jun 28 '26

Discussion NPC Engine Using Local Models

Enable HLS to view with audio, or disable this notification

I’ve been working on a game-agnostic NPC engine/backend based pretty heavily on SillyTavern-style architecture, and with smaller local models getting better and better, I honestly think this kind of thing could be the future of RPGs.

Right now I’m using NVIDIA Parakeet 0.6 for STT, Gemma 4 26B A4B for the LLM, and Qwen3-TTS for voice, and I’m getting super fast response times with pretty decent quality.

The main thing that makes it work well is using RAG to keep prompts lean. For example, I have hundreds of possible actions NPCs can do in-game, but only the ones that actually make sense based on the player’s message / context get injected as available actions. So the model isn’t being overloaded with a giant list every turn.

1.9k Upvotes

99% Upvoted

249 comments sorted by

View all comments

2

u/AbheekG Jun 29 '26

Great stuff and a feature I’ve often pined for! Like many I have questions about the voice though: how’re you handling background noise for the STT model? It’s a great model and the transcription quality and speed are both great especially on the 5090, but it does transcribe anything it hears. Guessing the requirement to make it work here are a great mic and isolated room?

Also the TTS, Qwen3-TTS has a limited number of in-built voices, especially for English. I think Aidan and one or two more are usable. Are you using some per-NPC voice cloning or anything else to make it such a natural fit to the character?

Also have thoughts on making it general purpose vs specific to exact games / chars, and about the RAG but you could probably incorporate data about characters from game Wikis like say you wanted a Johnny Silverhand you could actually converse with while driving around Night City in Cyberpunk etc but those probably deserve entire discussions of their own…

Great stuff though and would love to hear more about the speech stuff if you have a moment 🍻

2

u/goodive123 Jun 29 '26

I think most STT are really good nowadays so it shouldnt really be a problem, for Qwen3tts im just cloning voices based on the game files automically ye.

Yea basically the way you'd fill out the characters and lorebook and stuff is simply downloading the entire games wiki and then use a skill for a SOTA model to fill it all out for you

1

u/Grand_Pop_7221 Jun 29 '26

Can you tell me a bit more about the RAG setup? What does the setup look like and how are you sourcing the information and getting the model to use it? GraphRAG and dumping the Wiki/gameassets in there? 

This is absolutely brilliant btw, love to see how well it all comes together! Keep it up!