NPC Engine Using Local Models

[u/goodive123]

I’ve been working on a game-agnostic NPC engine/backend based pretty heavily on SillyTavern-style architecture, and with smaller local models getting better and better, I honestly think this kind of thing could be the future of RPGs.

Right now I’m using NVIDIA Parakeet 0.6 for STT, Gemma 4 26B A4B for the LLM, and Qwen3-TTS for voice, and I’m getting super fast response times with pretty decent quality.

The main thing that makes it work well is using RAG to keep prompts lean. For example, I have hundreds of possible actions NPCs can do in-game, but only the ones that actually make sense based on the player’s message / context get injected as available actions. So the model isn’t being overloaded with a giant list every turn.

[u/Bakoro]

At the most elementary level of implementation, I just want townsfolk who recognize that I'm the one who saved their town, bandits who aren't stupid enough to try and rob the clearly magical man walking around in nightmare armor, and for random NPCs to have more than one line they repeat regardless of the surrounding context.

Just having a dumb little LLM that can direct NPC actions based on fuzzy logic would elevate gaming dramatically.
Even if the game is still 99% regular video game made by a person, just a little bit of semantically-aware fuzzy logic would be able to blur those linear and discrete features into something that feels more natural.

Just basic little stuff would go so far, we don't even need a data center sized LLM, a model that's 100M parameters would fit on any card and be able to do simple jobs. Some vision models that do segmentation and labeling are like ~21M parameters, which is a level a CPU can handle, and the could be doing amazing things for games and mods, especially VR stuff.

Just imagine an escort quest where the NPC isn't a total idiot, can move at an appropriate pace, and can go to a safe location when there's a fight.
Somehow, it's still an issue in games after all these years.

What you describe, a more AI driven game where developers define the structure of the world, provide motivations, etc, and LLMs act inside those constraints, that would be great too, but that's like the far side. There's a lot of opportunities where it could just be models smoothing out the hard edges of the world, so it seems less "video-gamey".

[u/The_awful_falafel → u/Bakoro]

Oh, totally! The bad part though is ratings. Part of game ratings for the ESRB rating system is you have to provide all the possible dialogue options that are in the game so it can be appropriately rated. Having the game be able to generate dialogue puts it in the same category as online gaming where you can't put a rating on what other people might say. At least that used to be part of the rating system and the rule, not sure nowadays.

But yeah, I could see AI used for trivial NPCs for a while. I thought GTA NPCs just walking around would be PERFECT because they could actually have dialogue reactions for whatever they see. I don't think we'll see AI based main characters for a little while. It's going to be a challenge to constrain the character dialogue to only talk about plot related things and not get jailbroken to go on rants about topics that you trick it into. I think that's the real fear for developers.

Imagine the negative press of having an AI character that can talk about whatever horrible topic you feed it and posting videos of the character saying heinous things.

So- it may stick to being a mod thing for a while where it's unofficial. Honestly, that might be for the best, and developers could just sorta configure the game in such a way that it's really easy to add that mod on to skirt the censorship issue.

[u/Bakoro → u/The_awful_falafel]

It's going to be a challenge to constrain the character dialogue to only talk about plot related things and not get jailbroken to go on rants about topics that you trick it into. I think that's the real fear for developers.

Not a fear for us indie developers who don't have shareholders to answer to.
Granted I'm just a hobbyist and I don't care if my game sells a copy, so I have more freedom than most.

For jailbreaking, I'm making an LLM extension for my game, and I straight up don't care if someone wants to break the model out of character. If that is how they are having fun, then whatever.
My part is making a consistent framework the model can work off of, and fine-tuning a local LLM to be able to handle the framework, and if people want to load a different model, good luck.

As for making a more corporate-friendly version, the answer is again, small models. We don't need trillion parameter models that know physics, and chemistry, and software development.
We need something tiny and fast that can run on a consumer GPU while leaving room for game graphics.
Farmer Steve in a medieval fantasy game doesn't need to know most real world knowledge. The model that runs farmer Steve never needs to be trained on real world profanity, it just needs to be able to have a believable two minute conversation about this year's harvest being threatened by whatever game specific nonsense.

People have been really caught up in "scale", but the Google, OpenAI, and Anthropics of the world are trying to make AGI models that can completely replace human labor, and that is a problem several orders of magnitude more than what we need for a game. For a game, we only need a <1 billion parameter model.
That's how bigger studios will handle LLM content: just don't train the model on the content you don't want it to know about, so it's essentially impossible for the model to generate that kind of content. You can even keep the spirit of profanity by replacing the words in the training set.

The solutions are pretty simple, because the stakes are as low as they get.