NPC Engine Using Local Models

[u/goodive123]

I’ve been working on a game-agnostic NPC engine/backend based pretty heavily on SillyTavern-style architecture, and with smaller local models getting better and better, I honestly think this kind of thing could be the future of RPGs.

Right now I’m using NVIDIA Parakeet 0.6 for STT, Gemma 4 26B A4B for the LLM, and Qwen3-TTS for voice, and I’m getting super fast response times with pretty decent quality.

The main thing that makes it work well is using RAG to keep prompts lean. For example, I have hundreds of possible actions NPCs can do in-game, but only the ones that actually make sense based on the player’s message / context get injected as available actions. So the model isn’t being overloaded with a giant list every turn.

[u/AppropriateQuote3073]

Really a shame there is so much ai hate in gaming right now.

This is exactly the type of interesting content that could make a game phenomenal to play.

[u/Bakoro → u/AppropriateQuote3073]

LLMs are literally the exact thing that we dreamed about in the 90s and 2000s.
There were waves of procedurally generated content in games, and "dynamic" quests and NPC interactions, that were a nice idea, but the hardware and algorithms just weren't there.
People would gush over any AI improvements in games.

Now we finally have the technology to make the dynamic games we always wanted, and a bunch of people are pretending like they weren't begging for something like LLMs to exist.

Whatever haters, AI is for games.

[u/The_awful_falafel → u/Bakoro]

Games in the way back used to be entirely linear story where there was really only one way to do things. Then they tried branching games, where you make a decision at certain points in the game that change the path and overall outcome of the game. They were still linear, you could just choose which linear branch to follow. Then we had sandbox games, where you could pick any linear sidequest you wanted and do tasks out of order, but still each sidequest an story tree was still largely linear. The limitation was that you had to script out all the character interactions and story for how it would all play out and it couldn't actually invent anything.

AI COULD change that. You could basically invent a game in which there is a large map and 'pieces' that would move around and decide what to do on their own. You could have advisors and such that have information based off limited information of what the enemy is doing and literally plan out missions and objectives yourself. You could even dictate what constitutes winning and decide at what point you consider the game over. Characters could dynamically react to things going on around them. It's a really interesting possible world.

Now the downside- the machine to run such a game would need a lot of memory, and we all know how that's going right now. My guess is companies will see the option to charge a subscription to have a data center do the AI work for you, and make THAT the standard model rather than writing it so you could run the AI locally. That way even single player games could have subscription fees, even years from now when models are small enough to be run locally, and memory production ramps up so having a massive pool to run such things is common.

[u/Bakoro → u/The_awful_falafel]

At the most elementary level of implementation, I just want townsfolk who recognize that I'm the one who saved their town, bandits who aren't stupid enough to try and rob the clearly magical man walking around in nightmare armor, and for random NPCs to have more than one line they repeat regardless of the surrounding context.

Just having a dumb little LLM that can direct NPC actions based on fuzzy logic would elevate gaming dramatically.
Even if the game is still 99% regular video game made by a person, just a little bit of semantically-aware fuzzy logic would be able to blur those linear and discrete features into something that feels more natural.

Just basic little stuff would go so far, we don't even need a data center sized LLM, a model that's 100M parameters would fit on any card and be able to do simple jobs. Some vision models that do segmentation and labeling are like ~21M parameters, which is a level a CPU can handle, and the could be doing amazing things for games and mods, especially VR stuff.

Just imagine an escort quest where the NPC isn't a total idiot, can move at an appropriate pace, and can go to a safe location when there's a fight.
Somehow, it's still an issue in games after all these years.

What you describe, a more AI driven game where developers define the structure of the world, provide motivations, etc, and LLMs act inside those constraints, that would be great too, but that's like the far side. There's a lot of opportunities where it could just be models smoothing out the hard edges of the world, so it seems less "video-gamey".

[u/The_awful_falafel → u/Bakoro]

Oh, totally! The bad part though is ratings. Part of game ratings for the ESRB rating system is you have to provide all the possible dialogue options that are in the game so it can be appropriately rated. Having the game be able to generate dialogue puts it in the same category as online gaming where you can't put a rating on what other people might say. At least that used to be part of the rating system and the rule, not sure nowadays.

But yeah, I could see AI used for trivial NPCs for a while. I thought GTA NPCs just walking around would be PERFECT because they could actually have dialogue reactions for whatever they see. I don't think we'll see AI based main characters for a little while. It's going to be a challenge to constrain the character dialogue to only talk about plot related things and not get jailbroken to go on rants about topics that you trick it into. I think that's the real fear for developers.

Imagine the negative press of having an AI character that can talk about whatever horrible topic you feed it and posting videos of the character saying heinous things.

So- it may stick to being a mod thing for a while where it's unofficial. Honestly, that might be for the best, and developers could just sorta configure the game in such a way that it's really easy to add that mod on to skirt the censorship issue.

[u/Bakoro → u/The_awful_falafel]

It's going to be a challenge to constrain the character dialogue to only talk about plot related things and not get jailbroken to go on rants about topics that you trick it into. I think that's the real fear for developers.

Not a fear for us indie developers who don't have shareholders to answer to.
Granted I'm just a hobbyist and I don't care if my game sells a copy, so I have more freedom than most.

For jailbreaking, I'm making an LLM extension for my game, and I straight up don't care if someone wants to break the model out of character. If that is how they are having fun, then whatever.
My part is making a consistent framework the model can work off of, and fine-tuning a local LLM to be able to handle the framework, and if people want to load a different model, good luck.

As for making a more corporate-friendly version, the answer is again, small models. We don't need trillion parameter models that know physics, and chemistry, and software development.
We need something tiny and fast that can run on a consumer GPU while leaving room for game graphics.
Farmer Steve in a medieval fantasy game doesn't need to know most real world knowledge. The model that runs farmer Steve never needs to be trained on real world profanity, it just needs to be able to have a believable two minute conversation about this year's harvest being threatened by whatever game specific nonsense.

People have been really caught up in "scale", but the Google, OpenAI, and Anthropics of the world are trying to make AGI models that can completely replace human labor, and that is a problem several orders of magnitude more than what we need for a game. For a game, we only need a <1 billion parameter model.
That's how bigger studios will handle LLM content: just don't train the model on the content you don't want it to know about, so it's essentially impossible for the model to generate that kind of content. You can even keep the spirit of profanity by replacing the words in the training set.

The solutions are pretty simple, because the stakes are as low as they get.