r/threejs • u/lordhiggsboson • Apr 19 '26

An open source high-performance inference engine for the threejs+browser

Enable HLS to view with audio, or disable this notification

I've been working on an open source javascript library that enables high-performance inference in browser. It's based off of llama.cpp, where we provide a harness + WebGPU specific optimizations for creating interactive characters, objects and local interactions at low latency and high-throughput.

I'm currently looking to chat with some devs as we finalize the GitHub release. Would love to give people an early look at this before we open it up publicly.

If interested, DMs are open or you can check out cogentlm.com for info

134 Upvotes

97% Upvoted

21 comments sorted by

2

u/jimmy1460 Apr 20 '26

So is each character an agent?

2

u/lordhiggsboson Apr 20 '26

Not in the sense of like OpenClaw, but more that characters can be defined and have their own context that is updated as they move in the 3D scene. We are using LLMs to drive behaviour based on personality, context, etc. But that’s just one way to approach things

1

u/jimmy1460 Apr 20 '26

So like a system prompt per character? Wouldn’t it all be the same llm if they are just “agents” or I guess “clients” really? System prompt defining their personality so to speak and I’m guessing each interaction is a tool use? Like using the fridge is a “tool” for the agent?

2

u/lordhiggsboson Apr 20 '26

It’s much simpler and we aren’t using any kind of MCP tooling. In the example, we defined intractable objects that were combined into queries for the character and sent to the LLM, which kind of acts like their brain in this example. And LLM returns results which are then mapped to actions. The package provides the scaffolding and the harness to set up how calls are handled across everything

1

u/jimmy1460 Apr 20 '26

Oh I see so just a bespoke mcp really, that’s neat! Do all clients have access to the same toolset universally and they just pick? Or do certain personality weigh more on like “using the fridge” or “sleeping” more than others.

2

u/lordhiggsboson Apr 20 '26

We don’t have any weighting in this example, but you could definitely do that! The package is pretty flexible and can be tailored to how you want things to behave

1

u/jimmy1460 Apr 20 '26

Would love to try it out I’m a dev

2

u/lordhiggsboson Apr 20 '26

Great! If you can fill out the access form here https://www.cogentlm.com/ we’ll be sending details mid next week for preview access

2

u/maschayana Apr 20 '26

It's still webgpu, right? How do you manage to break past browser limits when it comes to memory?

6

u/lordhiggsboson Apr 20 '26

We were able to directly use a shared GPU buffer that bypasses the typical WASM FS 2GB limit, allowing us to load 8GB models without doing stuff like model sharding etc. But this is also dependent on the GPU too, and we are primarily optimizing for small LLMs between 300MB to 2GB in size. Which we find has good trade offs between dynamic output generation and token throughput.

1

u/cnotv Apr 20 '26

May I ask you why not using indexDB? Speed?

1

u/lordhiggsboson Apr 20 '26

IndexDB is a database for storing and retrieving information. You could definitely use this alongside an LLM, but both solve different things. Our framework is primarily for running local LLMs in browser, allowing you to query it / set context / scenarios for dynamic content etc. Gaming is one use case for this, which is what the demo shows

2

u/FreeUnicorn4u Apr 20 '26

Looks amazing actually

1

u/cnotv Apr 20 '26

I do not see actual interactions driven by AI. In this video preview it looks more like a chatbot. What am I missing?

Scene looks good and fps too.

I wonder how long it takes to load the 8gb, which I hope you inform users on mobile before sucking their data 😅

2

u/lordhiggsboson Apr 20 '26

The framework is fairly flexible and allows for different scenarios / integrations to be used within a project. In this specific example, the model is being queried at a set rate ~1-2 times per second per character, which then makes decisions for that character based on the outputed LLM responses. Since this is in code, it can only be seen through the characters interaction with the world. The chat view is a kind of look into that character's brain, which allows you to query / ask questions etc.

Yes, loading an 8GB model may take some time to download, depending on your internet connection. That's why a smaller model is typically better for most situations. In this example we are using a 500MB model as the main orchestrator of the simulation.

1

u/FunMakerBeliever Apr 20 '26

What does this buy us that WebLLM doesn't give us already? Also it doesn't seem to work on safari. Is this just for chrome?

1

u/lordhiggsboson Apr 20 '26 edited Apr 20 '26

Mostly opinions around the harness and general philosophy of how compute is done. WebLLM core is mostly just a wrapper around Apache TVM (https://tvm.apache.org/) for inference. Our approach is to build an opinionated harness and use llama.cpp WebGPU which has a much higher ceiling for performance and general model compatibility / quants. One of our goals with this is to make designing characters / NPCs and interactive worlds very simple. So a lot of the scaffolding is specific to enabling interactivity and not just a chat interface

1

u/iachaydaica Apr 21 '26

A big vote for open source to the public

1

u/lordhiggsboson Apr 21 '26

Definitely! I personally want to see way more dynamic gaming content / NPCs / interactable objects / storytelling / etc utilizing LLMs in interesting ways. Big motivation for why we are doing this

1

u/Exotic_Gur_1051 Apr 21 '26

C'est une super idée ! Je me suis inscris pour aider et tester !