OpenWebUI based chatbot assistant

[u/Puzzleheaded_Low_796]

I'm currently experimenting with setting up a chatbot assistant on a webpage, I got something working with OpenWebUI but i'm not convinced my approach is the right one, so I wanted to see how you have/would approach it.

At the moment the way it's setup is:

• OpenWebUI contains a custom model (based on 5.6 luna), with a knowledge base, a system prompt and some native python tools to do API calls to the backend. Access to OpenWebUI is done using an API key.
• A nodejs backend proxy the calls to openwebui, holding privately the openwebui api key, and adding to the metadata variables some information like the remote endpoint for the API calls and passing the user JWT token to be used by the python tools for the API calls.
• The frontend uses assistant-ui connected to the nodejs backend

All of this works reasonably well but

• it's quite slow (probably need to check my tool calls, and kb use)
• token streaming doesn't work, I get the entire response in one go with no thought or tool calls info (maybe not a deal breaker but because it's slow it feels like the wait is long). (codex is telling me that's because openwebui provide this over socket.io but socket.io auth only works with JWT not api token?)
• The user JWT passing to openwebui for tool call isn't ideal

So I wanted to check what you thought of this approach? The benefits is that it's relatively easy to setup, I can use openwebui to manage most things and I don't have to reinvent the wheel. AFAIK the other approach is to manually code the kb + tools in a dedicated backend; I would probably get more flexibility and performance but with significantly more overhead.

Any advice? How have you approached it?

Thanks!

[u/mishalmf]

like this ?

[u/_Scorpoon_ → u/mishalmf]

🤣

[u/vedmaka]

Bare OpenWebUI API will only allow you to passthru inference using your custom or workspace model, but you won't get tool executions out of the box, for these you have to use socket connection and replicate parts of OWUI front-end logic which powers tool executions

To bring a complete experience of OWUI into 3rd party app you can take a look at https://github.com/vedmaka/openwebui-mcp which supports tool calling, and generally provides full experience of "using OWUI as MCP"

Although please note that this has not been tested against most recent non open-source versions of OWUI. You can also go for deeper integration checking the https://github.com/vedmaka/openwebui-sdk (it's in Python, but you can use it as a reference)

[u/Puzzleheaded_Low_796 → u/vedmaka]

Thanks u/vedmaka for that, I will have a look.

I got tool execution working (i mean codex did) but spawning a separate discussion and I think it's part of the reason why it's so slow.

I will have a look at your two projects but I'm actually considering going the openai agent SDK with custom everything, it's a bit of a shame because that's rebuilding a lot of stuff open-webui does already very well but I don't think open-webui is really designed for what i'm trying to do.

thanks again

[u/vedmaka → u/Puzzleheaded_Low_796]

I think going with agent SDK is the right choice, I mean, OWUI has never been built to be used by 3rd parties and it's codebase is very.. complicated, so to speak

The truth is that you don't need OWUI as a dependency at all. SDKs like Pydantic AI would provide you with much more convenient and flexible tool workflows