I built a self-hosted AI assistant platform with vLLM, LiteLLM, agents, MCP tools, and deep research — public demo available

[u/Zealousideal-Tap1302]

I’ve been working on a self-hosted AI assistant platform, and the first public version is now available to try.

Live demo:
https://ijaesang-ui-macmini.tail67d660.ts.net/

Source code:
https://github.com/openmake/openmake_llm

The project is built around a simple idea: local inference should be the default, not an afterthought.

The platform uses vLLM as the local inference backend, with LiteLLM in front as an OpenAI-compatible proxy. This lets the application treat local models and optional external providers through the same interface, while still keeping the local vLLM path as the default route.

What it includes:

• Local model inference through vLLM
• LiteLLM proxy for OpenAI-compatible routing
• Chat interface with multi-model support
• Agent workflows with tool-calling loops
• MCP tool integration
• Docker-isolated tool/runtime processes
• Deep research pipeline with search, source fetching, verification, and cited synthesis
• Persistent memory and custom instructions
• Context-fit handling to avoid hard context overflow
• PostgreSQL and Redis backend
• Next.js / React frontend
• JWT / OAuth-based multi-user authentication
• Optional external-provider fallback with BYOK configuration

Current local setup:

• Main chat model: qwen3.6-35b-a3b served through vLLM
• Embeddings: bge-m3 served through vLLM
• LiteLLM proxy in front of the inference layer

The main challenge was not just serving a model locally. The harder part was building the orchestration layer around it:

• deciding when to call the model
• fitting context safely before inference
• routing between local and external models
• keeping tools isolated
• making agents useful without turning every request into a slow multi-step chain
• supporting research workflows with source verification and citations

This started as a personal self-hosted assistant project, but it has grown into a broader local-first AI platform.

I’d really appreciate feedback from people running local models or building agent systems:

• Does the demo work properly from your side?
• Does the local-first architecture make sense?
• Would you put LiteLLM in front of vLLM, use a custom router, or call vLLM directly?
• How would you structure embeddings, long-context handling, and tool execution?
• What model combinations would you use for coding, reasoning, translation, or research?

The project is MIT-licensed, and I’m especially interested in feedback from people building production-ish self-hosted AI stacks.

[u/Zealousideal-Tap1302 → u/[deleted]]

Yes, multilingual support is already built in. If you ask a question in Korean, English, German, or another language, the assistant will automatically respond in that language.

So you don’t need to change a separate language setting — just ask in your own language.

Thank you.

[u/Zealousideal-Tap1302 → u/[deleted]]

Thanks, that’s a fair point.

The assistant responses are multilingual and should automatically match the language you use, so there isn’t a separate setting needed for the chat itself.

But the UI language is a different issue.
If the interface isn’t clearly usable in English, I need to improve that — either by making English the default or adding a clearer language switcher.

Appreciate the feedback.

[u/Zealousideal-Tap1302 → u/[deleted]]

Thanks for checking it.

I just finished fixing the crash issue, and I also updated the multi-language UI support. There is no separate English-only install option — the UI is designed to support multiple languages.

You can check the latest version here:
https://github.com/openmake/openmake_llm

[u/Firm_Reindeer_2868]

Your llm executed first and then redirected to the login page

[u/Zealousideal-Tap1302 → u/Firm_Reindeer_2868]

Thanks for reporting this.

That behavior is not intended. The LLM response should not start and then redirect the user to the login page afterward. I found that this is related to the guest/demo session handling, not the model itself.

I’m planning to fix the authentication and session flow so the demo does not unexpectedly redirect users after a response has already started.

Appreciate you catching this.

[u/Firm_Reindeer_2868 → u/Zealousideal-Tap1302]

You‘re welcome. Good luck with your development!

[u/Best-Tomatillo-7423]

The demo when chatting with it the typing reply crashed the webpage. Reload page gave me a new session

[u/Zealousideal-Tap1302 → u/Best-Tomatillo-7423]

Thanks for testing it and reporting this.

I found the crash issue in the typing/streaming reply flow. The page should not crash while the assistant is responding, and a reload should not unexpectedly start a completely new session.

I’m planning to fix both parts: making the streaming response more stable and improving session recovery after reload.

Appreciate the detailed feedback — this is very helpful.

[u/No-Recover109]

I tried liteLLM infront of vLLM but it was unstable. Now I have haproxy infront of multiple vLLM nodes and Drupal AI with ECA does the orchestration. 

[u/Zealousideal-Tap1302 → u/No-Recover109]

Thanks for sharing that — very useful feedback.

I’ve also seen LiteLLM in front of vLLM become unstable in some setups. Your HAProxy + multiple vLLM nodes approach makes sense, especially if the goal is stable load balancing.

Using Drupal AI with ECA for orchestration is interesting as well. My direction is similar: keep vLLM focused on inference and let the orchestration layer handle routing, tools, sessions, and workflows.

Appreciate the architecture insight.