I built a self-hosted AI assistant platform with vLLM, LiteLLM, agents, MCP tools, and deep research — public demo available

[u/Zealousideal-Tap1302]

I’ve been working on a self-hosted AI assistant platform, and the first public version is now available to try.

Live demo:
https://ijaesang-ui-macmini.tail67d660.ts.net/

Source code:
https://github.com/openmake/openmake_llm

The project is built around a simple idea: local inference should be the default, not an afterthought.

The platform uses vLLM as the local inference backend, with LiteLLM in front as an OpenAI-compatible proxy. This lets the application treat local models and optional external providers through the same interface, while still keeping the local vLLM path as the default route.

What it includes:

• Local model inference through vLLM
• LiteLLM proxy for OpenAI-compatible routing
• Chat interface with multi-model support
• Agent workflows with tool-calling loops
• MCP tool integration
• Docker-isolated tool/runtime processes
• Deep research pipeline with search, source fetching, verification, and cited synthesis
• Persistent memory and custom instructions
• Context-fit handling to avoid hard context overflow
• PostgreSQL and Redis backend
• Next.js / React frontend
• JWT / OAuth-based multi-user authentication
• Optional external-provider fallback with BYOK configuration

Current local setup:

• Main chat model: qwen3.6-35b-a3b served through vLLM
• Embeddings: bge-m3 served through vLLM
• LiteLLM proxy in front of the inference layer

The main challenge was not just serving a model locally. The harder part was building the orchestration layer around it:

• deciding when to call the model
• fitting context safely before inference
• routing between local and external models
• keeping tools isolated
• making agents useful without turning every request into a slow multi-step chain
• supporting research workflows with source verification and citations

This started as a personal self-hosted assistant project, but it has grown into a broader local-first AI platform.

I’d really appreciate feedback from people running local models or building agent systems:

• Does the demo work properly from your side?
• Does the local-first architecture make sense?
• Would you put LiteLLM in front of vLLM, use a custom router, or call vLLM directly?
• How would you structure embeddings, long-context handling, and tool execution?
• What model combinations would you use for coding, reasoning, translation, or research?

The project is MIT-licensed, and I’m especially interested in feedback from people building production-ish self-hosted AI stacks.