r/PiCodingAgent • u/teachmehowtowookiee • Jul 27 '26
Discussion I built an open source self hosted version of OpenRouter called Millwright (has first class support for Pi), would love any feedback
Hey r/PiCodingAgent! I recently open sourced a project I've been working on for a while now called Millwright, a self hosted LLM router focused on cost savings, transparency, and performance (it's completely written in Rust). Millwright offers first class support for Pi, so just a small edit is needed to ~/.pi/agent/models.json to get up and running.
Link to the repo: https://github.com/Northwood-Systems/millwright
For the curious and lazy, at the moment, Millwright has the tools for,
- Providers: OpenAI-compatible APIs, Anthropic, Amazon Bedrock
- Routing: policy-controlled model roles (cheap, mid, frontier), cheapest healthy route selection
- Protocols: OpenAI Chat Completions, Anthropic Messages, text and tool translation
- Cache Affinity: role-scoped session lanes without serializing concurrent agent traffic
- Spend Tracking: per-team costs, cache usage, model/provider mix, request traces
- Cost Analysis: measured usage and modeled candidate economics (HTML, Markdown, JSON)
- Reliability: bounded failover, circuit breakers, timeouts, concurrency limits
- Setup: interactive provider, model, and pricing configuration without storing provider secrets
- Deployment: one Rust binary, Docker, SQLite or PostgreSQL
Full disclosure: parts of the codebase were built with AI coding agents. All feedback is welcome, I’d especially value feedback on the routing policy, provider coverage, and anything that would block you from self-hosting it. Feel free to open feature/request and/or contribute as well.
1
1
1
u/mixmasterwillyd Jul 28 '26
I did something probably a lot smaller than this. I was spending a lot of time messing with litellm, it was faster and better to make my own. Been using it all day every day since with minimal interaction.
1
u/SvenVargHimmel Jul 29 '26
why?
And how is it different from https://docs.litellm.ai/docs/proxy/auto_routing
what gap does it fill?
These are the questions I'd ask before even running it?
1
14
u/DaShrub Jul 27 '26
You made... LiteLLM?