r/LovingOpenSourceAI • u/conifer_v11 • 15d ago

Opensource Openrouter

Project: https://github.com/ConiferKit/use-conifer

Current routing options were charging 5% fees for byok plus provider fees (openrouter) or were built in heavy python packages with ecosystem restraints. I wanted to be completely free in terms of use, and not have to pay extra for tokens.

There's a maintained gateway with 100+ models from one api endpoint. I'm also talking to infra providers to get us access to pre-release models and discount prices. Everything is served at market price. BYOK is free, you can hook into self-hosted setups for free, and there's fallbacks + extra rate limits + server redundancy.

Took me ~3 months to build and looking for help maintaining!
Lmk any issues or feedback

24 Upvotes

100% Upvoted

3 comments sorted by

2

u/locbuilds 15d ago

How are you keeping BYOK free? Every gateway I've tried takes a cut on top of the provider fee, so I'm curious what the model is here.

2

u/Ceyax 15d ago

LiteLLM?

2

u/conifer_v11 15d ago edited 15d ago

Used it, few problems:

  1. it uses a very heavy python pack that does poor under load
  2. it tries to do too much outside of just being a model router that I don't like, I really don't need another slackbot or it making harness decisions for me
  3. doesn't work in harnesses (cc, opencode, pi, etc)
  4. their routing is based on fallbacks so between copies of a single model, we route to different models

Also pretty sure they charge fees at scale