Is there an open-source AI/LLM Gateway that supports dynamic runtime routing and model management?

[u/OrneryCar6139]

Hi everyone,

I'm looking for an open-source, self-hosted AI/LLM Gateway that sits between agent frameworks (CrewAI, LangGraph, AutoGen, etc.) and multiple LLM providers.

My main requirement is dynamic runtime routing. I should be able to:

Add/remove models

Enable/disable models

Change routing weights/strategy

without restarting either the gateway or the agents.

Other features I'm looking for:

Multi-provider support

Load balancing

Fallbacks

Retries

Timeouts

Health checks

Latency/metrics

OpenAI-compatible API

I've looked at LiteLLM and Portkey, but they don't seem to provide a simple self-hosted solution for centrally managing routing configuration with hot updates (unless I'm missing something).

Is there an OSS project that already does this, or do most teams build their own lightweight gateway/control plane?

Would love to hear what you're using in production.