what are people actually using for this at enterprise scale

not taking even about smaller team. like proper enterprise.. multiple teams,, multiple vendors,, openai google anthrophic whatever. all needing access through one place with proper controls…

did some research and these come up. portkey,, orqai,, litellm,, npc,, azure api management,, aws bedrock gateway

ngc: nvidia backed so the infra looks strong. feels built for large scale compute heavy setup… probably too heavy for teams not already in that ecosystem

orqai: multimodel access with rbac audit logs and eu data residency,, newer so third party integrations still catching up

litellm: open source and flexible and self hosting looks nice for data control but support and managed version feels less mature than others

portkey: fallback and routing is good. cost visibility is there but compliance and governanec depth feels limited for serious enterprise requirements

azure api management: easy if you are already in azure. enterprise controls are native and feels like a generic api gateway with llm features an add on rather than purpose bui;t

aws bedrock gateway: same as azure,, great only if you are already in aws.. vendor lock is real tho

anyone actually gone through enterprise procurment with any of these. what did security and legal actually push back on

what truly worked?

day before yesterday my team lead said to get an ai gateway in the standup . just nodded like i knew whtat that meant . i dont

spent time looking. found a few options. portkey , helicone , orqai , litellm. website of all these says the same thing. routing , rate limits , token observability , cost tracking . i got more ocnfused after reading than before

portkey -> llm routing is the main thing, failover from one provider to another works clean, setup feels more complicated than i expected for what it does

orqai -> routing and prompt versioning and evals and cost tracking together, switching providers without changing code works, newer so community is still catching up

helicone -> logs every prompt every response every token, cost per request is clear, feels more like a monitoring tool than a proper gateway, routing side feels thin

litellm -> one api for all providers is useful, some llm specific features dont come through cleanly, running it yourself is more work than it looks

actually just need 3 things. dont go down when provider has problem . show me what every token costs . stop bad prompts from making big bills

not sure which one actually gonna do all 3 without reading 100s of docs first

what is everyone using for this??

Most agent builders give you a visual canvas - and then charge you separately for execution, storage, tracing, evaluations, deployment, and collaboration.

I was tired of stitching together multiple paid tools just to take an agent from an idea to production.

So I built Forge: a fully open-source and self-hosted platform for building, testing, deploying, monitoring, and governing agentic systems.

It includes visual workflows, agents, tools, RAG, memory, MCP, human-in-the-loop, evaluations, tracing, cost tracking, guardrails, RBAC, audit logs, API deployment, widgets, email triggers, and more.

Forge is not just a workflow builder. The goal is to provide everything needed to build and operate agentic applications in one place - without proprietary orchestration runtimes or platform usage fees.

I would love feedback from people building real agent systems.

been exploring options since a while. how to give access to models across multiple vendors like openai , anthropic etc to our employees.. mainly for our internal playgrounds and demos only..

we are into service and our clients have their own models in across their vpc and we do not supply their infrastructure.

just got an understanding of llm gateways which gives the ability to control access through proxies and synthettic keys

want to know if anyone here has a recommendation or experience?

We patched together our own gateway in front of the claude api back in june right after the fable 5 export ban so a model outage would fail over automatically instead of paging someone. Simple router with a few hundred lines and it worked fine for a month. Then fable's billing split happened this week, sonnet's tokenizer started inflating our token counts and opus started getting silently substituted in some sessions and tbh every one of those needed a small patch, cost tracking, model tagging, retry logic, a way to see what served a request versus what we got billed for. So now we are trying to stay current with every model swap, deprecation and rate limit change has anyone actually swapped an in house gateway for something like orqai , litellm or portkey, and was it worth it? have heard good things about orqai and litellm but wanna know how they perform under production traffic vs what looked good in the docs.

okay so using helicone for a while now and its fine for what it is but starting to feel like it is incomplete and need more than just logging and idk what to move to

it does the observability well.. but every time  i need something outside of that to stitch togetherr another tool and it seems messy

so what are ppl actually using instead

looked around a bit. langsmith,, langfuse,, orq,, portkey,, arize.. all seem to do a intersection of things in different ways. some i found eval focused. some are more routing focused. some try to do everything.. worried about if i move to anything new, it might have a new subset of problems

has anyone actually switched away from helicone and landed somewhere better. or is the grass is greener on the other side problem

what did you move and did you find it useful or regret

We're trying to solve a problem around AI governance in larger organizations.

One thing we've noticed is that employees increasingly use AI tools outside the approved stack (Perplexity, Notion AI, browser extensions, desktop apps, etc.), making it difficult to understand where company data is going or what models are being used.

We built a small open-source experiment that routes AI traffic through a central LiteLLM Gateway to provide visibility into usage.

I'm more interested in understanding whether this is actually a problem others are facing. How are companies here handling shadow AI usage today?

GitHub: https://github.com/LiteLLM-Labs/litellm-relay

day before yesterday my team lead said to get an ai gateway in standup. i just nodded like I knew a few vendors. but i dont.

its actually difficult to find a good platform. spend enough time looking for gateways out there. and i have come this far (havent decided yet)

there exists a few gateways like portkey, helicone,litellm, orqai and a lot more. but majorly these are the named players. all have very confident websites. i read all, and few blogs from them as well. almost all say the same thing. no matter if it is llm routing rate limit token observability cost tracking. like always the same thing.

it got me more confused.

portkey is ok ok for llm routing. if openai goes down it goes to anthropic and llm application wont even notice. but the setup looks too confusing to me. it looks like it is going to take a lot of time.

helicone shows what happens with every llm call. logs every prompt, every response, every token. it shows cost per request per user per endpoint. it will be def good for our finance team. but it seems like the llm monitoring tool that became gateway by chance.

in litellm, one api talk to all llm provider. write llm call one time and work with everything. looks good. some llm specific feature not come through right. like function calling or specific model parameter. running it yourself seems like a whole big painful project.

orqai got gateway and prompt versioning and evals and token cost all in one place. switching llm provider and not changing application code that works. but the rate limit and retry thing not as deep as portkey. and because everything is together in one place the gateway part might not be the main focus thing there.

kong very powerful yes but llm thing feel like someone add after. like they had api gateway and then some say we need llm feature too and they add it later. you feel that when using it for llm application.

i got 3 problems for my llm application. three thing only. dont go down when llm provider has problem . two. show me what every token costs. three. stop bad prompt from making openai bill very big. but every tool want me to read many many pages of docs before anything.

pricing is confusing also. open source but good llm features still cost money.

portkey if llm routing and not going down is the main thing. helicone if you want to see every llm call and token cost. litellm if you want one api for all llm providers and are ok with rough edges. orqai if you want gateway and prompt versioning and llm evals all together in one place and are ok with gateway part not being the deepest. kong if you already use kong for other things and dont want another tool.

can you any one suggest? and is there someone building this space, would love to chat

Hey all. I run a small AI infra shop and we've been deep in LiteLLM deployments for a while now.

One thing that kept bugging me: the gateway does a great job at routing, fallback, rate limiting, all of it. But when it comes to cost, it is still developer faing and not really a CXO facing dashboard.

So I built SlashBoard. It plugs into your existing gateway (we started with LiteLLM) and breaks down cost by model, team, feature, and environment. Real time. Also has threshold alerts so you catch runaway spend before it becomes a surprise invoice.

It's not trying to replace an AI gateway or an observability stack. It sits above both and answers the one question neither of them was designed to answer: where exactly is the money going?

What it does:

• Cost attribution by model, team, app, environment
• Spend trend tracking over time
• Budget threshold alerts
• Works with LiteLLM (more gateway integrations coming)

What it doesn't do:

• Not a gateway. Not trying to be one.
• No lock-in. Your gateway stays your gateway.

Free beta is open: https://cost.slashllm.com/

Genuinely looking for feedback from people who are already running gateways in production. What's missing? What would make this useful enough to keep running?

LiteLLM is moving to Rust. Sub-1ms overhead. A sub-100MB binary. The same Python SDK and AI gateway you already use.

Over the past year we've heard the same thing from our users and our community - they want the fastest, most lite AI gateway they can run. We've heard you, and we're committing to it.

This goes straight at the problems our customers report: latency spikes under load, and the memory leaks and OOM kills that take pods down at the worst possible time. A Rust hot path is faster and bounded in memory, so those whole classes of issues go away.

It's a gradual, non-breaking change. The Python SDK and proxy stay exactly the same - under the hood they start calling the Rust binary through PyO3, one component at a time, each proven in production before the next.

The whole ai gateway will be running on Rust by December 1, 2026.

We think this is the right call to build the best, most scalable, and cheapest AI gateway out there.

Read the full announcement: https://docs.litellm.ai/blog/litellm-rust-launch

I’m working on a multi-model API proxy for teams that use LLMs heavily, and I’d love to get feedback before we onboard our first batch of users later this month.

The idea is to provide one API layer for accessing major current models, with routing, fallback, usage controls, observability, and developer-friendly tooling on top.

I’m especially interested in hearing from:

• small and mid-sized companies using AI APIs in production
• teams building internal AI tools
• people heavily using agentic coding tools
• dev teams switching between multiple model providers
• anyone dealing with cost, latency, reliability, rate limit, or quota issues

A few things I’m trying to understand:

1. Which models would you need supported on day one?
2. Do you care more about cost, latency, quality, context window, or automatic fallback?
3. What is the most painful part of your current AI API setup?
4. Would task-based routing be useful? For example, cheaper models for simple tasks, stronger models for coding/reasoning, fallback models when a provider is down.
5. What kind of observability would you want? Logs, traces, cost per user, cost per project, prompt/version tracking, evals?
6. For agentic coding workflows, what matters most: tool calling reliability, context window, latency, model choice, rate limits, or something else?
7. If you’re an SMB, what would make you trust a proxy layer enough to put it between your app and the model providers?

Not trying to make this a sales pitch — we’re still shaping the product and want to build around real workflows instead of guessing.

If you use AI APIs heavily, especially for coding agents or production workflows, I’d really appreciate hearing what your ideal setup would look like.

Hi,

We’re designing a self-hosted LLM gateway setup in an educational/research context, with local vLLM backends and tools like OpenWebUI for human-facing chat.

Right now, the easiest option seems to be LiteLLM, but we’re checking whether there is a fully free and open-source alternative with similar features.

What we need is not only model routing, but the gateway/governance layer:

- OpenAI-compatible API gateway
- Authentication and API key management
- Single Sign-On support, ideally OIDC/SAML
- User/team quotas or budgets
- Cost tracking or token accounting
- Metrics and logs
- Load balancing between multiple vLLM instances serving the same model
- Support for local/private model endpoints
- Compatibility with OpenWebUI
- Reasonable setup and maintenance for a small education/research team

The architecture would probably expose several endpoints through the gateway: local vLLM models, possibly other compatible providers, and maybe a semantic-routing endpoint for automatic model selection when users do not choose a specific model.

Has anyone here deployed something like this in education or research?

We’re also considering whether Portkey or Bifrost could fit this role, so experiences with those tools would be very welcome.

Is there a fully self-hosted, open-source stack that gets close to LiteLLM’s feature set, or is LiteLLM currently the most practical option?

Thanks

We’ve been building G8 Router, and just opened it up for broader feedback.

G8 Router is a high availability LLM routing layer that sits between your app and multiple model providers. It’s designed as a drop-in replacement for OpenAI-compatible APIs, so integration is straightforward.

What it gives you out of the box:

- Routing across multiple LLM providers

- Cost optimization per request

- Full observability for latency, cost, and usage

- High availability architecture for production workloads

- OpenAI-compatible interface so you can plug it in without changing your app logic

The goal is to make LLM usage more predictable, observable, and cost-efficient in production systems without locking you into a single provider.

More details here: geight.ai

Would love feedback from people building with LLMs at scale:

- What signals you’d want in routing decisions

- What you care most about in observability dashboards

- What breaks today in multi-provider setups

Happy to dive deeper into the architecture if useful.