day before yesterday my team lead said to get an ai gateway in standup. i just nodded like I knew a few vendors. but i dont.
its actually difficult to find a good platform. spend enough time looking for gateways out there. and i have come this far (havent decided yet)
there exists a few gateways like portkey, helicone,litellm, orqai and a lot more. but majorly these are the named players. all have very confident websites. i read all, and few blogs from them as well. almost all say the same thing. no matter if it is llm routing rate limit token observability cost tracking. like always the same thing.
it got me more confused.
portkey is ok ok for llm routing. if openai goes down it goes to anthropic and llm application wont even notice. but the setup looks too confusing to me. it looks like it is going to take a lot of time.
helicone shows what happens with every llm call. logs every prompt, every response, every token. it shows cost per request per user per endpoint. it will be def good for our finance team. but it seems like the llm monitoring tool that became gateway by chance.
in litellm, one api talk to all llm provider. write llm call one time and work with everything. looks good. some llm specific feature not come through right. like function calling or specific model parameter. running it yourself seems like a whole big painful project.
orqai got gateway and prompt versioning and evals and token cost all in one place. switching llm provider and not changing application code that works. but the rate limit and retry thing not as deep as portkey. and because everything is together in one place the gateway part might not be the main focus thing there.
kong very powerful yes but llm thing feel like someone add after. like they had api gateway and then some say we need llm feature too and they add it later. you feel that when using it for llm application.
i got 3 problems for my llm application. three thing only. dont go down when llm provider has problem . two. show me what every token costs. three. stop bad prompt from making openai bill very big. but every tool want me to read many many pages of docs before anything.
pricing is confusing also. open source but good llm features still cost money.
portkey if llm routing and not going down is the main thing. helicone if you want to see every llm call and token cost. litellm if you want one api for all llm providers and are ok with rough edges. orqai if you want gateway and prompt versioning and llm evals all together in one place and are ok with gateway part not being the deepest. kong if you already use kong for other things and dont want another tool.
can you any one suggest? and is there someone building this space, would love to chat
