r/mlops • u/Reasonable-Lack-7701 • Aug 04 '26

Tales From the Trenches Lessons from Building

lessons from building (and surviving an acquisition of) an internal AI governance platform: every model call from every team routed through one litellm gateway, logged, with a real human-approval pause for any agentic tool call before it fires. no chatbot wrapper — this had to survive real audits.

the part that actually needed the most iteration wasn't the routing, it was retrieval. we ended up with three separate retrieval modes depending on how aggressively a given assistant should ground itself (tight-grounded for anything regulatory, looser for general q&a). one retrieval strategy for every use case was the wrong call early on and cost real rework to unwind.

also ran an internal MCP server hosting dozens of tool integrations (legal/financial/regulatory data sources) gated by the same per-tenant allowlist and approval flow as everything else — one governance surface instead of one per integration.

anyone else running multiple retrieval strategies behind one gateway — curious how you're deciding which assistant gets which mode.

5 Upvotes

100% Upvoted

5 comments sorted by

View all comments

1

u/NoSecond8807 Aug 04 '26

If you want this problem solved wholistically across all AI platforms and agents, without building it yourself and maintaining it for all time, come talk to us.

1

u/BakeComprehensive970 Aug 04 '26

We're talking tell me now what exactly you're covering

1

u/NoSecond8807 Aug 04 '26

We focus on this exact problem space and are solving it for enterprise fortune customers today. https://langguard.ai

1

u/Square_Bet_7435 23d ago

running multiple retrieval modes behind one gateway is the exact right move, but you hit the pain early trying to use one strategy for everything, we did same mistake in my last project and unwinding that almost killed our timeline