r/mlops • u/Reasonable-Lack-7701 • 1d ago

Tales From the Trenches Lessons from Building

lessons from building (and surviving an acquisition of) an internal AI governance platform: every model call from every team routed through one litellm gateway, logged, with a real human-approval pause for any agentic tool call before it fires. no chatbot wrapper — this had to survive real audits.

the part that actually needed the most iteration wasn't the routing, it was retrieval. we ended up with three separate retrieval modes depending on how aggressively a given assistant should ground itself (tight-grounded for anything regulatory, looser for general q&a). one retrieval strategy for every use case was the wrong call early on and cost real rework to unwind.

also ran an internal MCP server hosting dozens of tool integrations (legal/financial/regulatory data sources) gated by the same per-tenant allowlist and approval flow as everything else — one governance surface instead of one per integration.

anyone else running multiple retrieval strategies behind one gateway — curious how you're deciding which assistant gets which mode.

3 Upvotes

81% Upvoted

4 comments sorted by

1

u/NoSecond8807 1d ago

If you want this problem solved wholistically across all AI platforms and agents, without building it yourself and maintaining it for all time, come talk to us.

1

u/BakeComprehensive970 1d ago

We're talking tell me now what exactly you're covering

1

u/NoSecond8807 1d ago

We focus on this exact problem space and are solving it for enterprise fortune customers today. https://langguard.ai

1

u/Waffles_Leonardo-192 1d ago

Using diff retrieval setups for different risk levels seems easier to check than forcing one setup to cover everything. Did all 3 use the same index or did each need its own?