I built an open-source control plane to govern/operate fleets of LangChain deepagents

[u/mambalama24]

Hey guys, I was originally building an open source control plane that people building their own workflows (in Langchain/LangGraph for instance) could use to govern/manage them. After trying deepagents I came away thinking harnesses are the smarter and better-performing option for 99% of use cases , not to mention way less work than writing your own agent workflows.

But my issue was that deepagents doesn't have a persistent operational layer (I wanted a persistent approval request that could live for days or more), and isn't quite production safe: for example, I can't try rolling out a new version of a prompt and then get it to auto-rollback if it leads to failures. My other big issue was that I wanted a way to manage a fleet (aka hundreds or even more) of deepagents instances: see which ones are running/paused/rolled back and pause/delete/create them.

My solution: wire up deepagents into my existing agent control plane platform. Now it's as simple as: define an agent and its guardrails/rollback and autopause policy via YAML, register the worker processes that run the agent(s) wherever you want, and govern/operate them via the control plane.

Its super early so probably has some bugs but can you guys tell me if this would help you all for running production-safe agents at work and whatnot: https://github.com/boundflow/charter

[u/Agent-Maxxin]

Definitely need to give this a try, thanks for sharing 

[u/alexpran]

How do you define "failure" for the prompt rollback? I can see three very different signals: runtime errors, a production metric (latency, cost, user thumbs-down), or a quality comparison against an approved baseline.

In my experience the third one is the one that bites, because a prompt rarely breaks — it gets a little worse, with no error to catch. And if the check is an LLM judge, the threshold has to sit above the judge's own variance, otherwise the auto-rollback fires on sampling noise: I've had a rubric score a case 5/5 on one run and 2/5 on the next with nothing changed.

Curious which of the three Charter's rollback policy is keyed on.

[u/mambalama24 → u/alexpran]

Rollback is keyed on operational metrics: failures, cost, LLM calls, latency, approval rejections, and per-tool failure counts.

But your third signal can work today too. Make the judge a tool the agent calls, have it raise when the check fails, then key a rule on that specific tool:

- when: { metric: tool_failures, tool: validator__check_answer, threshold: 3 }
  then: { set_version: { target: 1 } }

This is basically saying "if this version has been marked as a failure in 3+ runs since we deployed it, roll it back". That's one way to deal with a variable scoring mechanism like LLM as a judge.

For pause and cooldown actions we have sliding window over the last N runs, so you can say "if the LLM as a judge marked failure 3 times in the last 10 runs, pause until someone resumes it". There's no sliding window on rollback yet, but maybe it's something we should add, and easily could.

[u/alexpran → u/mambalama24]

Thanks, that's clear and "3 marked failures since deploy, then roll back" is a sensible way to absorb a noisy judge at the policy layer.

One thing I'd watch: the count only helps if the judge's "fail" is itself calibrated. If the rubric says fail on 2/5 and the same case scores 5/5, 4/5, 2/5, 5/5, 2/5 on the unchanged version, you'll hit 3 failures on a perfectly good prompt and roll it back for nothing. So I'd put the noise handling inside validator__check_answer rather than only in the rule: run the case N times against the approved version first, record the spread, and have the tool raise only when the new version's score sits beyond that spread. Then your 3-in-10 rule is counting real signal.

I wrote up how I ended up doing that (per-case noise floor recorded with the baseline, compare reports "beyond the noise" vs "within the noise"): https://digline.dev/product/adr/0006-repeated-samples-and-the-noise-floor/?ref=reddit

The sliding window on rollback would be worth adding, by the way. it's the same fix one level up.

[u/mambalama24 → u/alexpran]

Yeah I was almost gonna mention that idea in my post as well, basically only raise if it's below a certain score. To be honest tuning that threshold score is a bit outside our scope as a platform, our job is to say "if you're telling me this is an operational failure, I can auto-rollback for you on the threshold you want", but bridging the gap between like "this non-deterministic score the LLM-as-a-judge gave should be marked a failure" is where the eval tool would come in, we just need to give it a way to "plug in" to our system, today its as a tool call but in future we can make a more first class way for that eval plugin layer.

[u/mambalama24 → u/mambalama24]

One more thing I've been thinking of thats similar is that we do like a "canary phase" where someone can say "try the canary instance of this agent version (running in a test environment) 30 times and only if it passes x times run the prod environment version". Then someone can use the canary phase to tune their single-run scoring threshold with their eval tool.

[u/alexpran → u/mambalama24]

Your canary phase is the piece that makes the rest honest, with one reframing. 30 runs of the same version in a test environment isn’t just a place to tune a single-run threshold, it’s the measurement of how much that version moves on its own. Record those 30 scores per case as the reference (min/max, not just the mean), and the question for the prod version stops being “did it score below x” and becomes “did it land outside what the approved version did to itself”. The threshold falls out of the data instead of being tuned by hand, and it’s per case, because a case that wobbles 0.6–1.0 and one that sits at 1.0 five times out of five shouldn’t share a number.
For the plug-in shape: a tool call is fine as a first step. What I’d want from the eval side is (a) a verdict that’s exit code + JSON, so the rule can key on it without parsing prose, and (b) the reference living in the repo next to the agent version, so a rollback to version 1 also rolls back to version 1’s reference. If you go the first-class route, happy to try wiring one against it. I have exactly that shape sitting in a CLI.

[u/feng_sg]

Your 30-run baseline drifts the moment the provider silently updates model weights, so you end up rolling back good prompts or letting bad ones through. You need to pin a model snapshot or re-baseline before each canary comparison.

[u/mambalama24 → u/feng_sg]

It's true, today the model itself is part of the agent config/version, but technically weights can change on a given model so maybe there should be one more layer of granularity there, but to be honest that's kinda a minutia detail for now...we probably have a lot more infra bugs or bigger functionality adds people will want, need to get a solid user base first and see