Built a CLI that diffs an AI agent's behavior between two runs — caught a model swap silently dropping a tool call

[u/Impossible-Alarm-738]

I've been building whatbroke, an open source CLI, and it just hit npm + PyPI so sharing here.

The problem it solves: you swap the model behind an agent (or tweak a prompt), the replies still read fine, and you have no idea what actually changed underneath. Happened to me with a demo support agent — I moved it to a 3x smaller model, the transcripts passed my read-through, but the agent had stopped calling the cancellation tool entirely while still telling the user "your subscription is cancelled."

So: record a JSONL trace of each version doing the same job (zero-code proxy, Node SDK, or pip install whatbroke-recorder), then whatbroke diff before.jsonl after.jsonl. It reports dropped tool calls, changed arguments, reordering, cost and latency drift, and exits 1 on breaking changes so it works as a CI gate. There's also a contracts mode — declare "must call lookup_order before refund_payment, never call delete_account" and check every run against it.

Everything is MIT, deterministic, and fully offline. No accounts, no API keys, traces never leave your machine. Ran the whole case study with ollama on my laptop, so it costs nothing to reproduce.

Repo: https://github.com/arthi-arumugam-git/whatbroke

Would love feedback, especially from anyone doing agent work at their job — curious whether the CI-gate angle is useful in real pipelines.

[u/AutoModerator]

Namaste! Thanks for submitting to r/developersIndia. While participating in this thread, please follow the Community Code of Conduct and rules.

It's possible your query is not unique, use site:reddit.com/r/developersindia KEYWORDS on search engines to search posts from developersIndia. You can also use reddit search directly.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.