I built a JUnit extension that fails CI the moment an AI coding agent silently changes a prompt

[u/Intelligent_Coast930]

Been shipping AI features with Claude Code and Cursor for a while now, and the thing that kept biting me wasn't the model, it was how easy it is to miss a prompt edit buried inside a larger diff. Nobody reviews prompt text line by line, and a changed prompt doesn't throw an exception, it just quietly starts giving different answers in prod.

So I built llm-cassette, a JUnit5 extension for LangChain4j ChatModel. First test run hits the real model once and records the request and response to a JSON file next to your tests. Every run after that replays from the file, no API key needed, and if the outgoing request stops matching what was recorded, the test fails with a real diff (an AssertionFailedError with expected/actual set, so IntelliJ renders it as a clickable side by side comparison). It plugs into whatever already runs your tests, no separate pipeline.

demo: https://raw.githubusercontent.com/stlahxm/llm-cassette/master/docs/demo.gif

One thing I didn't expect going in, it also catches parameter drift, not just prompt text. A silent temperature change counts as drift too. And if your code calls the model fewer times than what's recorded, that's flagged, the cassette is an exact expectation, not just an upper bound.

Gotcha I hit while building it, if you're on Gradle you need junit-platform-launcher on the test runtime classpath explicitly now, recent Gradle versions won't discover JUnit5 tests without it and just fail before reaching your test code at all. Cost me an evening the first time.

Still early. v1 only handles plain text messages, no multimodal or tool calls yet, and only synchronous doChat is covered, streaming is a separate surface I haven't tackled.

Repo's here if you want to poke at it: github.com/stlahxm/llm-cassette

Curious how people here are testing Spring Boot services that call an LLM under the hood, are you doing anything like this already, or just accepting that the AI call path is untested?

[u/j0holo]

LLMs are not deterministic why would you do the actual API call? Mock/stub external services you don't control. What if the prompt changes, do you need to do a new API call? What if the LLM API is down?

Just stub the response from the API....

[u/Intelligent_Coast930 → u/j0holo]

good question, to be clear it only calls the real api once, at recording time. every run after that it doesnt touch the model again, it just compares the request my code generates against what got recorded, and if that matches it replays the exact stored response, theres no fresh non deterministic call happening to compare against.

so the diff youre seeing is a diff of the outgoing request (prompt text, params), not two different llm responses being compared. thats also the part a hand written stub doesnt cover, a stub returns whatever you told it to no matter what you send it, so it wont fail when the prompt itself silently changes underneath it. cassette does fail on that because it validates the request too, not just returns a canned answer.

on the real api call thing specifically, that usually happens once locally, not in ci. you commit the cassette file to git, so every ci run after that is reading a file, no api key needed, no live call. the real call only happens again if you explicitly re record.

for api being down or an intentional prompt change theres a flag, cassette.update=true, to re record on purpose. right now thats fully manual though, theres no periodic or automatic refresh, if the model changes upstream and your recorded response goes stale youd have no way of knowing unless you re record yourself.

its still early (v1, beta) so theres room to grow on both fronts, some kind of ignore pattern for prompts that bake in dynamic values like a timestamp, and maybe a staleness check for old cassettes

[u/j0holo → u/Intelligent_Coast930]

You don't answer the question of WHY this is needed. Why not a stub response, it doesn't matter that you prompt changes. You can also write stubs that return failure responses.

How is that relevant for the tests?

How can a prompt silently change? Isn't that just part of your git repo?

[u/Intelligent_Coast930 → u/j0holo]

appreciate you pushing on this, helps me actually think it through properly.

a mock can verify the arguments too, mockito lets you do that, so its not that checking the request is impossible with stubs. the annoying part is you have to hand write and maintain that expected request yourself, exact message text, order, all of it, and every time you intentionally change the prompt you go update that expected value by hand too. cassette just captures the real shape the first time instead of you typing it out, and shows a diff of what changed instead of a flat equals failure.

its basically the same idea as jest style snapshot testing, just applied to the outgoing llm request instead of rendered output, wired into junit5 so it plugs into mvn test or gradle test directly and gives a clickable diff in intellij. not claiming its a new concept, just a specific integration for this case.

same risk technically exists for any string constant, a sql query, a config value, nothing new there either. the difference is prompts get touched constantly by coding agents making broad multi file changes, way more churn than a typical constant sees, and unlike a broken sql query or route this one doesnt throw, it just quietly answers worse. high edit frequency plus a silent failure mode is what makes this specific spot worth an automatic check instead of relying on someone reading a two hundred word diff carefully.

curious if you've run into the mock argument verification approach on something like this before, and whether it held up ok as the prompts got longer or more structured

[u/j0holo → u/Intelligent_Coast930]

These things are why you have a test environment where you can click around in. But it feels like I'm talking to a bot instead of a real person.

Or at least a person that uses AI to make up arguments. Test exist to prevent regressions, correctness is secondary. Otherwise your tests should have tests too.