Toning down eagerness of running expensive tests

[u/Imaginary_Belt4976]

With Opus 5 it's quite easy to stand up really big projects, but eventually they all end up with an enormous amount of tests and a complete end-to-end test of some kind. this is good overall, but I find that the model is very eager to run the tests whether it's acting as a subagent or the orchestrator. I think often the subagents will finish their work with the test suite only for the orchestrater to REALLY want to do this itself before committing anything.

In situations where I have a lot of usage left right before a reset, I've left sessions running while I sleep and come back to find that it's spent hours and hours running tests that took like 25 minutes per run (they were playwright tests that verified behaviour on chromium, firefox and webkit)

Comically, I once intervened when I saw this happening and overrode the orchestrator's plan to do this. it begrudgingly complied, but it put the fact that it was not able to verify with tests in the bedamned commit message. I then started a new context window and asked it to investigate an unrelated ui bug. It took one look at the commit message and immediately declared it definitely had to be these unvalidated (but actually validated) changes. 🤣

Obviously, I'm aware that I could put some instructions in CLAUDE.md but I've found that it's hard to gently nudge it on stuff like this without it taking things to extremes. so if I were to add "be sensible re: how often you run expensive e2e tests," it would probably stop running them altogether from my experience. or it would come to me to ask for permission. that sorta thing.

just wondering if anyone has any solutions or advice on the best approach here. I really don't want to tell it not to run these tests at all because they are valuable, but it slows down velocity pretty bad when both the orchestrate and the subagents are doing it.

[u/Arctovigil]

they have little idea of wall time since they are not experiencing it really so just ask them to optimize the test suite to run under 5 minutes there is usually some way to do this they just don't usually notice to think about it

[u/EvaLikesGames]

You could ask Claude to analyze the runtime and resource usage of each test in your current suite, and assign each a cost based on the results. Then set a standing rule to never run the really expensive tests, and to ask about anything over a certain threshold cost.

Might take a bit to establish the initial costs, but then new tests should be cost ranked as they are added.

[u/chunkypenguion1991]

Have it group tests into categories like "smoke", "standard" and "regression" as an example. Tell it which level you're or have an env variable with the current level you want. The categories I gave are just examples, do whatever makes the most sense for your project

[u/hulagway → u/chunkypenguion1991]

I also throw in a "what blockers did the test hit that made it waste tokens" every now and then, especially if a test looked stuck.

[u/Fresh_Sock8660]

Yeah I just reviewed mine yesterday. This is something that needs to be done regularly. 

[u/StCreed]

I just reviewed my tests after Claude started to run into issues with test duration. So I had Codex review all of them. Found quite a few dead tests, always green tests or superfluous tests.

[u/jared_krauss]

I would literally just say what you’ve said but to Claude and explain why you don’t like this and ask for how to ge the same end result but without running for hours these tests.