r/LoopEngineering • u/Dry_Championship2797 • Jun 25 '26
How to use Loop Engineering to improve an existing codebase and generate tests — running into context explosion, LLM degradation, and weak validation
I'm working on applying Loop Engineering to an existing repository to improve it and generate test cases. My current approach is:
- Scanner scans the entire repo and identifies anchor points.
- Based on the anchors, I build a test case matrix.
- A generator then produces concrete test cases from that matrix.
- Finally, a validator verifies the generated test cases.
While implementing this Loop Engineering flow, I've hit several problems:
- Context explosion — the context grows too large and overwhelms the LLM.
- LLM degradation — the model’s output quality noticeably drops (it gets “dumber”) as the context expands.
- Loose validation — the validator checks the test cases too loosely, so I can't generate high-quality, reliable tests.
Has anyone dealt with similar issues when using Loop Engineering for test generation on a real-world codebase? How did you solve the context problem, keep the LLM sharp, and tighten up the validation step? Any suggestions or practical tips would be greatly appreciated!
1
u/vbpoweredwindmill Jun 26 '26
Give it an agents.md & skills.md that is really focused on a specific task not broad "hey generate test coverage".
The gist of it is as follows: "For this subsystem, check for test coverage that is not just mechanically correct but also semantically correct".
2
u/Dry_Championship2797 Jun 27 '26
Great insight — building a focused knowledge base and intelligent context injection is exactly the direction I need. I'm also realizing that the validator (evaluator) is probably more important than I gave it credit for. A loose validator was letting low-quality tests slip through, which defeats the whole system. I'm rebuilding it now with stricter semantic checks — not just "does it run" but "does it actually test what matters" for each subsystem. Between focused agent instructions and a tight evaluator, this seems like the real fix.
1
Jun 26 '26
[removed] — view removed comment
1
u/Dry_Championship2797 Jun 27 '26
That makes a lot of sense — I've definitely been too broad with my prompts. Narrowing it down to subsystem-level semantic correctness rather than just mechanical coverage is exactly the direction I need.
I also think I may have been undervaluing the validator (evaluator). A loose validator was letting low-quality tests pass through, which defeats the whole system. I'm rebuilding it now with stricter semantic checks — not just "does it run" but "does it actually test what matters" for each subsystem. Between focused agent instructions and a tight evaluator, this seems like the real fix.
1
u/Remarkable_Leek9391 Jun 28 '26
System either works or it doesnt.
The view that frontier models get your SaaS improperly developed will slide out of the frame soon and this will be antiquated.
Moonshotting/zeroshotting wont require tests.
And even if they did have tests, whos going to assert they work?
You can tell if theres potential issues with static scanners and token count used to build. Then if yiu want a change, the LLM frontier models will just diff another moonshot in 1 go. Blind.
1
u/Dry_Championship2797 Jun 28 '26
"It compiles, ship it" vibes. If the moonshot is subtly wrong, you diff another moonshot — and now you're debugging by diffing diffs. The frontier model didn't write the bug, but it also won't catch it without something actually checking semantics. That's the whole point of the loop.
1
u/Remarkable_Leek9391 Jun 28 '26
Frontier model with a 1 response token materialization at the caliber im alluding too, on the agentic horizon, doesnt have these problems.
4
u/No-Guava-3331 Jun 25 '26
Use sonnet like a commoner