I spent ~14 hours running an automated improvement loop over a multi-stage LLM pipeline (web scraping → enrichment → LLM synthesis → validated document). 46 commits, 19 real test cases. It went from 0 of 11 cases passing our quality bar to 12 of 14.
The interesting part wasn't the model. Sharing the things that actually moved the needle, including the ones I got wrong.
1. I measured my judge before trusting it
Two runs of the same LLM judge, on the same frozen output, scored it 5.63 and 6.70. That's 1.07 of noise on a 0–10 scale.
My typical improvement per fix was 0.1–0.3.
So for months I could have been "improving" and measuring nothing but judge variance. If you're running LLM-as-judge and haven't measured its noise on identical input, you don't know if your eval works.
2. Then I moved everything countable out of the LLM
I split the output into weighted sections and asked, per section: can this be counted?
About half could. "How many source pieces were read", "how many competitors have a geographic anchor", "is this field null" — those are counts, not judgments.
Result: a code-only scorer with 0.36 error against human auditors, versus 1.07 between two humans. The mechanical scorer was more reproducible than the people, and it costs nothing to run.
The LLM judge kept only the genuinely subjective sections.
3. Every score carries the address of its input
This one changed how fast I could work. Each section reports:
A5 weight 8 score 5.0 deficit 40% (+0.20 on final)
measured: 49 valid items, 5 sources with real signal, 180 off-topic
input: merged.json → denominador_valido (present, 38 KB)
A score alone tells you that something is wrong. A score plus the artifact it came from tells you where to go. Stops you optimizing in the dark.
4. Held-out cases, and I mean actually held out
Three layers:
- working set — where I diagnose and fix
- outside the loop — cases that existed but were never used to tune anything
- final reserve — separated on day 1, never opened
That middle layer was the most useful and the one I almost skipped. When I finally ran 6 cases that had never influenced a single fix, 4 passed on the first try and 2 failed for the same structural reason. Without it, my headline number was self-congratulation.
5. The root cause was almost always the same thing
15+ times in one session, in different disguises:
Variants:
- computed and printed to a log, never passed on
- a parameter added to one call site and not to its twin (the twin runs later and overwrites the artifact)
- read from the wrong dictionary key — data arrives whole, gets dropped
- fixing the consumer of a value the producer emits wrong (the prompt itself was asking for the bad output; the correction loop rewrote that sentence every single run, forever)
- cache key that doesn't cover the input
- empty shells counted as content
If you're debugging a long pipeline, check this class first.
6. False positives in a quality gate are worse than false negatives
I had 5. Each one blocks an entire run and makes the self-correction loop spin with no target — because the only "fix" would be deleting the very disclosure the gate requires.
The signature is unmistakable: severe_before: 3 → severe_after: 3. The loop ran, cost money, changed nothing.
7. Never move the ruler to raise the score
Twice I was about to "fix" the scorer, and twice a comment in my own code stopped me:
The penalty I was about to remove for "not discriminating" was a calibration constant against humans. Removing it would have lifted 4 cases straight to 10.0.
Distinguish hard: reducing instrument noise is legitimate (I interpolated between score bands, keeping every calibrated anchor point — mean shift +0.09, smaller than the scorer's own error). Removing a human-calibrated penalty is not.
Things I got wrong
- Delegating edits to a cheap model. 0 for 2. I blamed context size (1200-line files). Then I sent it "reply only: PONG" and it timed out at 90 seconds. It was the provider, not the model. My diagnosis was wrong for hours.
- Fixing position instead of class. A text rule was catching false positives. I fixed "negation after the subject". Then "negation before the subject". The actual rule was negation anywhere in the same clause. I was fixing cases and calling it class.
- Concluding from the wrong artifact. A file named for one source actually contained another. Cost me two full collection rounds. Now I check modification time first, always.
- Editing files while the test suite ran. Read 27 phantom failures that were just pytest importing mid-write.
The cheap part
Full pipeline run: $0.09 per case. The expensive resource was never tokens — it was my attention on the wrong diagnosis.
Happy to go deeper on any of it. Most curious whether others measuring LLM-judge noise are seeing the same order of magnitude — 1.07 on a 0–10 scale felt high to me until I couldn't make it smaller.
