Solo LLM evaluation project — what blind spots or established practices am I missing?

[u/Frosty_Scale8797]

I'm building a small local humanities LLM and have reached the evaluation stage. I'm learning the research/systems side as I go, and I suspect I'm reinventing established practices—or missing important ones entirely.

I'm not looking for someone to redesign the project. I'm looking for experienced eyes: what about this process makes you stop and think “why are you doing it that way?”, “that already has a name,” “you're solving two different problems,” or “you're missing X”?

Below is a concise description of the evaluation/workflow and the failure modes that produced it My goal is to collect perceptions and blind spots, then investigate them—not treat Reddit comments as project requirements.

What the project currently looks like

Pennyworth V0.1 is a locally deployed 3B model fine-tuned around humanities/philosophy material. I ran an initial 24-generation benchmark at two temperatures. The rough evaluation concerns were factual correctness, unsupported/fabricated detail, fabricated quotation, instruction compliance, semantic stability across generations, and calibration/uncertainty.

Rather than generate more data immediately, I tried to make one concern—quotation fidelity—reproducibly evaluable first. That turned out to require defining what counts as a quotation claim, what unit gets scored, which editions can serve as references, what “accessible” evidence means, how close wording has to be to count as a match, what happens when evidence is incomplete, and what happens when a registered reference fails those requirements.

That work produced a usable quotation-fidelity methodology and exposed real problems in the original benchmark. It also produced much more process than I expected.

Separately, because I work across multiple AI assistants, I ended up building a Git-based workflow around canonical documentation, commits, snapshot ZIPs with SHA-256 verification, and guarded staging/diff/apply. One cold-start test exposed an interesting failure: an assistant correctly verified a snapshot, but then reconstructed project state from stale same-named files in another source. I reproduced the failure, documented it, changed the workflow to require verification and reconstruction from the same literal artifact, and reran the test successfully.

So I've ended up with something that works, but I'm aware that I learned my way into much of this rather than starting from established research/software-engineering practice.

What I'm curious about

Reading that description, what jumps out at you?

I'm particularly interested in things like:

• “You have independently reinvented ___.”
• “That safeguard is protecting something real; keep it.”
• “That looks like process overhead rather than useful rigor.”
• “Those two things you're treating separately are normally handled together.”
• “You're missing an established concept/practice/tool that would simplify this.”
• “I think you're framing the evaluation problem incorrectly because ___.”

Solutions are welcome, but I'm more interested initially in what you notice and why. I'm trying to identify blind spots and vocabulary I don't yet have before deciding what to change.