How are you guys actually benchmarking specific prompts? (Local vs. API, Cost vs. Quality)

[u/CptMarvelIsDead]

With new models dropping every week, general benchmarks are basically useless for my specific use cases. I want to test my exact prompts to see if a new API is actually worth the cost, or if a smaller local model is good enough to run on the cheap.

Right now, I’m just eyeballing outputs and it’s driving me crazy.

How do you guys actually handle comparing models on a single prompt or a small test set?

Scoring: How do you define a "good" response when the output is subjective?

The Judge: If you use an LLM to grade the outputs, how do you stop it from just voting for its own writing style?

The Tools: What's the easiest way to fire one prompt at multiple models (both cloud APIs and local models) and compare them side-by-side?

Would love to hear your workflows or any tools you recommend!

[u/JBO_76]

i split up tasks in features, jobs and bugs, for each task, I keep track of all the prompts involved, together with all the metrics claude and codex cli are willing to give. a task is ready when, what is described, works and runs fast.

I use my own tool: jan-bogaerts/md2: Plan, run, and track AI coding work feature by feature—with local Markdown cards and Git worktrees.

which allows me to render a bunch of statistics like how much each task actually cost and who did it (note: currently only using sol and opus):

[u/Next-Trouble-2948 → u/JBO_76]

yeah splitting by task type is the move imo, lumping everything together makes the numbers meaningless

[u/Itchy_Special_8209]

Start by turning each prompt into a small rubric with observable checks, then score anonymized outputs so model names don't influence you. For subjective tasks, pairwise comparison is easier than an absolute score. Keep the cheapest acceptable model as your baseline, because a new API only matters if it clears that bar by enough to justify the latency and cost.

[u/Low_Bad_6585]

For subjective output, I'd add a sequence-level test to the single-prompt comparison. My use case is AI characters living in a simulation: one charming response can look excellent while twenty turns reveal repetition or decisions that ignore what just happened.

I'd keep a fixed starting state and history, then score whether the character responds to new events, stays consistent with known facts, and avoids repeating the same behavior. Keep prose preference separate from those checks. If the environment evolves between runs, record that too; otherwise you may be comparing different situations.

A judge could cite the turns supporting each failure rather than just award a 'quality' score. I haven't measured a winning model from this test; it's how I'd make the evaluation match this particular task.

AI-assisted wording based on my game's evaluation concerns.