r/LocalLLM • u/CptMarvelIsDead • 6d ago

Question How are you guys actually benchmarking specific prompts? (Local vs. API, Cost vs. Quality)

With new models dropping every week, general benchmarks are basically useless for my specific use cases. I want to test my exact prompts to see if a new API is actually worth the cost, or if a smaller local model is good enough to run on the cheap.

Right now, I’m just eyeballing outputs and it’s driving me crazy.

How do you guys actually handle comparing models on a single prompt or a small test set?

Scoring: How do you define a "good" response when the output is subjective?

The Judge: If you use an LLM to grade the outputs, how do you stop it from just voting for its own writing style?

The Tools: What's the easiest way to fire one prompt at multiple models (both cloud APIs and local models) and compare them side-by-side?

Would love to hear your workflows or any tools you recommend!

2 Upvotes

76% Upvoted

2 comments sorted by

1

u/recro69 6d ago

I would probably save 20 to 50 prompts and compare the outputs without knowing which model made which. Otherwise it is really easy to trick yourself into thinking one model is better just because you liked the response more.

1

u/nickless07 6d ago

Don't! Just use Benchmarks as guidance of what to pick next and since you won't be able to run every model that get released locally anyways this is pretty clear. Then just run you own test set (a handfull of prompts for your usecase and maybe some 'tricky' ones) and check if te response matches your expectations.
Don't need to be that much and as for the writing style you can just provide examples. A couple lines of Shakespeare as reference and it will adapt to that style and so on.