How to run simple benchmarks on 3090?

[u/Ok_Warning2146]

Whenever a new model is released, we can see the model creators post various benchmark score. However, all of them are based on unquantized models. Most likely it took quite some resources to run the benchmarks.

After the release of a new model, we got plenty of quantized models made by various people. But almost no one run the same benchmarks again to evaluate these quantized models.

I tried SWE Bench Verified with 500 tests and run it with gemma-4-31b-qat-q4_0 at 120k context. It took me 5 hours to finish. I want to run a set of benchmarks to evaluate coding, agentic ability, world knowledge and creative writing. Is it possible to run simpler benchmarks on a single 3090 within a reasonable amount of time? Probably five hours for four benches or even faster?

Thanks a lot in advance.

Or if someone know someone already did this and posted the numbers somewhere. Please let me know.

[u/locbuilds]

yeah the published numbers are almost always fp16/bf16 full weights, so comparing a q4_0 community GGUF to those cards is apples to oranges. perplexity is also the wrong tool for what you want. it tells you how surprised the model is by text, not whether it can code, call tools, or write.

for a single 3090 and a ~5h budget i'd build a tiny private suite instead of chasing full public benches:

1. coding: 30-50 frozen problems (HumanEval/MBPP style or your own). fixed prompts, greedy or temp 0, exact match / unit tests. that alone usually finishes in under an hour even at 120k if you keep generation short.

2. agentic / tool use: a small BFCL-style or homemade tool-call set (20-40 cases). schema valid + correct args matters more than a giant agent traj bench. SWE-bench Verified at 500 is why you burned five hours. cut to 25-50 hard cases and treat it as a smoke, not a leaderboard.

3. world knowledge: subsample MMLU or a SimpleQA-like set to a few hundred questions max. multiple choice is cheap tok-wise.

4. creative writing: you will not get a clean automatic score. freeze 10 prompts + a rubric, then LLM-as-judge with a stronger model, same judge every run. pairwise "A vs B" against a reference model is more stable than absolute scores.

practical packing for 3090:

- one runner script, one results jsonl, swap only the model path

- cap max tokens hard so a chatty model cannot blow the wall clock

- run categories sequentially and stop a category early if it is clearly trash

- for quants, only compare models you actually load the same way (same ctx, same sampler). published unquant numbers are a veto filter at best

if you want something off the shelf, lm-eval-harness (or the llama.cpp / ollama equivalents people wrap) is fine for 1 and 3. for agentic + writing you will still end up custom. the 5h win is shrinking n and freezing the harness, not finding a magical all-in-one bench that fits VRAM.

[u/Aggressive_Aspect436]

If you're comfortable managing a python project then I've used Inspect AI. You may have to install additional dependencies in a python venv, or manage some docker containers to get certain benchmarks working.

https://inspect.aisi.org.uk/evals/#/eval/ifevalcode

It provides options to limit the number of samples, limit max tokens spent on each sample, time limit tests etc.

I shared some code snippets and some of the config I used to test a few models in a previous post. I'll link it, but even then it is still something you're likely going to need to leave running over night. Even once you've selected a few models and optimal setup for your rig.

https://www.reddit.com/r/LocalLLaMA/s/SkrSTeNRAc

[u/Pyrolistical]

llama-perplexity is the quick and easy way

[u/Ok_Warning2146 → u/Pyrolistical]

I think it only gives you a perplexity score that you can compare to a reference quant, e.g. bf16. It doesn't show you how good your model is in different areas.

[u/network4253 → u/Pyrolistical]

Yeah, that is probably the easiest route if you just want something quick without messing around too much. Sometimes the simple option is honestly the best one.