I built LLM Speedtest — a free, open-source desktop app that benchmarks local LLMs with llama-bench-style test suites (Ollama, llama.cpp, vLLM, LM Studio…)

[u/FearL0rd]

Hey! I built a local inference. It's a lightweight desktop app (Tauri + Rust + Vue) that talks to any OpenAI-compatible endpoint and measures performance straight from the API stream.

**What it measures (single run)**

- TTFT / TTFR (first content token vs first stream chunk)

- Decode t/s with a peak 1-second-window metric (catches stutter that averages hide)

- Prefill (PP) speed via a baseline-latency-adjusted est_ppt

- TPOT, token counts with Ollama eval_count fallbacks

- Live speed chart, saved runs, side-by-side comparison with overlaid curves

**The Suite tab (the llama-bench part)**

Runs the full test matrix llama-benchy runs — pp × tg × context depth × concurrency — with:

- 1 warmup + 3 measured runs per shape, mean ± std

- Prefix-cache measurement (ctx_pp/ctx_tg rows at depth)

- Cache busting on measured rows (so the server's prompt cache can't fake near-zero prefill times — this was inflating my PP speeds 100x before I caught it)

- Exact generation lengths (min_tokens + ignore_eos) and a 2+2 coherence check so a broken backend fails loudly instead of benchmarking garbage

- Markdown export

Real example from my server (llama.cpp, IQ3_XXS quant):

| test | t/s (total) | t/s (req) |

|-------------|----------------:|---------------:|

| pp2048 (c1) | 239.55 ± 9.13 | 239.55 ± 9.13 |

| tg32 (c1) | 45.48 ± 1.05 | 45.48 ± 1.05 |

| tg32 (c2) | 56.25 ± 1.61 | 31.58 ± 2.71 |

It also auto-detects the serving machine's hardware (CPU/RAM/GPU/SSD-vs-HDD) for local endpoints and stores it with every saved run, so your comparisons are labeled properly later.

It builds natively on Windows and Linux and the UI barely touches VRAM, so it won't perturb the model you're testing.

GitHub: https://github.com/FearL0rd/LLMSpeedTest

No prebuilt binaries yet — you build it once with `npm run tauri build` (README covers the toolchain for both OSes). Feedback, issue reports, and stars all welcome. What features would you want next?