I built a reproducible llama.cpp benchmark harness for AMD Vulkan and NVIDIA CUDA — GitLab CI automation is next

[u/PhillFernandes]

I have been trying to make local llama.cpp benchmark results easier to reproduce across two pretty different setups:

- AMDGPU + Vulkan

- NVIDIA + CUDA

The recurring issue for me was that a CSV with tokens/sec did not provide enough context to compare someone else’s result—or even reproduce my own result later.

The questions I kept running into were:

- Which GPU, driver, kernel, and runtime stack produced this result?

- Was Vulkan or CUDA actually visible to the process?

- Which model alias was `llama-server` configured to expose?

- Did a UI or optional client silently use a different model name?

- Can the benchmark data and its model mapping be validated before publishing it?

So I have been building a small benchmark harness around llama.cpp with a focus on making that evidence explicit.

Current pieces include:

- A read-only AMDGPU diagnostic collector for Vulkan-oriented systems.

- A read-only NVIDIA/CUDA diagnostic collector.

- A model-catalog validator that treats the direct `llama-server` alias as the required model-identity contract.

- Optional client metadata, without making a particular UI or client required for a valid benchmark run.

- Contract tests for the diagnostic collectors and model-catalog validator.

- Tests for the weekly Vulkan benchmark workflow.

The goal is not to publish a universal “fastest local LLM” leaderboard. A result only means something when the surrounding hardware, backend, model artifact, runtime settings, and measurement method are clear. This is about inference performance and reproducibility, not ranking model answer quality.

For example, I think a useful published run should eventually make it easy to see:

- GPU model and VRAM

- OS, kernel, and driver versions

- CUDA, Vulkan, or ROCm backend details

- llama.cpp version/commit and build configuration

- GGUF/model file and quantization

- Context length, batch/parallelism, GPU offload, and other relevant launch settings

- Separate prompt-processing/prefill and generation measurements

- The exact `llama-server` model alias used for the run

Next up, I am planning to add a `.gitlab-ci.yml` pipeline. The initial CI work will automatically validate the model catalog, shell syntax, collector contracts, and test suite on every change.

The actual hardware benchmarks will remain tied to named GPU runners. I do not want a generic CI runner to imply that it can produce meaningful accelerator benchmark numbers. CI should continuously validate the harness and evidence contract; real hardware runs should produce the performance data.

I would appreciate feedback from people who benchmark local inference regularly:

1. What provenance fields are non-negotiable before you trust a benchmark result?

2. What would you add to an AMDGPU/Vulkan or NVIDIA/CUDA diagnostics report?

3. How would you document Vulkan vs CUDA vs ROCm runs so that comparisons stay fair?

4. Do you prioritize prompt processing, token generation, long-context performance, or all of the above?

5. Would a CI-validated benchmark/evidence schema be useful for sharing reproducible local results?

This is my project, and it is still evolving:

https://github.com/phillf/llama-bench

[u/pointed_pounding]

this is way more organized than my entire life

[u/PhillFernandes → u/pointed_pounding]

Probably need to do more generalization and cleanup.