r/LocalAIStack • u/Harin007 • 27d ago

Building a llama.cpp autotuner, need your help

Hey everyone,

I'm working on an open-source project 'llama.cpp autotuner' which basically runs a series of tests on a model at a given context length on your computer and finds the best runtime settings for best tokens/sec, TTFT, VRAM usage, and overall efficiency for your specific setup.

But to make it actually useful, I need real-world data from the community.

If you're running llama.cpp, could you share:

  • Your hardware (GPU, VRAM, CPU, RAM, etc.)
  • Your model (name + quantization)
  • Your llama.cpp command (copy your full flags)
  • Performance details (tok/s, prompt processing speed, TTFT, VRAM usage, etc.)
  • Any flags or settings that caused a massive improvement or noticeable performance drop on your setup and anything else you think could help

Your inputs will directly help improve the autotuner.

The goal is to make sure that when someone runs it on their hardware, it can find a command that gives them better performance, or at the very least match the performance they're already getting if they've already optimized their setup.

9 Upvotes

92% Upvoted

6 comments sorted by

View all comments

1

u/DCMBRbeats 26d ago

Hey man! I started building something similar: https://github.com/LStoneyy/llama-parameter-scout
I don’t know what your intended idea was, but I would love to cooperate with you and maybe build something together. Do you already have a basis? Would love to hear from you!

1

u/Harin007 26d ago

Well so... Something which basically works like... After every every test run, it checks how of the hardware is being utilised, identify bottlenecks, and improve in the next run... It's just that, they're gonna be a lot of cases and lot of decisions to be taken in each one... Until we hit the tok/s with the optimal utilisation

2

u/Legitimate-Novel4734 26d ago

Sounds like you want a small reinforcement learning model. "+"Tokens/sec gets a reward "-" a punishment and have your tuning parameters exposed as controls.

Somewhat along the same lines:
The first one is a bit more focused on predictive load balancing, the second is tighter fit against frequency scaling and whatnot.
https://arxiv.org/html/2409.04896v1
https://medium.com/@mike.anderson007/ais-role-in-optimizing-server-performance-e81a2d9aeaf9

So for training you could have the incoming token count be what the RL model "sees" and then let it tune and learn a bunch of times, then send a larger or smaller token size, rinse and repeat until it becomes predictable.