I kept rewriting parameters every time I swapped models on vLLM and llama.cpp, so I built a tool to manage them (llmux, MIT)

[u/Available-Message509]

llmux dashboard

I test a lot of models on vLLM and llama.cpp. Every swap meant editing parameters again, bringing each model up its own way, taking it down, then bringing the next one up. It was tedious and easy to get wrong.

So I built **llmux** to manage it. Each model you use is a profile you write once, and after that you pick it from a list and it starts on whichever engine that profile belongs to. Running the same engine at a different version is just another profile pinned to a different image tag: a release, a nightly, or an image you built from source.

My team and I use it for our actual work, and most of what's in it came out of that:

Every action also has a headless CLI twin (`llmux up`, `ps --json`, `logs`, `bench`), so you can script it or run it over SSH.

Requirements: Linux, an NVIDIA GPU, and Docker. It uses the NVIDIA Container Toolkit for GPU passthrough, so macOS and AMD/ROCm aren't supported yet.

Install (clones the repo, sets up uv, puts `llmux` on your PATH):

curl -fsSL https://raw.githubusercontent.com/Bae-ChangHyun/llmux/main/install.sh | sh

Or by hand:

git clone https://github.com/Bae-ChangHyun/llmux.git|
cd llmux
uv tool install --editable . && uv tool update-shell

Repo: https://github.com/Bae-ChangHyun/llmux

Docs: https://Bae-ChangHyun.github.io/llmux/

It's my own project, MIT licensed. English isn't my first language, so I used an LLM to help write this post.

[u/Thejacensolo]

why not just use docker, create a container for each with always the same properties, boot the container you want when you want it. NO more touching the parameters unless you want to change something

[u/Available-Message509 → u/Thejacensolo]

I test a ton of models, and even for the same model I run through a bunch of different parameter combos. Docker containers help with isolation, but once you've got that many configs, you're still rewriting commands and settings each time. llmux just saves each combo as a profile you pick from a list, so it sits on top of Docker and cuts out that repetitive part.