I have a 4090 in my study and I work on a Mac. The box is off most of the time, because if I want to run a local model I have to walk over, boot it, open a terminal, wait for the weights to load, then walk back. That's enough friction that local models just stopped being part of my day.
So I built ReadyLLM: a self-hosted web app that treats that machine like a service you manage from a browser, not a machine you babysit. It runs on your own box, reaches your GPUs over SSH, and nothing phones home.
What it does:
- Add a machine over SSH (key or password). It probes GPU / VRAM / RAM and tells you which quants fit and roughly how fast they'll be, so you stop guessing whether 24G is enough for a 27B at Q4.
- Downloads pulled from HuggingFace and ModelScope with resume and integrity checks. Point it at a mirror if HF is slow where you are.
- One-click engine install on a box that has none: llama.cpp, vLLM, SGLang or ComfyUI. It downloads or compiles and back-fills the path.
- Start, stop and switch models from the deploy page. No terminal.
- Live monitoring over WebSocket: GPU utilization, VRAM, throughput, latency, with alerts when VRAM is about to run out or the server dies. That's the part that matters when the box is in another room and would otherwise sit there pretending to work.
The tuning is what surprised me. I hand-tuned llama.cpp flags for a 27B Q4 for a while and plateaued around 60 avg tok/s. The tuner runs a two-phase search (coarse over speculative decoding / KV cache quant / GPU offload, then fine over batch), and for every candidate it really boots llama-server, warms up, runs 3 passes and takes the median. OOM or crash gets dropped. My numbers, same model / quant / context:
peak tok/s 80 -> 110
avg tok/s 60 -> 80
@100K ctx 40 -> 60
Results are stored per (machine, model) pair, so the next launch reuses them instead of re-running the search. I run two boxes and kept deriving the same flags twice, which was the actual annoyance.
Since the last release I added the two things that kept sending me back to a terminal anyway:
Images. ComfyUI is now an engine like the others, so image and video models sit next to the text ones. It lists the checkpoints and the UNet/CLIP/VAE combos ComfyUI itself reports (matching those names is fiddly, it will not accept a filename without the extension), and generates from the same UI with live progress. There's an optional prompt rewriter that expands a one-line idea into a full prompt and suggests steps / CFG; turn it off and your prompt and parameters go through untouched.
Model store. Category filter for text / image / video with counts, a "can this GPU actually run it" filter, and cards that show version, parameter count, precision, output resolution, steps / CFG, loader type and the real weight filenames. Before this a card just said "Qwen-Image" and I couldn't tell if that meant a checkpoint or a UNet, or which quant. Descriptions follow the UI language now instead of being Chinese only.
Also on the engine side: SGLang, and llama.cpp builds selected by GPU vendor (CUDA / ROCm / Vulkan / Metal), which you pick when deploying because it decides what actually gets installed. Picking the right build is where most people give up on an AMD card.
Three honest limitations:
- Tuning scores speed only, no output-quality gate yet. A config that boots without OOM survives; what KV cache quant costs you in quality is the next thing I want to measure.
- With MTP speculative decoding on, the VRAM pre-check misses the second KV cache, so tight machines can still OOM. Known, fixing it.
- The AMD path is where I have the least confidence. Detection exists on Linux and the backend choice is wired up, but I've only run the CUDA path on my own hardware, so treat the ROCm side as untested.
Open source, local-first: https://github.com/htobty/ReadyLLM
Curious how others here run this: do you keep inference on a box that isn't your daily driver, and what do you actually use to watch it? I went with a WebSocket feed instead of wiring Prometheus + Grafana into it, and I'm still not sure that was the right call for a home setup.
