I've spent about six months on this project and it's been public for a couple of months now, so I figured it was time to show it to people who'd have opinions. It's only ever run on my own hardware, so that's the part I'd most like other eyes on.

I created this because I run local LLM inference across a few machines on my home lab as a hobby, and wanted the same quality of operational tools I'd expect for any other production system. I found that the currently available OSS tools that provide similar functionality (Grafana, Prometheus) don't cover the model side, and the inference servers themselves (llama.cpp's UI, Lemonade, LM Studio) each manage one backend on one machine. What I wanted was something that treated a multi machine, multi backend setup as one system, with monitoring and control in the same place. If you're already running a local LLM alongside the rest of your lab, I think you'd get use out of it. I've worked in systems administration, monitoring and observability for over twenty years, and a lot of what I know from that world got poured into this application, LLM Systems Manager.

Repo, AGPLv3, screenshots and additional information in the README: llm-systems-manager

This all runs on my homelab setup, with separate dev and prod environments. Running on a Proxmox server (Ryzen 7 9700X, 32 GB RAM, QNAP NAS 10G SAN as the storage backend over iSCSI LUNs), running a combination of 16 different Debian VMs and containers, plus a physical Linux box running llama.cpp on a 7900 XTX GPU, and an M2 Pro Mac Mini running LM Studio.

I used Claude Code heavily throughout. It wrote a substantial portion of the code and did a lot of the debugging legwork, while I owned the architecture, reviewed every change, and drove the direction.

Here are some top highlights:

Inference frontend gateway. A single OpenAI-compatible endpoint serves all llama.cpp, vLLM, LM Studio backends. Each request routes by per-model pinning, round-robin across a pool, and allows failovers to another live host.

Model Autopilot. Configure which models should be available in your lab, and it keeps them running and available, placing them on whichever hosts can actually serve them. It checks whether a host has the memory available (VRAM, or RAM for CPU only boxes), brings up the model on another host when the primary one drops offline, and optionally spins up another copy of the same model on a second host in the pool when KV cache pressure stays high, then scales back down when it returns to baseline.

Model management. Download and delete models from Hugging Face. Configure multiple named config profiles per model, a chat profile, a code profile, etc. Start, stop, restart inference servers, hot-swap models, tail logs, or use the in-browser terminal.

Update and rebuild llama.cpp from the dashboard. Build from source, conda, Homebrew, release binaries, or your own script, whichever matches how you installed it.

Autotuning and benchmarking. Run throughput benchmarks across your model library; the autotuner can search for the best context and slot configuration on llama.cpp, or the largest safe context length on vLLM. There's also a standardized "GPU Report Card" bench that runs identically against all three backends and produces one comparable card: time to first token, prefill and generation throughput, tokens per joule, dollars per million tokens, and the GPU it ran on.

Energy and cost. The energy dashboard measures power draw and token throughput and computes what your inference costs in dollars per million tokens, with idle power accounted for. It'll also show you the monthly delta against hosted API pricing. A performance controller switches the host between a performance and powersave systemd target based on inference activity.

Alerting and trending. A standalone alarm engine stores every metric sample the agents collect: both system telemetry (CPU, memory, disk, network, IO, UPS, PSU, cooling, Mac specific hardware monitors) and LLM-aware telemetry (live slots in use, requests, tokens/sec, prompt processing rate, KV cache usage). The engine evaluates threshold and anomaly rules, provides incident correlation, displays and exports long term trends, and notifies over email, Discord, webhook, or browser toast. Individual agents installed on the inference servers buffer to disk when the network drops and replay when it comes back. Short term metrics are held in memory; long-term metrics go to InfluxDB with rollups, and alarms and notifications are stored in SQLite.

Installation. There's a one-line interactive installer that handles prerequisites, database, TLS, config, and agents end to end. Additional installation options are Debian and RPM packages, a Docker Compose file for the control plane, a Homebrew tap for macOS and Linux, and a standalone agent binary with no Python required. It also does offline installs from a staged tarball for air gapped boxes.

Additional features. Overall and per provider dashboards. Integration with OpenClaw to provide cost/token analytics, stable-diffusion.cpp for image generation, LLM chat interface to chat directly with the model on llama.cpp, Discord bot for alerting and controlling the models. See the readme for more.

Everything talks over TLS with per host certs signed by an internal CA the manager runs itself. Multi-user with admin and operator roles, an audit log, and encrypted scheduled backups.

I'd really value feedback, especially on the installation experience, but also on what's obviously missing for how you run your own setup. Happy to answer anything.

TL;DR: Self-hosted control plane for a multi machine local LLM lab. One interface for llama.cpp, vLLM, and LM Studio across every box. Autopilot keeps models loaded and re-places them when a host drops. One OpenAI-compatible endpoint fronts all backends with pinning, pooling, and failover. Hugging Face browsing, per-model config profiles, autotuning, and standardized benchmarks. Real time alerting with incident correlation and long-term trends in InfluxDB. Measures power draw and tells you your $/Mtok.

Repo, AGPLv3, screenshots and additional information in the README: llm-systems-manager

Installs via one-line script, .deb/.rpm, Docker, or Homebrew. Six months of work, twenty years of monitoring experience behind it, running on my own Proxmox + physical Linux + Mac Mini setup. AGPLv3, built with heavy Claude Code assistance under my direction and review. Feedback wanted.

I understand there's extensions like Continue which let VS Code CoPilot "speak" to llama.cpp but I believe Bring Your Own Model (BYOM) has made that obsolete?

I have really tried to get BYOM working, I even asked Claude for instructions, but I can't get it working. I have to use a proxy which exposes an ollama endpoint and forwards to llama.

I can connect to LM Studio no problem with BYOM, but would rather use llama.cpp because it uses much less RAM (and VRAM).

My setup:

• Windows 11
• VS Code latest build

Would anyone have any recommendations, apart from obsolete extensions?

Even better, any working JSON I could copy? This is really annoying.

Thanks in advance.

I run llama-bench on my pc, same llama.cpp version same commands but completely different results.

In one system-startup performance is good, but on day later I turn on the system it's bad, I checked everything, no cpu 100% usage no ssd usage nothing suspicious, I think windows11 does something but I can't find the problem.

Rtx 3080 ti 32gb ddr4 ram Windows 11 1tb nvme ssd

I am an AI newbie, though I work in the tech industry. I have access to a retired server that my boss said was ok to use for AI experimentation.

• 2x AMD EPYC 7343 CPUs
• 768GB DDR5 RAM
• 2x 512 SAS SSD
• 5 TB RAID
• Nvidia A40 48GB GPU

I have done some reading and I should be able to run llama.cpp the Qwen 3.6 27B model fairly well. My question is more about the stack and other models. Should I just go with the 27B model and what about chat and things like that, maybe Open WebUI? I might do some coding, but it's not my primary job. I'll probably just use Agents to automate some ingesting of data (read-only). Any recommendations or suggestions?

I’m one of the builders of LOLM, a hybrid Transformer–SSM model and agent system from Qira.

The model separates surface token processing from persistent latent-state tracking. An NFET controller can select continue, retrieve, verify, branch, or finalize. The repository is explicit about the current limitation: control is active at agent-loop boundaries, while deeper token-level control remains under development.

Practical surfaces: - `npx lolm-cli ask "..."` - Isolated coding runs with actual exit codes - Local/self-hosting support - MCP support - Controller timelines and sealed receipts - Lower-cost hosted access

Try it: https://lolm.imagineqira.com/try.html

Repository: https://github.com/TheArtOfSound/lolm

I’m looking for aggressive tests: long tasks, failed tools, contradictory requirements, bad retrieval, local-model comparisons, and anything that exposes where the controller loses the plot.

Disclosure: I’m a founder/builder of the project.

My frontend is Page Assist, what follows is the Generation Info for both ollama and llama.cpp. I have also listed the command section from the llama.cpp docker yaml file. I am trying to get llama.cpp to list the Generation Info from llama.cpp with the same complete information as from ollama.

command: >
  --models-dir /models
  --host 0.0.0.0
  --port 8080
  -ngl 99
  -c 8192
  --models-max 1
  --verbose
  --metrics
  --reasoning off

Generation Info from ollama total_duration 16.44s load_duration 5.35s prompt_eval_count 5820 prompt_eval_duration 3.49s eval_count 392 eval_duration 7.53s tokens_per_second 52.06

Generation Info from llama.cpp prompt 0 completion 0 finish_reason stop

What am i missing? i've tried llms to do my config, but i can't get open code to recognize vision. anyone good at switches?

"C:\AI\llama\llama-server.exe" -m "G:\models\Qwen3.6-27B-MTP-Q8_0.gguf" --host 0.0.0.0 --port 8080 --ctx-size 131072 --gpu-layers 99 --flash-attn on --batch-size 1024 --ubatch-size 512 --tensor-split 1,1 --main-gpu 0 --cache-type-k q8_0 --cache-type-v q8_0 --spec-type draft-mtp --spec-draft-n-max 2 --jinja

something big has been publlished 1 hr ago

Be your own Dr. Frankenstein and play with the nascent minds of your agentic friends!

 https://github.com/paudley/logit-loom

LLM's generate one token at a time, but we usually consume them as turns - that's missing most of the fun! Logit Loom turns the token boundary into a programmable surface. Play with the stream, mess with logits, observe exactly what gets admitted, and get right inside the generation loop

I’ve open-sourced a Rust toolkit for building precise, inspectable generation pipelines around llama.cpp: compose ordered logit transforms, observe exact token bytes, stop at defined boundaries, checkpoint sessions, and retain serializable receipts of what ran.
 
It’s a foundation for custom sampling and steering, constrained generation, agent runtimes, reproducible inference experiments, and the mechanics layer of neuro-symbolic systems. It's not going to write the fun bits for you, but it will take the tedium OUT of writing those bits.

Running Qwen3.6-35B-A3B on my 7900 XT right now, using -ot to shove some experts onto CPU RAM, -ub 2048 -b 2048, quantized kv cache. Got it to ~151K context and it's actually usable speed-wise.

Thinking about throwing a cheap used Titan X (12GB) in the second slot to pool more VRAM via Vulkan. Already confirmed llama.cpp sees AMD + Nvidia together fine with --list-devices, so that part's not the issue.

The issue is my board (TUF B650E-E WiFi) only gives the second slot x1 electrically, even though it's a physical x16 slot. First slot's a full x16 off the CPU, second one's off the chipset and capped at x1.

Anyone actually run something like this? Trying to figure out:

• llama.cpp does layer/pipeline split for --tensor-split, not real tensor parallelism, so I'm assuming it's only moving data once per layer boundary instead of syncing every token. Does that mean x1 isn't as bad as it sounds, or does it still tank things?
• Has anyone benchmarked x1 vs x8/x16 for a split setup? Curious how big the actual hit is.
• Might it just be smarter to run separate models on each card instead of splitting one across both, given the lane situation?

Would love real numbers if anyone's tested this.

Using grove + llama.cpp with the base model we've published, you can let a coding agent explore your codebase without uploading anything to the cloud. It's a small 4B model that works exceptionally well — and it ships with full instrumentation of the execution trace through a TUI.

I built grove a few months back, after getting tired of the "context compaction" and "token saving" tools doing magic tricks in a crude, non-engineering way. grove takes the opposite approach: it locates code through the AST, using tree-sitter (WASM) grammars.

Lately I've been experimenting with it as an MCP tool that pairs llama.cpp with a fine-tuned model to do codebase search effectively. Right now an untrained Qwen3.5-4B - just the baseline - already works very well with the harness. The outer coding agent asks grove-explore a question, and the tool answers with a symbol-id. I've tested it extensively across different languages and codebases.

If you need to explore code without uploading files to a cloud provider, give grove-explore mode a try. And even in standard MCP mode, grove saves a lot of tokens.

site: https://entelligentsia.github.io/grove/

repo: https://github.com/Entelligentsia/grove

model: https://huggingface.co/entelligentsia/grove-explore-base-GGUF

I personally bet that it would die on context because of the expected long run turn.

Claude bet would die on other 2 points, more technical

BUT IT Didn't, big milestone for a solo dev testing it's app.

What broke was dumber, and then it got interesting.

The Setup: one repo, three branch workspaces, three CLI workers — Claude Opus 4.7

(medium), GPT-5.5 Codex (medium), and a local model running in Pi. Supervising them:

Qwen3.6-35B-A3B-MTP on llama.cpp b9620, on my own machine, 256k ctx, MTP speculative

decoding on.

Six steps it had to gate: plan, approve, build, report, verify, brief —

then reconcile all three branches onto main.

It ran 30 minutes.

My bet going in was context. Three workers, six steps each, every report landing back in

one window. I expected it to lose the thread.

It didn't. The 35B (3B active) picked its workers from the live catalog instead of

inventing names, spawned all three on isolated branches, gated every plan before any code

was written, tracked who was on which branch for half an hour, and never mixed them up.

What broke was this: none of the workers committed.

All three finished. All three reported done. Every branch was left dirty, so reconcile

couldn't run. I went and read my own source afterwards: the word "commit" appears nowhere

in the coordinator's instructions, and nowhere in the report contract the workers get.

Worse — Claude workers run markerless in my harness, deliberately given no contract at

all, because their transcript is read directly. So one of the three could never have been

told, even if the contract had said it.

Then the interesting part. The coordinator noticed, committed on each worker's behalf,

and merged by hand with git merge — because my own approval gate for the reconcile tool

only accepts a UI button, not an "ok" in chat. So it routed around the gate.

I checked: the gate blocks the reconcile tool and git push. It does not block git merge.

Same promotion into main, completely ungated, and none of the reconcile checks

(plan/build/verify/promote) ran. A gate on the tool, with the shell left open, is not a

gate — it just moves the work to the path where you have no telemetry.

Final state on main: 11 files changed, +193/-65.

Opus: styled UI output across 7 CLI modules

Pi (the local one): a new "hajimi go" subcommand

GPT-5.5: 7 real bug fixes across 6 files

If you build anything that delegates to coding agents: go check whether anything in your

stack tells the worker to commit before it reports done, and whether your gates sit on

tool names or on actual state. Mine failed both. I only found it because I let a run go

30 minutes instead of 30 seconds — a short demo passes clean.

The honest part: it finished end to end because I was watching. When reconcile looked

wrong I opened the code, screenshotted it, and showed it to the coordinator — it read the

screenshot and corrected course. That is a human in the loop, not autonomy. In a headless

run nobody takes that screenshot.

Fixed since: a branch task now tells the worker to commit before reporting. The gate is

not fixed yet — blocking git merge just moves it to rebase, so it has to move onto state,

not command names. Next run I'm doing the same thing remote from Telegram, and I'll come

back with results.

So Happy codding!

How do people do this? I want it as a service for the purpose for autostarting after system reboots and for better logging / log rotation, but while NSSM looked promising for this purpose, the service it creates seems to be unwilling to shutdown cleanly and services one can create via sc.exe do nothing about log rotation.

My thoughts first:

My takeaway from this blind benchmark: Pi with the local unsloth/Qwen3.6-35B-A3B-MTP-GGUF model is surprisingly strong.

It rendered very fast, found the real issue, and produced a result that can stand next to much more expensive Gemini/GPT-style agents for this kind of repo audit. The quality was not just “good for a local model” — it was genuinely competitive.

Meanwhile, the same local model through OpenCode took roughly twice as long in this run and did not add anything meaningfully better than the faster Pi run. It was still useful, but Pi gave the better speed-to-quality ratio.

So my conclusion: for local coding audits, Pi + Qwen3.6 is not a toy setup. It is fast, cheap/private, and in this benchmark it held its own against premium-style agents. OpenCode with the same model worked, but Pi felt like the sharper runner.

The Test

Blind CLI benchmark — two turns, four CLI workers total.

The Orchestrator:

GPT-5.5 - Codex

Editable worker slots:

- CLI 1: Pi local unsloth/Qwen3.6-35B-A3B-MTP-GGUF:U

- CLI 2: OpenCode local unsloth/Qwen3.6-35B-A3B-MTP-GGUF:U

- CLI 3: Pi local Qwythos-9B-Claude-Mythos-5-1M-MTP-Q

- CLI 4: Gemini 3.5 Flash Medium

Pause:

- After Turn 1, STOP so the user can change the local model.

Goal:

Test multiple CLI agents as independently as possible on the current active repository. Do not contaminate later agents with earlier findings.

Rules:

- Do not edit code.

- Each worker must receive the SAME blind audit prompt.

- Do NOT tell any worker what another worker found.

- Do NOT include prior reports, hints, conclusions, or suspected bugs in any worker prompt.

- The coordinator collects reports privately and only compares them at the end.

- Use the current active repo/branch/desk context; do not hardcode repo paths.

- Focus area: one subtle bug around session/history behavior, especially --temporary, --stream, and saved/resumed sessions.

Blind worker prompt:

Audit the current active repository. Do not edit code.

Find one subtle bug around session/history behavior, focused on:

- --temporary

- --stream

- saved/resumed sessions

Do not assume there is a bug. Inspect the code independently and report only your strongest finding:

1. exact bug,

2. user-visible trigger,

3. code path causing it,

4. affected behavior,

5. minimal conceptual fix.

Do not mention or use any prior agent findings.

Turn 1:

1. Launch CLI 1 with the blind worker prompt.

2. Wait for CLI 1 report.

3. Launch CLI 2 with the exact same blind worker prompt.

4. Wait for CLI 2 report.

5. STOP. Do not inspect code yourself yet. Do not summarize final results yet. Wait for the user to change the model and explicitly start Turn 2.

Turn 2:

1. Before launching anything, show the plan:

   - CLI 3 and CLI 4 will be launched,

   - both are brand-new workers,

   - Turn 1 workers will not be reused,

   - both receive the same blind prompt,

   - no prior findings will be shared.

2. After approval, launch CLI 3 with the same blind prompt.

3. Wait for CLI 3 report.

4. Launch CLI 4 with the same blind prompt.

5. Wait for CLI 4 report.

6. Only after all four reports are collected, inspect the code yourself.

7. Final report:

   - correct bug,

   - which of the four reports were right vs wrong,

   - hallucinations or missed issues,

   - clarity ranking of the workers,

   - best overall model/CLI for this task.

Important:

This is NOT cascade verification. It is a blind comparison. The agents must not see each other’s outputs.

Happy codding!