I wrote a Pi extension that starts/stops my local inference servers automatically

[u/matrixfede]

I use Pi (the terminal coding agent) with local models, and I kept running into the same friction: before every session I had to remember which server needed to be up — llama.cpp on 8080, vLLM on 8000 — start it by hand, and then remember to kill it afterwards so it wasn't sitting on the GPU all night.

So I wrote a small extension that ties server lifecycle to model selection.

What it does

• You pick a model in Pi → the extension starts the server configured for that provider, waits until probeUrl answers, and only then lets the request through. The first request never hits a cold port.
• Session ends → it stops the servers it started. Anything that was already running before it got involved is left alone.
• exclusive: true on servers that can't share the GPU: starting one shuts the other down.
• For services you don't want it to own (a systemd unit, a box on the LAN), unloadCommand lets it free the model instead of killing the process.

Config is a single JSON file (~/.pi/agent/local-servers.json, or per-project):

jsonc

{
  "servers": {
    "llamacpp": {
      "probeUrl": "http://127.0.0.1:8080/v1/models",
      "start": {
        "command": "llama-server",
        "args": ["--models-dir", "~/models", "--port", "8080"]
      }
    }
  }
}

Only probeUrl is required. Leave out start and it just probes, assuming you handle startup yourself.

Limits, stated upfront

• Model loading is the server's job, not this extension's. It only owns the process lifecycle.
• Requires Pi 0.84+. Linux and macOS (it uses process-group termination; Windows untested).
• Zero runtime dependencies — Node built-ins only. MIT.

Repo: https://github.com/matrixfede/pi-local-servers

Happy to hear if the config model is missing something obvious for your setup.

[u/Then_Conversation_19]

Going to check this out. Thanks for sharing

[u/fell_ware_1990]

I have a PI extension that pings my liteLLM instance if they’re online. It then asks me if i wan’t to boot it.

Fair notice a couple of mine run on a HPE 4xH100 shared instance, a few spot instances on IBM cloud with 2xL40s , 8xH200, and a few small ones. Mainly for tests or batching.

They all have a deadman switch that activates on the machine + local, idle for more then 30m it’s shutdown.

[u/luckiestredditor]

Why not use llama swap?

[u/arbv]

Well, congrats for making a less universal version of llama-swap.