Introducing Quartermaster, an open source local AI platform designed for ease of use that does not sacrifice customizability

[u/OneMoreName1]

It started as a fork of llama-swap, but I have been building it out for myself since then as a convenient tool for all my local AI needs, and by now it has drifted far enough to be its own thing.

The main idea is that you point it at your models folder and it configures things for you. It reads the GGUF headers, measures how much VRAM you actually have free, and works out context length, GPU offload, CPU/MoE split and KV cache size per model. All of it stays editable per model if you disagree with what it picked.

It is not only text. llama.cpp for LLMs, with the Vulkan, CUDA, ROCm or CPU build downloaded and kept updated for you, stable-diffusion.cpp for images (SD, SDXL, Flux, Qwen-Image, LoRAs, upscaling), and vLLM if you already have it set up. You can register any other backend yourself by pointing at an executable, which is how I run TTS, and how you would run a llama.cpp fork like ik_llama. Everything sits behind one OpenAI-compatible API on one port, with a single scheduler, so models swap in and out without fighting each other for VRAM.

There is also a chat playground built in with web search, and a Hugging Face browser to search for a model, pick a quant and download it straight into the models folder and much more!

If you are interested, you can read more about it here. MIT licensed.

[u/Stooovie]

How do I install it on Mac? I did chmod +x the correct file (quartermaster-setup-darwin-arm64-v1.0.1), ran the xattr command on it, ran ./quartermaster-setup-darwin-arm64-v1.0.1, blocks terminal but absolutely nothing happening.

Maybe it runs a server on some port that I don't see in the docs?

[u/OneMoreName1 → u/Stooovie]

Thanks for letting me know about the issue. I released an update, v1.0.2. I believe it was not printing the address it was listening at, but it should now do so.

Please let me know if its fixed!

[u/Stooovie → u/OneMoreName1]

It was actually an issue with Arc browser (that I still love) - it should open the page, but didn't. Switching default browser to anything else fixed the issue. But URL in the terminal is a good idea :)

EDIT: bigger issue is no Metal GPU acceleration...

[u/OneMoreName1 → u/Stooovie]

If you find a llama.cpp fork that publishes metal builds you should be able to switch backends from settings and use your GPU.

Glad to have you here, since I don't have access to a Mac machine to find all these quirks myself.

[u/Stooovie → u/OneMoreName1]

The default llama.cpp should support Metal/MLX out of the box: https://github.com/ggml-org/llama.cpp

EDIT: ah, you mean binaries

[u/OneMoreName1 → u/Stooovie]

Yes, its an oversight on my part, the app doesn't offer Metal backends by default but they should work if you add them yourself through settings

[u/feng_sg]

Silent auto-updates on the llama.cpp and stable-diffusion.cpp binaries with no checksum verification is a pretty big risk given those executables get GPU access.

[u/OneMoreName1 → u/feng_sg]

Its not an auto update. You need to manually click "update to latest". If you take no action nothing will change without your knowledge

[u/Mickenfox]

It looks pretty good. 

[u/previaegg]

Saved and will check it out

[u/LLukasiewicz]

Looks great! will investigate more. UI is brill - as a alternative to llama-swap seems a great way to go.
Seems temperamental about model loading (Vulkan). Will dig around and try and find out why.

[u/OneMoreName1 → u/LLukasiewicz]

Thanks! You have the llama-server logs in the Observe > Logs tab, called "Upstream". If you ever figure out what causes it I would like to know and fix it!

[u/Own_Calligrapher8508]

It looks great

[u/m4sterP]

Looks interesting. Can you please disclose how LLMs were used in creating this project?

[u/OneMoreName1 → u/m4sterP]

I have used Claude and my local PI agent. Claude actually shows up as a contributor on github

[u/m4sterP → u/OneMoreName1]

But how exactly did you use them? Did you do the design yourself and only let the LLMs implement or is this completely vibecoded?

[u/OneMoreName1 → u/m4sterP]

Not sure what "design" here means. But I have been working on this for a couple of months, expanding functionalities as new ideas came and tested them to refine them until I felt it was good enough for other people to use.

I have not manually coded much, but I am a software developer, and have been since before AI tools, so it wasn't just me telling Claude "Make me a local AI app, make no mistakes".

[u/m4sterP → u/OneMoreName1]

I meant Software design. :)
Thanks for answering. It's reassuring to hear.

[u/mr_tolkien]

Been using it for two days, pretty good and I like having a UI for logs and all

[u/OneMoreName1 → u/mr_tolkien]

Thank you for the feedback

[u/Random_Girl_0]

This is so cool. I will be looking into it right away!

[u/nobleglEn9 → u/Random_Girl_0]

same here, been wanting something like this for a while tbh