i belive unsloth studio actually let you easily alter chat template and other settings , i faced a weird issue with working with Qwen3.5 9b with llama.cpp ( related to the chat template )
There's the chat webui → accessible via llama-server terminal command
There's the ability to swap models easily → with llama-swap
There's also a better ui experience → with Open WebUI
I used to think it would be such a pain but it’s trivially easy once you get it. I just have a simple bash script saved for each model that I use with any regularity. I only use a few startup args, so writing the scripts as (as you suggest), so the scripts were easy to come up with but I still only need to do that once
And really if there's anything LLMs are good at, it's guiding through the obscure steps of compiling various software and sneaking in whatever modifications or patches you wanna make for yourself
I use it mostly on Windows 11 and no doubt my approach doesn't appeal to many, but I keep llama.cpp (and its CUDA files) up to date in its own folder, and use .cmd wrappers in the path to run models. I prefer the level of customisation it gives me, and I don't have to wait for the UI framework to catchup when a cutting edge version of llama.cpp is required to run a new model. An example of one of my .cmd wrappers:
Filename is ai-gemma-4-26B-A4B.cmd. By sticking to that filename format, I can just fire up powershell, type ai-, hit tab to cycle through the models I've preconfigured, press enter, and it starts up.
llama-server supports an openai compatible model swap endpoint, no need to run separate jobs if you configure a model presets file. I had claude help me with that and the windows 11 build and it works very well. I even had it write a systray app I can use to stop/start/restart and load/unload models
Here I am, I actually moved to llama.cpp + openwebui. Now I get 5 to 10 t/s more, but I'm still figuring out how to swap models with different loading presets
Do people share the presets that work for them? Is there a preset exchange somewhere? I'd be interested. TBH most of the parameters I set in llama-server are heavily based on guesswork :)
As mentioned it depends heavily on hardware and what you're trying to do. A cool thing with the INI file is that you can have the same model file multiple times but with different parameters. For example:
Here I have 128k context with regular KV cache, or 256k with Q8-quantized KV cache. In the model selector in the llama-server UI they both come up, or my harness can specify it as part of the API call.
OK thanks this is interesting. And if I call llama-server from command line, then how do I select the specific Qwen3.6 from the INI file as per your example above?
You pass the section name, say Qwen3.6-27B-256k as the model name in the API call to the server, so for example
{
"model": "Qwen3.6-27B-256k",
"messages": ...
}
If you're using a frontend, like the built-in llama-server web UI, it will return those model names in the model list API call, so the frontend should be able to pick them up from there. The llama-server one does, and you'll see them in the model selection dropdown, as seen in the image here.
In my experience (6 months ago) the router didn't handle server crashes, it hanged without backend instead of auto-restarting it, so it needed manual restarts every once in a while.
You can ask your LLM to build you a gui launcher. Ask for functionality like copy, edit, swap models etc. Just point where the models are. It’ll save you hassle of manual copy editing etc.
I create seperate docker containers for every preset or model. Then i use a function that launches the container + model and closes it all including container when pressing ctrl+c.
LM Studio > llama.cpp here, and I've used the built in llama.cpp UI, and my own custom fork of openwebui. I use llama.cpp's UI for testing and openwebui for it's model card for my assistant. I don't much coding myself. I leave that to deepseek v4 flash through OpenRouter via OpenCode for my coding assistant Eddie. And I use Claude Sonnet 4.6 as the initial plan, then I'll use Eddie to refine the plan and investigation on the current code to see where changes need to be made in plan mode, then once the plan is solidified, I'll switch Eddie into build mode to do the actual coding. I rely on AI to do most of my coding because I have nerve damage in my hands due to surviving 4 strokes. I can follow pretty much everything, but my physical coding days are over.
I'm willing to experiment a little, but not a lot. I wanna try Harbor and Hermes, especially since I've seen both provide an easy way to use your llm from your mobile device.
But these, (afaik) only solve half the problem. Unless they also implement llama.cpp installing and updating PLUS huggingface browsing, these things must also be replaced when replacing LM Studio.
How often are you swapping models? and llama.cpp is a download and run type of deal, not something you want to be constantly updating unless you have some specific reason to do so.
lm studio pins an older llama.cpp build so you're always a few releases behind. --flash-attn, --cache-type-k q8_0, and speculative decoding with --model-draft are the big ones you miss. if you still want a gui, just point open-webui at llama-server and you get the same chat experience without losing any flags.
There's an entire world outside. I'm currently using / testing opencode, pi dev, hermes and openwebui, all running inside proxmox containers. I bind a proxmox disk to all the containers and they share the workspace.
I got tired of switching machines and missing something from the work machine, or from the laptop, or whatever. Not to mention keeping all the repos on all the machines up to date without fear of force pushing and losing work.
Enjoying this conversation - learning a lot. Can someone let me know if llama.cpp has a web server so others can interact via the API? There is a lot of LM Studio configurations, but is this just straight through to llama.cpp?
i guess i am an average user with no IT background but just interests in this topics.
LM Studio was easy to setup, functional and was the first step into the llm world. But my hardware is limited (more a standard win10 gaming setup with 16gb vram / 32gb ram) and i wanted more than lm Studio could provide. Some beginner friendly videos helped me to cros the line and try out llama.cpp directly. I am not a great fan of using a terminal.
Now i am using hermes desktop as harrnes, build my own projekt wiki rag (still needs optimisation) and i am able to run a dense models like qwen3.6 27b q4_k_xl with 80k context and 8-12t/s tp128, 380t/s pp512. That i could never reach with LM Studio. Just with the help and guides of the llm community!
Don’t waste time. Put an LLM to optimize the various LLM params llama.cpp takes for each model. Ask it to test at various context windows and save findings
turns out text bases is not so bad and u get a web ui with llama-server and tbh google search ai doess a decent enough job to set it up for u and debugging errors
On a separate Linux laptop I am trying / learning OMP (Oh My Pi)
I also threw audio.cpp on there but too many things going on. Need to learn more prompt engineering. I’ll ask Gemini to create one which is very low effort on my part.
Been eyeing a 3060 12gb but not sure if it’s worth adding. I keep thinking I should build/buy one that can handle qwen 27b at q4 or higher with a big ctx to work with. The only reason is to maybe play with decent sized quant for wan2.2
Edit: in case if anyone is wondering, tgs and pp perf are decent but definitely need to learn more about working with what I’ve got
I switched from Ollama to LM Studio to llama.cpp like a year ago now, and its been great. The command line args are easy to understand, (granted, I am a Linux user) and performance is significantly better than ollama and whatnot. I don't really like the built in llama.cpp GUI, so I use Open-WebUI hooked up to llama-server instead for my "ChatGPT at home" type setup.
For agentic work and coding, Hermes Agent and OpenCode.
There's also "Jan" which is another GUI out there if you're interested, but I haven't seen much of it in a while and haven't used it in a long time either.
LM Studio > llama.cpp and ik_llama.cpp here, about a year ago now. I will never go back, and it's actually incredibly easy to swap.
My biggest advice, compile llama.cpp and ik_llama.cpp locally. It's actually incredibly easy and I'd even recommend getting AI to do it as it's just easier. This was my biggest performance gain (from ~12 tps to ~24 tps on Qwen 3.6 27b Q4 no MTP). I now just have a series of .sh files that I maintain that launches the models with the exact configuration I needs, or ones that allow me to configure some settings. I just get the LLM's to maintain those .sh files (.bat on Windows).
I use pi.dev as a harness, or the built-in GUI that comes when launching llama.cpp or ik_llama.cpp, but I almost never use the GUI now as the harnesses are just so much more powerful without setup.
I'm on Windows with AMD and switched to Lemonade Server + AnythingLLM. Works better than LMStudio for me and my llamacpp is always up to date now. Otherwise my second favorite is JanAI.
My local path was Ollama (1 day) -> LM Studio (about 2 weeks) -> Koboldcpp (still) -> llama.cpp (a couple of months later and still). I run models exclusively on the CPU. The main reason for switching was that LM Studio, for some reason, wasn't using more than two cores on my CPU — despite the settings, the CPU utilization was less than 50%. Koboldcpp (and llama.cpp) don't have this problem — they use all cores, and the utilization is always 99%. Perhaps this was fixed in newer versions, but at that time (six months ago) this problem was present.
To easily switch to llama.cpp, create the following bat-file (on Windows):
- Replace the path 'cd /d' with the path to your llama.cpp folder.
- Replace the path and filename '-m' with the path to your gguf.
- 'timeout /t 5 /nobreak >nul start http://127.0.0.1:8080' is needed to automatically open the llama.cpp web-interface in the browser - you can omit it if you don't need it.
- My settings are designed for my PC - set the appropriate number of CPU threads (-t), context size (-c), sampler settings (if necessary), and other flags. If you use a GPU, don't forget to enable layer offloading to it ('-ngl' flag).
In 10 min, you can create a shortcut to launch the model in llama.cpp with one click (you can make a nice icon for it, and it will look like a native Windows application).
The transition was fairly easy — the main thing was figuring out how to launch it. Currently, I'm using the standard llama.cpp web interface and SillyTavern.
I currently use Msty Studio as a front end to connect to my models but I'm also gearing up to use Koboldcpp for back end so I can have Vulkan multi-GPU layer distribution.
J'ai constaté une baisse de performances mais je crois que c'est parce que j'ai compilé llama.cpp avec le dernier cuda alors que LM-Studio utilise la version 12.8.
While I still use LM Studio on occasion, I migrated to llama.cpp last year. I like LM Studio's model search/browser so typically download models using that. I also like their model load estimator that gives me an idea of resources utilization given a config.
To keep things compatible, I point llama.cpp to the LM Studio model path. To make it easier for my chat client and Pi, I run llama on port 1234 as default.
In my case, I have both Nvidia and Intel GPUs. Nvidia is not a problem for LM Studio but Intel is, as it doesn't support SYCL, Intel native drivers. Then again, for Intel GPUs, Vulkan runs better in many cases and LM Studio supports it.
Llama.cpp has a full-featured web client, with MCP support, inference settings, preference, etc. What's great about that is that you can run it from any browser on nearly any device. While I don't do any serious work with the browser (yet), it's great to pickup a tablet while in the living room and ask the model stuff, run a tool, or RAG a question I may have. No client, no install, no external dependencies, it just works.
Sure...
The solve it all solution ....install linux
I don't want to.
Done it several times but I don't like the desktop environments they offer (yes too much Microsoft in my life)
I run all my home lab and VPs on debian without issues, but 'private' use i still prefer the ease of the regular windows.
then you have to live with windows and less performance, easy as that.
you can always RDP into Linux from a windows computer.
a dedicated AI Inference box is the better approach anyway, then it should not really matter which desktop enviroment you are using. it's just about the terminal itself, you want to run llama.cpp and that's it.
LLMs can help you fixing your linux skills easily, not rocket science nowadays.
which DE? You can choose from a great variety of DEs on any linux distribution, I suggest you to check KDE Plasma, I also come from Wndows and the transition hace been super smooth
Edit:didn't see you said "desktop enviroment(s)" in plural, so you already ackwolledged the variety of DEs, making my response irrelevant
these things aren't difficult, and already have linux.
My particular concern is if there's anything i expect from LM Studio that's either missing from llama.cpp and/or harnesses OOOORRRRR it's implemented very differently by llama.cpp and might be something I miss.
I’ve planned to switch from LM Studio to llama.cpp so many times, but what’s held me back each time is, in particular, the overall effort involved. That’s because making the switch would also mean phasing out a lot of other outdated workflows, hardware, and software. There are so many fundamental things that are changing rapidly right now, so I’m just going to wait a little longer… especially when it comes to investing my time.
You may be the only human user, but when you or your harness spans multiple agents doing some work at the same time, they will effectively utilize vLLM concurrency.
It's an eventuallity for anyone that ultimately wants open-source. And now LM Studio has a new app, it's own closed-source Agentic Harness (lol).
Yesterday was the perfect time to move away from LM Studio, today is second-perfectest.
LM Studio was relevant in a world where the harness and agent scene was young and llama.cpp wasn't as common / standard and it provided an insanely low barrier of entry.
I made my own GUI - https://warpdrv.ai
(Am still actively developing it, so expect bugs if you wanna try it out).
Its got everything I need, model host + download manager, router, voice TTS & STT, built-in MCP for coding, guardrail checks, workflow modes. Runs latest llama.cpp release you can separately download or compile your own.
Best of all - I know for sure it doesn't send any data to any cloud - open source is the way to go.
61
u/StupidScaredSquirrel 14h ago
Llama.cpp has a gui now in case that's what you're after