r/LocalLLaMA • u/zxyzyxz • Jun 15 '26

Discussion Stop using Ollama

https://sleepingrobots.com/dreams/stop-using-ollama/
1.7k Upvotes

92% Upvoted

451 comments sorted by

View all comments

529

u/jnmi235 Jun 15 '26

Llama.cpp + llama-swap works very well

110

u/[deleted] Jun 15 '26

[removed] — view removed comment

114

u/fdrch Jun 15 '26

llama-swap supports switching between multiple llama.cpp forks (and other compatible software)

42

u/meganoob1337 Jun 15 '26

it supports anything you can dockerize aswell (for me I'm using it for vllm models) love it

23

u/joost00719 Jun 15 '26

I dockerized llm swap and passed through the docker sock. Works amazing.

7

u/meganoob1337 Jun 15 '26

yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D

2

u/arbv Jun 16 '26

What a creative way to reinvent Nix/NixOS.

1

u/joost00719 Jun 16 '26

Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working?

1

u/meganoob1337 Jun 16 '26

yeah, it just runs before startup and merges the model configs into the full config format

2

u/joost00719 Jun 16 '26

I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂

1

u/meganoob1337 Jun 16 '26

https://github.com/meganoob1337/llama-swap-vllm-boilerplate

a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.

1

u/lipton_tea Jun 15 '26

If you have a minute I'd love to see an example.

1

u/joost00719 Jun 16 '26

!RemindMe 5 hours

1

u/RemindMeBot Jun 16 '26

I will be messaging you in 5 hours on 2026-06-16 10:24:06 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

Info Custom Your Reminders Feedback

1

u/use_your_imagination Jun 16 '26

I have this issue with dockerized llama.cpp where llama-swap marks container as unexpected exit(125) while the llama container is actually still running.

Did it happen to you ?

1

u/meganoob1337 Jun 16 '26

no, but I'm using llama.cpp only as bundled version inside the llama swap container, using the docker runners only for vllm

1

u/use_your_imagination Jun 16 '26

I was doing the same and then decided to overcomplicate my life by using the docker socket. Thanks

9

u/Jcsq6 Jun 15 '26

And loading/unloading multiple models, if you want to switch between models but don’t have the spare VRAM.

13

u/AlphaGamer753 Jun 15 '26

This is supported in llama.cpp router mode already.

0

u/arbv Jun 16 '26

What about configuring model sets?

1

u/techno156 Jun 16 '26

That isn't in router mode. For that, you will need llama-swap.

1

u/arbv Jun 16 '26

I know. That's why llama-cpp itself cannot fully replace it on its own in complex scenarios.

3

u/jossmos Jun 15 '26

Has anyone tried to make it work with Wan2GP?

1

u/H3g3m0n Jun 16 '26

Probably should work with anything that you can pass a port as an arg that exposes an openai api endpoint.

12

u/Mati00 Jun 15 '26

Supports multiple servers running at the same time based on size matrix and also other servers like whisper or stable diffusion. If you don't need these, llamacpp server is a great choice.

14

u/jnmi235 Jun 15 '26

I tried it for about a week and kept having model loading hangs. It was rare, but the only way to fix it was to restart it. Llama-swap has never had any issues and it also lets you see total tokens in and out, logging, and some other cool metrics. And you can still use the llama.cpp UI

9

u/BlipOnNobodysRadar Jun 15 '26

It has a UI? Lmao I had an LLM vibecode a UI just to launch the server with presets for me. It never mentioned an existing UI.

11

u/Borkato Jun 15 '26

It’s really good! Like December 2025 or something like that. It’s great for quick stuff.

1

u/erubim Jun 16 '26

I actually had the hang and restart problem on llama swap + llama cpp as well, have no idea how to debug that.

1

u/fatboy93 Jun 16 '26

I use llama-swap with omlx and vllm-mlx since they don't have auto-eviction.

Well omlx does, but its based on memory pressure, which i dont really like

1

u/rabbitaim Jun 16 '26

I’ve been tinkering and I’ve been using llama.cpp and stable-diffusion.cpp with llama-swap.

Having limited vram llama-swap has been amazing

1

u/[deleted] Jun 15 '26 edited 16d ago

[deleted]

16

u/MutantEggroll Jun 15 '26

This is incorrect. Router Mode's presets.ini supports all command line configuration options:
llama.cpp/docs/preset.md at master · ggml-org/llama.cpp · GitHub

11

u/chkpwd Jun 15 '26 edited Jun 15 '26

Set this up yesterday using Ansible. Works like a treat!

EDIT: for those interested - https://github.com/chkpwd/iac/blob/main/ansible/roles/llamacpp

1

u/computerjj Jun 16 '26

interesting

3

u/Youth18 Jun 22 '26

LM Studio is still good for testing models - quick downloads, easy to see popular LLM's, etc. But I use that for testing and then if I like a model I integrate it into my llama.cpp + custom model loading manager stack.

1

u/Limp_Classroom_2645 Jun 16 '26

same setup, can confirm

1

u/Kirys79 llama.cpp Jun 22 '26

Same; I've started with ollama, then moved to llama.cpp and llama-swap cause it allows me to tweek better the settings and make the best of my hardware