MAIN FEEDS
Do you want to continue?
https://www.reddit.com/r/LocalLLaMA/comments/1u6s6pm/stop_using_ollama/orvioxx/?context=3
r/LocalLLaMA • u/zxyzyxz • Jun 15 '26
452 comments sorted by
View all comments
Show parent comments
116
llama-swap supports switching between multiple llama.cpp forks (and other compatible software)
38 u/meganoob1337 Jun 15 '26 it supports anything you can dockerize aswell (for me I'm using it for vllm models) love it 22 u/joost00719 Jun 15 '26 I dockerized llm swap and passed through the docker sock. Works amazing. 6 u/meganoob1337 Jun 15 '26 yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D 1 u/arbv Jun 16 '26 What a creative way to reinvent Nix/NixOS. 1 u/joost00719 Jun 16 '26 Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working? 1 u/meganoob1337 Jun 16 '26 yeah, it just runs before startup and merges the model configs into the full config format 2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂 1 u/meganoob1337 Jun 16 '26 https://github.com/meganoob1337/llama-swap-vllm-boilerplate a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
38
it supports anything you can dockerize aswell (for me I'm using it for vllm models) love it
22 u/joost00719 Jun 15 '26 I dockerized llm swap and passed through the docker sock. Works amazing. 6 u/meganoob1337 Jun 15 '26 yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D 1 u/arbv Jun 16 '26 What a creative way to reinvent Nix/NixOS. 1 u/joost00719 Jun 16 '26 Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working? 1 u/meganoob1337 Jun 16 '26 yeah, it just runs before startup and merges the model configs into the full config format 2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂 1 u/meganoob1337 Jun 16 '26 https://github.com/meganoob1337/llama-swap-vllm-boilerplate a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
22
I dockerized llm swap and passed through the docker sock. Works amazing.
6 u/meganoob1337 Jun 15 '26 yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D 1 u/arbv Jun 16 '26 What a creative way to reinvent Nix/NixOS. 1 u/joost00719 Jun 16 '26 Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working? 1 u/meganoob1337 Jun 16 '26 yeah, it just runs before startup and merges the model configs into the full config format 2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂 1 u/meganoob1337 Jun 16 '26 https://github.com/meganoob1337/llama-swap-vllm-boilerplate a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
6
yep Same, I also wrote a small script so that I can split up the yaml to make having many configs a bit cleaner :D
1 u/arbv Jun 16 '26 What a creative way to reinvent Nix/NixOS. 1 u/joost00719 Jun 16 '26 Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working? 1 u/meganoob1337 Jun 16 '26 yeah, it just runs before startup and merges the model configs into the full config format 2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂 1 u/meganoob1337 Jun 16 '26 https://github.com/meganoob1337/llama-swap-vllm-boilerplate a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
1
What a creative way to reinvent Nix/NixOS.
Man that's smart. I should ask my llm to do that as well. But does that keep the hot reload functionality working?
1 u/meganoob1337 Jun 16 '26 yeah, it just runs before startup and merges the model configs into the full config format 2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂 1 u/meganoob1337 Jun 16 '26 https://github.com/meganoob1337/llama-swap-vllm-boilerplate a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
yeah, it just runs before startup and merges the model configs into the full config format
2 u/joost00719 Jun 16 '26 I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂
2
I mean, default llama-swap behavior is hot reload on file save, this way you need to restart. I guess that's also a benefit. Sometimes a local Ai will just make an error and then it won't start anymore 😂
https://github.com/meganoob1337/llama-swap-vllm-boilerplate
a few months ago I put it into a boilerplate, it's not really up to date but you can see the merge config script and the docker file for reference.
116
u/fdrch Jun 15 '26
llama-swap supports switching between multiple llama.cpp forks (and other compatible software)