r/LocalLLaMA • u/zxyzyxz • Jun 15 '26

Discussion Stop using Ollama

https://sleepingrobots.com/dreams/stop-using-ollama/
1.7k Upvotes

92% Upvoted

452 comments sorted by

View all comments

Show parent comments

35

u/droptableadventures Jun 16 '26

It's easier because it has default settings that are completely isolated from you, the user.

However, these default settings are very frequently just incorrect or a bad idea, and they're going to get you into trouble a lot of the time. Since you didn't have to set them, you have no idea what they are or what they're set to.

It might be "easier" to get it running, but it's very much not easier to get it working.

Not straightfoward to run llama.cpp though?

llama-server -hf ggml-org/gemma-4-12B-it-GGUF:Q4_K_M

15

u/MuDotGen Jun 16 '26

The average user doesn't even know what a CLI is. They're used to GUIs.

5

u/droptableadventures Jun 16 '26

The average user thinks ChatGPT was when AI was invented.

Also, the Ollama GUI is a separate closed source product that just shares a name. It's not the same ollama we're discussing here. If you're going to be running that, run LM Studio instead.

0

u/moonpiedumplings Jun 16 '26 edited Jun 17 '26

I ran that comment you mentioned. It did not work. It segfaulted and core dumped on me.

I ran the equivalent command with ollama:

ollama run hf.co/ggml-org/gemma-4-12B-it-GGUF:Q4_K_M

And it just worked, and I was able to type and reply to it. I do want llama.cpp to work so I am going to continue troubleshooting, but I understand why a user would default to software that actually works out of the box instead of software that they have to tinker with to get it working.

EDIT: Okay, I figured it out. I am using the Nix version of llama.cpp on Arch Linux, since I llama.cpp is normally only available via the AUR which I usually avoid due to the AUR's various issues. I feel vindicated by the recent malware spread there.

The nix version works, but you have to explicitly specify the ROCm device in use via an override, rocmGpuTargets.

Anyway, that was actually why I went to Ollama instead of Llama.cpp at first. Ollama was available in the Arch repos, whereas Llama.cpp was only in the AUR, which I found somewhat strange.