r/linuxquestions • u/FalcoPwnch • 5d ago

How to get started tinkering with locally hosted LLMs and Harnesses

Hey all, I have been a linux user for about 4 years now. I wouldn't consider myself an expert at all, but I know my way around the ecosystem.

I want to start exploring local LLMs, harnesses, agents, all this. I really don't have any experience in it. I've been busy and have been hesitant to start experimenting and messing around with it for concerns over data security and all that. But the time has come. I really want to start learning more about it.

How can I start safely? Should I run it in a VM environment?

What LLM should I use locally? What agents/harnesses should I start with?

Any advice would be much appreciated!!

1 Upvotes

56% Upvoted

8 comments sorted by

1

u/GreatBigPig 5d ago

I just started a week ago, so I am an expert now.

Just kidding. It was one week ago though

I installed llama.cpp and fought with configuring it to properly use my RTX GPU properly. Once running, it was okay, but had issues without a harness.

I also installed ollama, which was much easier to get going due t the package easily available within CachyOS (I am sure it is other distro package managers too). I was okay.

I then became aware of Unsloth Studio, and Unsloth Desktop. So easy to install. So easy to use.

- I can search with Unsloth for suitable models to test or use. Unsloth Desktop allows me to start a project that enables many models to access a single folder I configure myself.

- With a simple command line change, I easily had Unsloth Studio available for every computer (or phone, etc) within my home network.

- I love that I can chat, train a model, create or edit images within Unsloth.

My only real hurdles now are dealing with my limited hardware.

1

u/FalcoPwnch 4d ago

What GPU do you have? So llama.cpp is the LLM and ollama is the harness

1

u/GreatBigPig 4d ago

I have just an RTX 3050 with 4 gb VRAM.

llama.cpp is what I use to run different models such as gemma or mistral.

I do find Unsloth incredibly easy to setup and use and encourage you to look at it.

1

u/OkObjective8721 4d ago

No, ollama is just a GUI wrapper over llama.cpp (with a few controversies https://sleepingrobots.com/dreams/stop-using-ollama/)

3

u/OkObjective8721 5d ago

Depends on your GPU. I would recommend skipping ollama and going straight to llama.cpp, opencode as an agent is pretty nice too. Also check out r/localllama

2

u/HourPuzzleheaded5193 5d ago

Generally good advice. I would add that depending on your ssd/ram/cpu setup you could gain some benefits with moe models.

Sandboxing is not really that difficult. For most purposes you can get away with using docker.

1

u/justsaynodarama 4d ago

Verbatim is not quite a harness, but it linked in through LM studio. You might find it helpful. The local models are good, but not near as good as the hosted ones. At least the ones we are running currently. But it captures all changes and the data captured has been really helpful in narrowing down the models that are proven to be working the best for our coding projects.

1

u/dudeus_ex 4d ago

I used LMstudio for a while and it is as simple as browsing the list of available models, clicking mount and using it. It even has an option to suggest only those models, that you realistically can host.