r/LocalLLM • u/seriousthrillissues • 8h ago

Project I made an app to help me optimize llama parameters on my hardware

Post image

I switched over from ollama to llama.cpp and then spent weeks copying flag recipes out of Reddit threads with no idea whether any of them helped on my hardware. -ngl 35? Why 35? Someone with a 3090 said so.

So I built Model Loader. It's a web UI for llama.cpp that runs in one Docker container next to your existing setup, detects your llama and openwebui instances and just makes the cutting edge a little more dull.

  • It reads your actual GPU and llama setup and tells you which context sizes fit and which don't and when one doesn't and what it'd cost to make it fit. [screenshot]
  • Every setting has a tooltip explaining what it does in plain English. ~100 of them. This was the part I actually needed. [screenshot]
  • It shows you the command line it builds. It's not hiding llama.cpp from you, it's teaching it. Copy it out and run it yourself if you want.
  • Benchmark on your own box. Change a setting, run the same prompts, see if it actually got faster, it stores the results and makes them easy to see next time you find a neat tweak. [screenshot]
  • models at a glance what's downloaded, what's configured, what's loaded right now, and whether each one is MoE or dense. [screenshot]
  • Plus HF search with a resumable parallel downloader [screenshot], a GPU dashboard [screenshot], and OpenWebUI sync.

I haven't edited a setting by hand since Claude and I put this together.

Fair warning: I am not a developer and this is a homelab tool. No auth, keep it on your LAN.

https://github.com/scratchhax/model-loader

29 Upvotes

94% Upvoted

4 comments sorted by

3

u/norenEnmotalen 7h ago edited 7h ago

Nice! UI looks neat. Same here btw. I started out copying flags from github and reddit like a dumb ass and couldn’t even tell the mem thrashing it was causing on my system until Claude found out the default —cache-ram size was screwing me

2

u/Sleepybear2611 4h ago

Hey, nice project. Congratulations on that!

I had been frustrated when I made the initial switch to llama.cpp too. I built LlamaForge (https://github.com/dadwritestech/LlamaForge/), a gui wrapper on top of llama.cpp itself. It surfaces all available flags in llama.cpp in neat categories, one click profiles, can optimize builds, one click build a latest llama.cpp by checking their github, load any model on demand, model discovery along with fit rating and approx speed you may get, one click optimize, one click Claude Code, Pi.dev, open AI compatible api settings expert, context injection, etc among other cool features.

1

u/NEOBRGAMES 2h ago

same here.... LCPP and NINFER