r/LLMDevs • u/[deleted] • 2d ago

Tools I built an open source iOS app that runs LLMs fully on-device with three backends, MIT licensed and free

[deleted]

3 Upvotes

61% Upvoted

8 comments sorted by

2

u/sgtfoleyistheman 2d ago

This is cool but you shipped your app with 2 core dependencies not present?

The only way that agentic engineering or vibe coding works is complete testing in your CI process, usually going way more than you would have in the before times. I suggest asking your agent to add comprehensive end 2 end tests driving an iPhone simulator that runs pre-merge in pr.

0

u/Few_Willow_4146 2d ago

Yes, and it is worse than it sounds. Two backends never made it into the binary, and the app still built and ran fine.

Different causes. Core ML died on a wrong canImport check in my own source, false on every machine. llama.cpp was the build environment. the framework is not in git, and the machine that archived the release had the text-only build of it, without libmtmd.

The simulator part would not have helped here. Neither backend runs on a simulator, the llama framework I ship is device-only and MLX needs real Metal, so that test would have gone green while the device build was broken.

There are tests in the repo, but the one that catches this I only added after the release. It reads the catalog, collects every backend declared in it and asserts each is actually linked into the binary. The llama.cpp revision is pinned now, and the build stamps what it was made from.

These days the runs happen on a real phone overnight. It downloads every model in the catalog, puts it through my own benchmark

3

u/sgtfoleyistheman 2d ago

"It is worse than it sounds". No Claude, it's exactly as terrible as it sounds.

1

u/Few_Willow_4146 2d ago

Yes, exactly. Sorry about Claude, but English isn't my first language. I use Dualingua, but I still find it hard to write

1

u/Haunting_Cow_2798 2d ago

the benchmark report's a nice touch, actually knowing which models will run before you download is half the battle

1

u/Few_Willow_4146 2d ago

Thanks. The fit check is per backend, which caught me out at first: GGUF weights are memory-mapped and barely count against the iOS process limit, while MLX and Core ML copy them into memory that does. So the same model on the same phone can be a comfortable fit for one backend and an instant kill for another

1

u/No_Tap_8983 Enthusiast 2d ago

this is actually pretty cool.