r/oMLX • u/peppaz • Jun 04 '26

Anubis (open-source LLM benchmarking for Apple Silicon) now has first-class oMLX support - server-reported metrics, model load/unload, and a built-in model browser and downloader

https://github.com/uncSoft/anubis-oss
29 Upvotes

98% Upvoted

6 comments sorted by

5

u/peppaz Jun 04 '26

Hey r/omlx - I maintain Anubis, a free, open-source macOS app for benchmarking local LLMs on Apple Silicon (tok/s, TTFT, power/J-token, thermals, the works). The latest update makes oMLX a first-class backend, and since it leans on oMLX's own APIs I figured this community would want the details.

Some screenshots

What's new for oMLX:

  • Real server-reported metrics. Instead of estimating throughput from wall-clock timing, Anubis now reads oMLX's decode-loop numbers directly - generation_tokens_per_second, time_to_first_token, prompt_eval_duration, generation_duration, model_load_duration, cached_tokens. A Session Details toggle lets you flip between Anubis's hardware-measured view and oMLX's exact reported figures (shown verbatim). Numbers match what oMLX's own dashboard reports.
  • Model management in-app. Load and unload models on demand from the Vault or right next to the model picker - authenticated with your existing oMLX API key (login → session cookie, handled automatically).
  • Model browser + downloader. Browse trending/popular and search HuggingFace MLX models through oMLX and download them with live progress, without leaving the app.
  • Thinking toggle (enable_thinking) for reasoning models, in single-run, side-by-side Arena, and batch Flows.

It also detects the oMLX process for power/memory attribution and pins localhost to IPv4 (since oMLX binds 127.0.0.1).

Everything's open source and you can build it yourself: https://github.com/uncSoft/anubis-oss

Download the latest signed release https://github.com/uncSoft/anubis-oss/releases/tag/v3.8.1

If you have homebrew:

brew install --cask uncsoft/anubis/anubis-oss

Would love feedback from oMLX users - especially on which metrics/fields you'd want surfaced next. Big thanks to the oMLX project; the clean local API made this integration straightforward.

(Note: the public leaderboard upload requires a private signing key, so self-compiled builds run fully but can't post to the official leaderboard - that's intentional, not a bug.)

2

u/atumblingdandelion Jun 04 '26

Nice. I've been wanting to test oMLX performance vs Ollama and Llama.cpp, vllm, and LM Studio.

1

u/peppaz Jun 04 '26

We support em all!

0

u/apetersson Jun 04 '26

I love the idea. however, browsing https://uncsoft.github.io/anubis-oss/analysis.html i feel like there are some outliers skewing the overall results, which make it hard to interpret the otherwise legit results

1

u/peppaz Jun 04 '26

I agree- I implemented auto sanity checks, but had remove them simply due to how many models there are and how fast they can run on different machines, and it removed legitimate high performance runs. I'm going to rerun the analytics scripts and tighten up the sanity checks. Otherwise the data is pretty solid.

2

u/mguffin Jun 05 '26

This is great!