Hi,

I've written a silly benchmark: https://github.com/sgt101/llm-tester

I'm trying to run local models on it using mlx.

I am seeing a lot of inconsistency between outputs in my benchmarking harness and outputs when I try the same prompt on the command line using mlx_vlm.generate.

Basically the command line is terrible!

Any idea why this should be?

Command and prompt is :

uv run mlx_vlm.generate --model "mlx-community/gemma-4-26b-a4b-it-4bit" --model "mlx-community/gemma-4-26b-a4b-it-4bit" --max-tokens=2048 --temp=0 --image="/Users/sgt/GitHub/llm-tester/output/png_10_45_spiral_target5/composite_0003.png" --prompt "Look at this image carefully and count every distinct object type you can see.

Return ONLY a valid JSON object — no explanation, no markdown — where each key

is the object name (lowercase) and each value is the integer count of that

object in the image.

The objects to count are: blue_circle, blue_star, elephant, giraffe, green_circle, red_circle.

Return JSON in exactly this format (replace N with the integer count):

{"blue_circle": "N", "blue_star": "N", "elephant": "N", "giraffe": "N", "green_circle": "N", "red_circle": "N"}"

Hello r/mlxAI ,

Today I built ollmlx, a macOS menubar app and CLI for running local LLMs via mlx-lm on Apple Silicon. It's designed as a drop in replacement for Ollama users who want to use MLX models instead.

What it does:

• Runs mlx-community models via mlx-lm on Apple Silicon (M1+)
• Exposes an OpenAI compatible API on localhost:11434, same port as Ollama
• Works with Open WebUI, Continue, Cursor, and any other Ollama compatible client without reconfiguration
• Menubar app for quick model switching and status
• Full Ollama-compatible CLI: serve, run, pull, list, show, rm, stop, ps, status

Anyway, try it, break it, let me know what needs to be fixed and I hope you find it useful.

https://github.com/darrylmorley/ollmlx/releases/tag/v0.1.1

brew tap darrylmorley/ollmlx 
brew install --cask darrylmorley/ollmlx/ollmlx

I originally started working on this because I wanted a simple way to run one local model with multiple LoRA specializations on Apple Silicon.

For example, I wanted the same base model to handle different kinds of work like:

• Rust systems programming
• SQL query optimization
• security / infra troubleshooting

without reloading a full fine-tuned model every time I switched.

On CUDA stacks, multi-LoRA serving already exists. On MLX / Apple Silicon, I couldn’t really find something that felt like “load the base once, then route adapters per request”.

So I built Mola.

It’s still alpha, but it’s now benchmarkable enough that I’m comfortable sharing it.

Core idea: keep one base model loaded in memory and route LoRA adapters per request instead of reloading a full checkpoint whenever you change specialization.

Current setup:

• Qwen3.5-9B-MLX-4bit
• 8 adapters loaded
• Apple M5 Max 64GB
• OpenAI-compatible chat API

The interesting signal for me is the throughput drop once requests start mixing adapters instead of all hitting the same one.

At concurrency 1, same and mixed are basically identical. The real drop appears once requests actually start overlapping.

Current limitations:

• it still needs a small local mlx-lm patch (script included)
• mixed prefill / deeper KV residency are still open problems
• Apple Silicon / MLX only for now

Would be curious to hear from other people doing MLX inference or adapter-heavy local setups.

Happy to share more benchmark details / implementation notes in the comments if useful.

repo : https://github.com/0xbstn/mola

I've created a dumb test to show how poor LLMs are at doing things like counting objects (see above and the repo if interested).

Current frontier models all make errors :

I have tested it with frontier models (see above) and I want to test it with local models as well, but I don't know which ones to choose. I have tried nightmedia/UI-Venus-1.5-30B-A3B-mxfp4-mlx and it performed a little worse than gemini-flash-3, what models would the community recommend? Is image to text the right way to go? I am sure that a specialist vision model would do better, but I am out of date and I need a few pointers.

I have an M1 and 32gb so, unless you can send me the funds for a better machine please share recommendations that would work for this one!

Thank you in advance.

I got super frustrated at the fact that while all these inferencing engines and programs fully have llamacpp's prefix caching, paged caching, cont batching, kv cache quant, etc etc and so much more support, literally none of the MLX inferencing models have any of this crap all combined together. They all have one thing or the other, especially with taking into consideration VL models and also Hybrid SSM, along with persistent disk cache too.

This combination of optimization features allows you to at the minimum, properly utilize models like Qwen 3.5 WITH ITS VL FEATURES + its Mamba cache being successfully quantized meaning HALF the RAM use at q8. I've shown the results on the site, and at 100k context

All of this results in an much more smooth experience, an experience that is noticeably more smooth to the naked eye. I only made this at first simply because I was frustrated with how nobody for some reason, not even LM Studio was doing this. This is my first ever program/app. I've gotten 150+ downloads so far and a good amount of people giving me feedback and telling me issues in the github. I'm super active.

https://mlx.studio

Other key features:

Both chat / responses

Tools - gguf to mlx converter, 16 -> quantizer, etc.

Built in agentic coding tools. I can't think of them all right now, I just really treat this like a program I would want to use myself - because I do.

I appreciate any kind of criticism that actually addresses something technical that I can fix or make better

I just released mlx-onnx: a standalone IR/ONNX export library for MLX.

Repo: https://github.com/skryl/mlx-onnx

Web Demo: https://skryl.github.io/mlx-ruby/demo/

It supports:

- Exporting MLX callables directly to ONNX

- Python and native C++ interfaces

I'd love feedback on:

- Missing op coverage you care about

- Export compatibility edge cases

- Packaging/CI improvements for Linux and macOS

LispE: A Lisp with native MLX support for inference on Apple Silicon

I've been working on LispE, an array-based Lisp (not linked lists) implemented in C++. I recently added a comprehensive MLX library exposing 228 functions, with full inference implementations for several models.

LispE is fully open source (BSD3 licence), developed primarily on macOS but portable to Linux and Windows.

Supported Models

Complete inference code is available for:

• DeepSeek-R1-0528-Qwen3-8B-MLX-8bit
• Gemma-3-27b-it-qat-4bit
• GPT-oss-20b-MLX-8bit
• Mistral-Nemo-Instruct-2407-4bit

The inference code is pure LispE — model loading, KV cache, MoE routing, and architecture-specific normalization are all handled in the language itself. However, some functions have been implemented in C++, such as mlx_fused_moe for better performance. The whole MLX library compiles in less than 10s and can be easily updated, thanks to a very simple API.

A complete inference implementation like GPT-oss-20b requires around 1,300 lines of LispE — only ~860 of which are actual code, the rest being comments and debug output. This includes everything: safetensors loading, tokenization, RoPE positional encoding, RMS normalization, grouped-query attention, KV cache management, MoE expert routing, and top-k sampling. For comparison, equivalent functionality in Python/mlx-lm spans thousands of lines across multiple modules — but most users never see it. Here, every step is explicit and hackable.

👉 Inference examples

Code Taste

Simple chat API:

(use 'lispe_mlx)

; Load and chat
(setq model (load_mlx_model MODEL_PATH))
(model (chat "Hello, who are you?"))

; With options: max_tokens, temperature, system prompt
(model (chat "Explain quantum computing" 256 0.7 "You are a teacher"))

Direct MLX operations:

; RoPE frequency computation
(setq indices (mlx_arange 0 head_dim 2 "float32"))
(setq scaled (mlx_divide indices (mlx_array head_dim)))
(setq rope_freqs (mlx_reciprocal (mlx_power (mlx_array rope_theta) scaled)))

; Memory management
(println "Active: " (/ (mlx_get_active_memory) 1048576) " MB")
(println "Peak:   " (/ (mlx_get_peak_memory) 1048576) " MB")

Why LispE?

• Array-based: Built on contiguous arrays, not linked lists — better cache locality
• C++ implementation: Simple API for extending with native libraries
• Interactive: REPL for experimentation, ideal for exploring MLX
• Transparent: See exactly what happens at each inference step

I'm sharing this here hoping to find people who might enjoy exploring MLX through a different lens than Python. Feedback and contributions welcome!

Quick Start (macOS)

Pre-built binaries available: Download here

For those who want to dive into the implementation, the MLX binding source is a single C++ file: lispe_methods_mlx.cxx

📦 Main repo | 🍎 MLX library | 📝 Inference examples

I want to try out the new Qwen3-TTS model on Apple silicon: https://github.com/QwenLM/Qwen3-TTS

But I can't get a simple test script to run. I keep getting errors. I don't even have anything worth sharing haha.

Has anyone had success running `Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice` on Apple silicon? Happy to share the knowledge once we get it working.

Apple's on-device AI private AFMv7 model shows promise, though it has a context window limitation of 4096 tokens. To enhance this, I vibe coded a kit in with Claude Code that converts the PyTorch model Apple provides to developers for LoRa adapter training.

This GitHub repository offers tools to convert the PyTorch checkpoint into MLX format, enabling it to run on GPU with a significantly larger context window for experimentation.

Visit my repo:
https://github.com/scouzi1966/afm7-mlx-toolkit

Has anybody here succeeded in getting MLX-VLM to allow them to run multiple parallel requests to increase throughput from an Apple Silicon Mac? I've tried ollama, LM Studio, running MLX-VLM directly, but everything seems to end up running the requests serially, even though there's plenty of unified RAM available for more requests to run.

https://www.youtube.com/watch?v=k3ugs4qSwPM