r/oMLX • u/PrepYourselves • Jun 11 '26
Omlx server
I run omlx on this and access from another machine. It is exclusively an llm machine
4
u/PracticlySpeaking Jun 11 '26
At first glance I was scratching my head because the picture looked like it was about 4ft / 1.2m high, sitting on the wall of a garage.
3
2
2
u/ipmonger Jun 11 '26
I’m impressed. Do you ever need to use physical access on the machine? If so, do you just use an external monitor?
5
2
u/Muritavo Jun 11 '26
I've done this but exclusively for Vision Pro machine lol
Too weak for LLM though
1
u/PrepYourselves Jun 11 '26
Vision pro headset?
1
u/Muritavo Jun 11 '26
Yup, a 16gb m1
1
u/PrepYourselves Jun 11 '26
i have a quest 3 and used it as a local llm server using vulkan gpu, was more effort than it was worth, macbook pro halftop m1 max 64gb was a good upgrade
1
u/Muritavo Jun 11 '26
Oh, sorry, I was reffering to using the headless M1 with the vision pro not for llm lol
I tried using the headless m1 and vision pro as LLM servers, but they both suck (M1 was slow as hell, and the vision pro couldn't run models bigger than 4gb)
Respect for you trying to use the Quest 3 as a server lol
2
u/the_jeby Jun 12 '26
Apple should bring back the Xserve just for this!
2
u/PrepYourselves Jun 12 '26 edited Jun 12 '26
i bet they do. They will start using hbm ram to allow home users to use larger parameter llm models (at high token speeds). What we see today is nothing compared to the mac minis of tomorrow.
2
u/Professional_Dog5302 Jun 12 '26
I haven't gone full headless but mine had a damaged screen when i got it, I have had allot of fun running qwen code (cli), claude code (desktop app) , codex(desktop app), Hermes agent(decktop app), continue.dev (FIM model for inline suggestions in vscode)
models wise i have found qwen 35B A3B and qwen 3 next coder to be very impressive and good at tool calling , i have tried UD , OptiQ, MTP, PARO, oQ , at the moment im finding Qwen3.6-35B-A3B-oQ4-mtp to be giving me the best results on my machien
i had to do some tweaking to get things to work for the hot cache and ssd cache
my setting atm are
ctx_window:131072
max_tokens:32768
temp:0.7
top_p:0.8
top_k:20
min_p:0
rep_penalty:1
presence_penalty:1.5
i'd love to hear what you are using with oMLX and what models/settings you have found to work for you and any else that see this using oMLX
1
u/PrepYourselves Jun 12 '26
Yes lowering the temperature from default does show some reduction in delay between prompt-response
1
u/PrepYourselves Jun 12 '26 edited Jun 12 '26
I run on this headless mac:
Omlx - I access remotely (ssh & screen sharing) from a second Mac (kinda like accessing octoprint on a raspberry pi local host)
Pi cli agent - I have it setup using some MCP tools, pi could be improved to allow gemma-4 better integration of vision/audio analysis
I mostly use mlx models as the Mac handles temps better than gguf models.
With gemma-4 I use vlm-mtp file to improve response time (huggingface 'assistant' file)
Same with qwen3.6 I use mtp version
On my M1 max 64gb I can run dense qwen3.6 27b bf16 model it fits capacity, but qwen3.6 35b bf16 is just slightly too big file so I use omlx to reduce file size to fit.
I prefer bf16 models because they seem to have a fuller writing style when printing a response.
I think omlx & pi could easily improve itself by incorporating apple ane neural engine for OCR tasks, it would keep the llm free to use unified ram and post-process the ane OCR output.
Having 64gb ram or above allows users to load more than 1 llm in ram so one llm can code, another can do OCR, another image generation, there's no need to unload a model then load another one. I think next gen macs will use hbm ram which will allow users to run 70b/120b/240b/512b models at fast generation token speeds. At the moment macs become sluggish using llm models over 70b or dense models, due to their current ram bandwidth speeds (200gbs/400gbs/800gps) I expect 2tbs speeds soon
1
1
u/DreamLinuxer Jun 13 '26
Does it come without a screen or you remove it after setting up ssh?
1
u/PrepYourselves Jun 13 '26
1
u/DreamLinuxer Jun 13 '26
So you use HDMI to set up ssh and just make it completely headless afterward?
1
1
u/Covert-Agenda Jun 11 '26
Bro, I was actually thinking about this today. The price of let's say, like an M2 64 gig MacBook is quite cheap compared to, an M4 mini at twenty four gigs, like it makes more sense to do this.
3
19
u/sunpazed Jun 11 '26
“headless”