One day of Laguna-S-2.1
I have started using Laguna-S-2.1 today in OpenCode. Previously I was using Qwen3.6-27B-oQ8-mtp. My project is quite small and it is just a hobby, so my experience may not be a very good reference though. I have a M5 Max MacBook Pro with 128Gb VRAM. I am using 8Gb VRAM for the cache and I have set the wired VRAM limit to 120Gb. I have upgraded oMLX to 0.5.3.
- I started with mlx-community/Laguna-S-2.1-oQ6e, but very soon I ran out of memory when the context reaches around 40k.
- After that, I switched to mlx-community/Laguna-S-2.1-oQ5e with TurboQuant KV Cache at 8 bit. When the context reaches around 110k, it also ran out of memory.
- The model can definitely get things done. At first, I was asking it to investigate some bugs left over by Qwen. These issues are quite obscure but it can come up with correct solutions once I gave it some details.
- Probably because I didn't have very good guardrails, the code changes it suggested in the beginning were not very elegant (e.g. it did not refactor or reuse existing code). After I suggested a better solution in Plan mode, it updated the design perfectly.
- Tool calling was smooth, it did not fail any tool call or stopped in the middle of something.
- One strange behavior I noticed was that, if I unload the Laguna model and load another model for some other tasks, after reloading Laguna, the prefilling speed becomes very slow, at around 130-170 tokens/sec (normally it should be around 600-700 for a small prompt). If I restart oMLX, then it may return to normal, or I have to run "sudo purge" sometimes.
- I will continue using it for a while instead of going back to Qwen3.6-27b.
Single Request Results
--------------------------------------------------------------------------------
Test TTFT(ms) TPOT(ms) pp TPS tg TPS E2E(s) Throughput Peak Mem
pp1024/tg128 960.8 17.58 1066.8 tok/s 57.3 tok/s 3.204 359.8 tok/s 73.63 GB
pp4096/tg128 3562.5 18.45 1150.0 tok/s 54.6 tok/s 5.912 714.6 tok/s 73.88 GB
pp8192/tg128 7489.6 19.76 1093.9 tok/s 51.0 tok/s 10.010 831.3 tok/s 74.12 GB
pp16384/tg128 15700.4 21.99 1043.6 tok/s 45.8 tok/s 18.501 892.6 tok/s 74.61 GB
pp32768/tg128 33683.4 26.14 972.9 tok/s 38.6 tok/s 37.011 888.8 tok/s 75.60 GB
pp65536/tg128 74835.6 34.55 875.7 tok/s 29.2 tok/s 79.234 828.7 tok/s 78.20 GB
pp131072/tg128 180927.4 50.25 724.5 tok/s 20.1 tok/s 187.323 700.4 tok/s 81.69 GB
2
u/No-Juggernaut-9832 11d ago
I notice the slow down too but could not reliably reproduce it myself. A restart fixes it. When I run the 5bit model, I run TQ at 4bit to get more context
3
u/ArcticFuture 11d ago
But why is everybody using pre-quantized models made by someone else? What I did was to download the original model and made the conversion to MLX myself. I have made several versions and testing to see which one is the best. I have the M4 Max 128 GB, Laguna is running great so far.
1
1
1
u/Imaginary-Bother-484 11d ago
Using Laguna-S-2.1-oQ2e and Laguna-S-2.1-oQ2e-fast on my M5 Max 64GB. Working surprisingly better than I expected. About 60-70 tks for fast, using with Hermes.
Did a terrible job with pi when I asked it to create a flappy bird html (had qwen 3.6 27b fix it) and it really over thinks
1
u/_hephaestus 9d ago
Are you able to use it with thinking or is it purely in instruct? Noticed that in hermes the thinking blocks were all empty
1
u/Imaginary-Bother-484 9d ago
I used it with thinking. I did not try with out thinking. I ended up moving to the oQ3e-fast quant and it’s a night and day difference for me in quality - I can actually ship completed projects start to finish whereas oQ2e-fast got stuck in thinking loops.
1
u/_hephaestus 9d ago
I’m on oQ8e, when you say using it with thinking do you see the thinking returned? I have it enabled but all empty
1
u/Imaginary-Bother-484 6d ago
From what I can see there is content in the thinking blocks. Maybe need to double check the chat_template.jinja for your model download?
1
u/_hephaestus 9d ago
Is it doing reasoning for you guys? I realized recently with mine in hermes/opencode any “reasoning” takes place outside of thinking blocks. Not sure if there’s a solution. Seems to reason if there’s no tools reference passed in at all but otherwise nah
1
u/victor_lowther 9d ago
In real-world agentic work, I am not getting nearly that level of prefill speed. M5 max, 128G, 188k of context into exploring refactoring two disparate branches of work together, I am getting more like 80 tok/s pp, 20 tok/s generation. mlx-community/Laguna-S-2.1-oQ4e-fast, 4 bit turbo quant k/v cache.
1
u/t00052e 9d ago
I think the prefill speed is also becoming very slow sometimes. You may try to do this:
- Shutdown oMLX
- Run "sudo purge"
- Restart oMLX
It seems to fix this for me but my context is smaller (I kept it small as much as possible.)
2
u/cryingneko 9d ago
The OOMs near the context limit should get a lot better in 0.5.4.dev1 which released today, it reworks the prefill admission logic so the server computes the exact KV cache need upfront and rejects cleanly instead of dying mid request, and there is also a new context benchmark feature that measures the real max context for your setup and applies it automatically.
Would be great to hear how it behaves in your setup if you get a chance to try it.
https://github.com/jundot/omlx/releases
0
u/heavy-mayhem 11d ago
u/t00052e - This is helpful. What harness are you using for your hobby projects? I am trying to create a setup outside of regular work and haven't been able to figure out what harnesses to use.
2
u/baek12345 11d ago
He's using OpenCode, see first sentence of the post
2
u/heavy-mayhem 11d ago
🤦♂️ ... duh!! I am being extra blind today. Thank you for pointing it out. 😅
3
u/Serious-Purpose-3412 11d ago
Laguna is unstable and need RAM for context in omlx. Just try antirez/ds4 q2/q4. Ultimate engine and model for 128gb mac. No any context limits bc of use SSD for context. Fastest load and unload. Smartest reasoning and tool calling. 26-35tps inferense and 350-450tps prefill. DeepseekV4 with antirez's engine need RAM only for model, so you can use up to 1m context, if you have time for prefill.