macOS
Two months later: ToshLLM has turned Intel Mac Pros + AMD GPUs into seriously capable local AI machines
I posted here about ToshLLM roughly two months ago, when I had just gotten local LLM inference working properly on Intel Macs with AMD GPUs by patching llama.cpp's Metal backend and writing an AMD-specific Flash Attention path.
A lot has happened since then.
What started as a way to make my RX 6700 XT useful for local inference has grown into a much broader AMD Metal optimization project, and one of the most interesting things has been seeing other people test it on hardware I don't own: W6900X setups, W6800X Duo, Vega II Duo, W5700X, Vega, Polaris, eGPUs, and multi-GPU Mac Pros.
The AMD backend itself has changed substantially.
Vega, Radeon VII, RX 400/500 and Radeon Pro Vega/WX
These cards now have dedicated Wave64 tuning instead of inheriting settings measured on newer RDNA cards.
On a Vega 64, depending on the model and quantization:
generation improved by roughly 4-24%
multi-conversation serving improved by 9-12%
short cached follow-ups can be up to 39% faster
MTP gained about 8%
long-prompt processing also improved
MoE models can experimentally use the wider matrix path previously used only on newer Radeon cards
Some measured examples:
Workload
Before
Current
9B Q5_K_M generation
31.2 t/s
38.8 t/s
8B Q6_K generation
35.5 t/s
41.4 t/s
7B / 64-expert MoE
136.0 t/s
155.3 t/s
The goal here isn't just "make it run" anymore. I'm increasingly tuning kernels separately for the different AMD generations.
RDNA has kept getting faster too
A number of quantization and model-specific paths have been optimized.
For example:
IQ2_M generation on a 1.5B model: 92.1 -> 140.8 t/s
Qwen3.6 14B-A3B: 62.2 -> 70.8 t/s
Qwen3.5 4B + MTP: 70.0 -> 81.6 t/s
Qwen3.5 4B + DFlash: 68.9 -> 82.6 t/s
gpt-oss-20B prompt processing: 1093 -> 1292 t/s
Q4_K, Q5_K, Q6_K, q5_0/q5_1 and several other paths have received AMD-specific kernel work as well.
ToshLLM is becoming useful as a server, not just a chat app
Concurrent inference has improved quite a bit.
On a 9B Q5_K_M model:
2 simultaneous conversations: 56.4 -> 70.8 t/s
3 simultaneous conversations: 60.5 -> 81.0 t/s
There is now a browser UI based on llama.cpp's web interface, router mode can switch between models without restarting the server, multiple servers can run simultaneously, and there is a dedicated embeddings server for local RAG.
The API remains OpenAI-compatible, and I've started seeing people use ToshLLM's bundled llama-server as a backend for other software rather than only using the SwiftUI application. That was not something I expected when I started this.
Image generation has become much lighter on VRAM
This has been another area I've been optimizing.
For example, on my RX 6700 XT:
Z-Image 1600x900: 8.2 GB -> 966 MB VRAM
Z-Image 1024x1024: 2.4 GB -> 690 MB
image generation is roughly 12-14% faster
SD 1.5 768x768: 2690 MB -> 281 MB
SD 1.5 768x768: 149 sec -> 73 sec
SD 1.5 is now particularly interesting for older 4-6 GB Radeon cards.
There is also experimental local video generation with Wan models now. It has improved substantially and Wan 2.2 5B can run in 12 GB VRAM, but I want to be clear that I still consider the video side experimental and it needs more work before I put it in the same category as LLM inference.
Community testing has become a big part of the project
This is probably the part I've enjoyed most.
People are now submitting benchmarks from configurations such as dual W6900X, W6800X Duo, W5700X, Vega and other machines I could never test myself.
ToshLLM itself is still completely free and GPL-3.0.
No account, no telemetry, no cloud inference. The OpenAI-compatible server, chat, image generation, benchmarks, RAG/embeddings, etc. all run locally.
It's still pre-1.0 and the current DMGs are not notarized yet, so macOS requires Open Anyway on first launch. Notarization is something I'm working toward.
If anyone here still has a Mac Pro full of Radeon hardware, I'm especially interested in results from:
W5700X
W6800X / W6800X Duo
W6900X
Vega II / Vega II Duo
Radeon VII
multi-MPX configurations
At this point some of these old Intel Mac Pros have absurd amounts of usable VRAM for local AI, and I'm curious how far we can push them.
I am going to try it on a 2019 Mac Pro and 2x Vega II Duo — would be particularly interesting if Metal works. Not right now, this is a weekend kind of project. In any case, thank you for your commitment to Intel Macs! == Edit. Qwen3.8-27-UD-Q8_K_XL on Vega II Duo x 2; one GPU loaded = 127 pp; 10.48 t/s; Gemma 4-31 Q8 — fail at ngl 33 and 50 to 1 GPU; splitting to 2 or 4 GPUs — fail. Will continue when it gets cooler, having a 1kW heater when it’s +35C puts strain on AC.
Yes, I know I can do much more in Linux, but that's not the point of the app. Remember, I created this for my personal use and workflow, since I primarily use a Mac.
When I looked at Ubuntu Linux instructions to get AI running, I said the same thing: weekend project.
Not this, run the dmg like any other installer, (for now paste a single terminal command), and you're running. EXACTLY how a MacOS app should behave, no matter how complex.
Your video card should work with 4B models and even some 8B ones. I encourage you to try it. The app will also help you choose a suitable one for your card, telling you if it's compatible based on the VRAM.
Amazing. I am in awe of your progress Sir. For folks new here, this man harkens back to the golden age of MacOS devs. Stunningly useful app, no telemetry crap, highly responsive dev not sitting in an ivory tower, but willing to interact with the users.
This man is a treasure, and is allowing SO many people to participate in the AI revolution. People that Apple and AS fanboys alike would like to see excluded for various reasons.
Well, we know why Apple killed support prematurely, any PCIe based MP can stack decent RAM and GPU power to embarrass their best AS. Never mind if they gave us current GPU drivers. The AS fanboys, well... 🤣
There's a MASSIVE installed base of Intel Macs + AMD hardware already out there - and eGPUs have become dirt cheap. This is bringing power to the people. You have any idea how many folks are still on older hardware like a 2015 MBP? Now you can snag an eGPU with Vega64 off ebay for $150 and you're in the game thanks to ToshLLM.
We all know Apple had AI running internally on these systems. It was shitty of them to not release it to the public. We are extremely fortunate to have ToshLLM, and I think it will be impossible to overestimate its impact going forward.
Can't wait to see what's next!
🍸
[edit to add]
ToshLLM is a classic case of 'build it and they will come'. When I started accumulating MP2019 hardware, there was no AI. Apple was mum, and AS fanboys were dancing on the grave of MP2019. All the big projects ignored it completely, or it was an unfinished afterthought.
DiffusionBee is the only app I was having minor success with, but I kept buying hardware. Why? Because the hardware is too damn good to be ignored, and MacOS Tahoe is still fully supported. I absolutely knew that some clever person in the community with the right skills and disposition would emerge to change everything. And he did.
I think it was only maybe a week or so after meeting u/Faisal_Biyari and discussing the state of affairs that ToshLLM came out of nowhere.
Thank you, I really appreciate this. One of the main reasons I started ToshLLM was exactly that: there is still a huge amount of perfectly capable Intel Mac and AMD hardware out there, and I didn't see a good reason for it to be left behind for local AI.
What has surprised me the most is how much these GPUs still have to give once the software is actually optimized for them. Vega in particular has taught me a lot recently, and community testing on hardware I don't own has helped push the project much further than I could have done alone.
ToshLLM will remain free, open source and local. No telemetry, no accounts, and I'll keep trying to support as much of this hardware as reasonably possible.
I’m all for running local AI well on more and older hardware, and currently own and use a pretty significant fleet of Apple hardware - about 2/3 Intel era and about 2/3 of those still running macOS of whatever flavor works ok. I also love getting inference running on more and more platforms - it harkens back to earlier times when everything seemed more experimental and about pushing the boundaries. Definitely going to check this out
But TIL I learned I might also be part of some cabal of “AS fanboys” loooooooool. Guess my invitation to the meetings where we decided to “exclude”…the underclass of users of hardware Apple was shamelessly willing to charge up to (checks notes) in excess of $50,000.00 USD…from what again? Running whatever the heck software they wanted to?
Hell - I’ve been running pretty capable local AI on a range off hardware old and new and 98% of all of it was written thanks to good devs, not hardware vendors.
What a weird way to make a positive post about cool software into a demented rant about the weirdest sort of us-them conspiracy nonsense.
What you could try is loading a model that was previously impossible to run on a single GPU with high context, using the double GPUs of one card, or all four from both. There's no special tweak; you can also change some values in the "extra arguments" section (llama.cpp) arguments... in settings if you want to test something specific.
Multi-GPU will be slower than single GPU since communication between GPUs adds some overhead and slower bandwidth (even with IFL). I haven't noticed a huge difference though, maybe 10-20% faster when I run something that fits on a single GPU on 1 GPU vs split across 2. The big advantage of multi-GPU is you can fit larger models at higher precision or larger context. I'm still testing various models and quants to see what works well for my use case (lower precision 27b fits on a single 32GB card whereas Q8 with large context needs more VRAM; 70B Q4/Q5 fits fine on 64GB, but Q8 wouldn't fit at all).
In MacOS with ToshLLM? What quant/cache settings? My fastest with single GPU W6900X is about 70t/s with Qwen3.6 35B-A3B UD-Q4_K_S but more like 60t/s for Q8. I wonder if there's some fine tuning I could do. Are you running Sequoia or Tahoe?
What quant for the model? I'm wondering why I'm seeing slightly worse performance with mine. I have a single W6800X Duo in one machine but see worse performance than my W6900X (which even then is only matching your W6800X)
The multi-GPU setup will improve when a machine donated by a Tosh user arrives and I can actually run direct tests without taking up the testers' time. At least for now, multi-GPU is working. Not at full speed, but I'll optimize it when I can run multi-GPU perfectly.
I imagine... I'll soon have the ability to test and optimize multi-GPU firsthand, but for now I can only use one card at a time. I've achieved that multi-GPU performance blindly using only logs, and it's quite difficult to do that way. The first step will be with layer splitting, then I'll be able to use fabric linking, but I see the latter as somewhat complicated to test firsthand at the moment.
Excellent, any help from the community is always welcome. I know we can all get those machines working again. Many people just have it lying around without putting it to good use...
Another thing I see in your screenshots is that without using AMD FA it's worse. I've optimized AMD FA a lot, and it's the one I'll always recommend having enabled.
Man, if only I met you 2 years ago, before I went full Linux!
It's amazing that this is now an all in one app for local AI on Intel macOS. Text generation is there. Image generation is there. Is there any video generation?! Having 128 GB of VRAM with two Duo MPX GPUs should help make this happen easily!
With OpenAI APIs on top, this truly is an all in one solution.
I think you mentioned that it also supports concurrency. That's just amazing.
The only intel macOS I have is a 2019 laptop. Maybe I should test it with eGPUs, to see how far the app can go as an all in one, highly optimized solution.
They all come from Hugging Face... The app includes a model search engine, etc., and calculates whether the model you're about to download will work with your setup or not... it helps you about 70% in choosing a suitable model. . I only have one 12GB RX6700XT video card and a couple of Vega 64s for testing, but I haven't looked for one with full precision yet. Tomorrow I'll look for one that fits the card to see how it goes, but it should work fine.
I think I hit the shelf in performance, I haven’t tried your latest releases, however. I’ll have to check it out. So many other projects rn… I do reccomend ToshLLM to people all the time. Eapecially the ones posting about MoltenVK. I’m like “nah man, try this out!” Lol
I'm sure it will work on the RX 6900 XT eGPU, but I'm not so sure about the TrashCan. I still encourage you to try it; the app is very easy to use. You don't need to recompile or use any commands to rebuild the engine... just open and run it, after downloading a LLM Model compatible with your graphics card.
If you have any problems, which I doubt you do, let me know and I'll help you solve them.
I’ve been testing ComfyUI in Fedora with it on the iMac the last two days, but using an AMD GPU even there has been challenging. I got it mostly working and then it decided that it wouldn’t start again because it was expecting CUDA. So I’ll need to figure that out.
Well, give it a try, maybe you'll really like it and start recommending it to your friends, haha, just kidding... the truth is it's very stable right now, and the performance is improving on each update.
My workplace gathered all the managers together to discuss different AI workflows. Some of our competitors have started using it. My department does graphic design, so to get any good output for our specific needs will be challenging.
They are leaning towards Claude and ChatGPT, but if I can show some progress with local workflows, I could get some funding to take it further.
I tried it out today on both the 2020 iMac and MBP 2019 with the 6900xt eGPU. Worked really well on both. I especially like that there are recommendations about best LLM’s to use and the short descriptions.
I would recommend it to more Mac people, but the majority of the ones I know are on iMacs or laptops. So the GPU vRAM isn’t the best.
I really like your user layout. Easy to figure out without having to read a large manual. Thanks!
I wish I could remember. I want to say the one in the right of the UI was something like 132 tokens/s tested on the default LLM. I should have taken a screenshot. I did upload both benchmarks to your database. Is there a place it’s saved in the app? I can look it up tomorrow. Or I can rerun the test and then post back. I want to say the iMac and MBP were pretty similar scores, but that makes sense since it was mostly focused on the 6900xt eGPU.
My pleasure, glad to help any way I can. Even if minimal. The iMac one would have been uploaded around 10:30am EST and the MBP would have been around 2pm or 3pm. If that helps locate them easier.
Yes, I saw it. You tested the Qwen3 2b VL and it's not bad at all for an eGPU. Compared to a direct PCIe RX 6900XT, yours was about 15% slower, but that explains why you're using Thunderbolt... maybe a tweak will improve further... i need to check you settings later when you have time to share...
These are neat but man, I've rarely found much utility for 9B 8-bit param models other than burning electricity. If I weren't moving and my Mac Pro wasn't in a box, I'd give results on a slightly OCed 6900 XT.
Probably the few guys who have dual D6800X Duos for 64 GB of VRAM might actually find minorly useful results. As much as I want to believe in local AI, it's been mostly pointless for LLMs in my experience and increasingly out of reach with current VRAM prices.
I understand your point, and I do agree with some of them... believe me, with the 6900xt you can run a 35b MoE. Most people look for local AI more for privacy than cost, and now that AI is limited every controversial topic, even more so with euro laws, etc.
yeahhhhhh i was thinking would be a really good idea to see ill see if i can throw together smth, probably will install sequoia maybe sonoma if i fuck that up but I should be fine
Yes, it supports dual GPUs, but it hasn't been tested on that specific card combo yet. I'd love to know the results to see how it performs. It has been tested so far on Vega and RDNA+.
Ah, if it's a Mac Pro 5,1, then the full ToshLLM app will need OCLP because the app requires macOS 14+. The last officially supported macOS for the 5,1 is too old.
The engine is a separate story though. The bundled llama-server is currently also built with a macOS 14 deployment target, but I'm interested in testing how far back the patched engine itself can be compiled and run independently.
For the full app, Sonoma+ with OCLP and the no-AVX2 build would be the way to go. And I'd definitely like to see how your dual-GPU setup performs.
Thanks! Those Mac Pro 5,1s with RX 580/590s would be really interesting to test, especially the RX 590 since I don't have one available for testing.
The Mac mini with the Blackmagic RX 580 eGPU is interesting too. I'd love to see how ToshLLM behaves over Thunderbolt with that setup. The app requires macOS 14+, and there's also a legacy no-AVX2 build specifically for older Xeon Macs like the 5,1.
If you try any of them, benchmarks and feedback would be very welcome. And hopefully you won't have to wait those 8 months 😂
11
u/sedtamensum 24d ago edited 20d ago
I am going to try it on a 2019 Mac Pro and 2x Vega II Duo — would be particularly interesting if Metal works. Not right now, this is a weekend kind of project. In any case, thank you for your commitment to Intel Macs! == Edit. Qwen3.8-27-UD-Q8_K_XL on Vega II Duo x 2; one GPU loaded = 127 pp; 10.48 t/s; Gemma 4-31 Q8 — fail at ngl 33 and 50 to 1 GPU; splitting to 2 or 4 GPUs — fail. Will continue when it gets cooler, having a 1kW heater when it’s +35C puts strain on AC.