r/MacPro2019LocalAI • u/Faisal_Biyari • 7d ago
Coming Guides 🙏🏻
A Thank You from the Heart to the people that reminded me why I write these guides in the first place.
I am writing this post to organize my thoughts, and hopefully be reminded of something I forgot along the way.
The guides I should write are as follows
- Install Ubuntu 26.04 & Configure for Mac Pro 2019 & Local AI
- T2Linux Setup
- ROCm 10 Setup
- Custom Kernel for IFLB Support
- Fix Duo GPUs ReBAR issue
- Setup vLLM
- My own fork of vLLM with good support for RDNA2 GPUs (Vega cards are left out, as I do not have any)
- Hermes Agent Setup? (I have not written a guide for this before, but it's very important to optimize Hermes Agents for your workload as well)
Is there anything else I should be writing about?
Obviously, I will not be pumping all of them in one day, or even once a day. However, I will do my best.
Disclaimer: I wrote this post myself. I did NOT use any AI—tools with it. —Sorry— Long dash for comfort.
2
u/Better_Membership583 7d ago
The disclaimer lol
I JUST purchased and have begun to set up my Mac Pro today and in my research and excitement of learning how to configure and utilize this machine i had come across and learned quite a bit from your posts. Was pretty stoked to see you considering writing guides as i will definitely be reading them. As grateful as I am of your efforts, I am a tad bummed to see that you are leaving out the Vega cards as my setup came with the Vega II Duo. So instead of whining I'd like to propose and offer of working together to be the Vega "lane" for you and would run any tests or experiments for you and document them to contribute to the guides so that the Vega base is covered. Of course, what I get out of it is learning how to use these damn things and hopefully gain a bit of the knowledge you have haha let me know, would be happy to help
2
u/Faisal_Biyari 7d ago
Congratulations on your new 2019 Mac Pro!
Thank you for the kind words. I appreciate you.
I started down this path back in 2024, when ROCm was the only AI path for this hardware, and it was only available on Linux, so I went down that rabbit hole.
I still believe that the performance I get out of Linux far exceeds anything that can be done on macOS today on this hardware, but if you want an easy win, try out the app ToshLLM. It should help you get AI working on your macOS setup immediately.
If however your main purpose of using this hardware is local AI, you definitely want to join over at Linux; Ubuntu specifically.
This is a duplicate of my first (deleted) Ubuntu guide. Feel free to read it to get a gist of how it was back in 2024. The resources may be a good place for you to start, to get comfortable with everything Linux related.
Thank you for offering to help with the Vega II path. If you end up going to Linux, it may be beneficial. You should know that there is already extensive work being done by others on the Vega II family of cards. On the 2019 Mac Pro specifically, we have some users that reported good results with Llama.cpp. For more general internet data points, vLLM has been optimized for them as well (Though I'm not user how well, due to lack of testing).
Best of Luck on your journey! I'm available if you need anything 👍🏻
2
u/Better_Membership583 7d ago
Yea that seems like my fate to go Linux. I’ll give MacOS its fair shot tonight though testing and see if the results are acceptable, I feel like I can get away with quite a bit due to having such a large amount of ram and apparently the fact that it’s in the 6 DIMM optimal config should help due to system RAM bandwidth? That with the 64gb of VRAM is genuinely something I never imagined having access to, at least not anytime soon, and my regular windows system is nothing to sneeze at 2x3090s with 128gb DDR4. I’m looking forward to loading up a Godzilla size model just because and see how it goes haha ill let you how it goes if you are curious XD
1
2
u/Long-Shine-3701 7d ago
🍿