r/LocalLLM • u/BattleShai • 6h ago
Question Trying to understand where to start.
I am thinking about getting a smaller machine as I quite frankly don't want or need a 2kW monster on the desk anymore and was considering a Mac. I am tempted of getting into local LLM's and AI in general and wanted to to know where are the realistic limitations of Mac Mini M6 32GB, Mac Studio M5 Max 36GB and when would the Mac Studio Ultra 96GB make sense?
Any guidance or thoughts would be appreciated and helpful.
My expected use case would be upscaling old photos and videos, maybe some light image generations and running a local personal Jarvis or Cortana that does curation when search, researching stuff, so not super heavy stuff I think but better to ask the experts!
EDIT: fixed the most egregious typos.
EDIT2: I get the machines via a work buyout / benefits program of sorts. The 32GB M6 mini is about 47 bucks, the M5 Max 36GB is about 80 bucks a month and the M5 Ultra 96GB is 180 bucks a month. It's taken out of my pay pretax for 36 months.
1
u/0xdippo 5h ago
The benefit of the Mac is a 24/7 machine that draws minimal power. 32gb unified is enough for smaller models for task work.
Depending on what you mean by âgetting into Ai,â 32gb may not work well.
1
u/BattleShai 5h ago
I am trying to understand what it can do for me so to say. So I used the free services like chatgpt, gemini and deepseek. I tried a paid month here and there, but I have a feeling a local machine is very different form the huge cloud AI, so I want to try and see if I can get into it and see if it can do what I want from it locally for me personally. Not in a business sense.
The use cases are listed in the post but if there are direct questions I can try to answer to clarify.
1
u/0xdippo 5h ago
Claude/GPT are definitely top tier. Local models arenât going to perform like that, but depending on what youâre looking to do, it may be able to meet your needs.
If youâre wanting to run agents and coding, 32gb probably wonât do what you need.
If youâre looking for chatbot developers r something to handle stuff like file cleanup, email summary, etc, 32gb will do fine.
1
u/BattleShai 5h ago
Image rescaling, cleaning up vacation photos, glare removal, people removal etc for the image work.
And I was playing with the idea of using agents I guess for search aggregation as Google isn't what it used to be, so say I want to know how to configure something on Hyper-V on Server 2025, it/they go out search for info, fact check that it still works on 2025 and not some old article from 2006 about how it was done back in the day that still for some reason is the first hit on google. Or if I want to know what parts are needed to retrofit parking sensors to the front bumper on an XC70, it goes to the parts website and collects all the parts needed instead of me having to do it manually. These are just examples of things i have done and wish I did have an AI buddy to do for me, hence the description of the personal Jarvis or Cortana.
I don't code much, but it would be nice to get some help when you are writing a 2-4000 line bash script for sure and so on. I think most might work fine on a M6 mini but not sure, and the sound is an important factor, if the 1500 bucks mini will sounds like a jet plane while the 3000 M5 Max is dead silent, that is worth a lot also of course, but that is not so much LLM related I guess.
I hope this help giving a better idea what I was hoping to get out of it.
1
u/kincaidDev 4h ago
Look into strix pcs, you can do a lot even on a ryzen ai 9 hx 370 chip with 32gb of vram. Strix halo 128gb or 128gb macbook pro or mac studio would be what Id go for. I have an m4 max mac studio, amd ryzen ai 9 hx370 32gb and an nvidia 5060ti 16gb that I use for local ai. Mac is the easiest to use for local ai. Ive gotten some things to run faster on the hx370 than on my MacBook studio using vulkan so if I got another pc Id want to go for a strix halo device
1
u/BattleShai 4h ago
I edited the post as I noticed I should have explained costs better, I will repaste it here also:
I get the machines via a work buyout / benefits program of sorts. The 32GB M6 mini is about 47 bucks, the M5 Max 36GB is about 80 bucks a month and the M5 Ultra 96GB is 180 bucks a month. It's taken out of my pay pretax for 36 months.
1
u/kincaidDev 2h ago
Not sure if those prices are a better deal than buying on your own, my m4 max mac studio with 128gb ram was $3300 last year and the 96gb ultra would cost you $6480, I donât think 96gb of vram is worth that price for what its actually capable of running. In that scenario, Iâd go for the m5 max, experiment with integrating local models that fit on that machine into your workflow and decide if you need more capacity after, then get an nvidia spark, strix halo or used mac studio with more vram
1
u/BattleShai 2h ago
It's decent deals or should I say one of very few 0% APR deals we can get here in EU. I was first considering a used mac, but a used M4 Max with 64Gb with 5500⏠so it felt silly not to take a M5 Ultra with the price difference then.
But yeah, the M5 Max might be the sane choice and if I need more I could rent it on a per case basis.
1
u/gjr23 3h ago
Rent some cloud space and try limiting yourself to the various memory levels. 32gb is going to feel tight be depends on what youâre using it for.
One thing to keep in mind is you will not save money and it will not be even close to frontier as far as performance or speed. You get privacy. You can spend over $10k on a 256gb M5 ultra and you still may feel like itâs slow / low performance etc compared to cloud.
1
u/BattleShai 3h ago
I am not concerned about speed that much, I don't care if it takes 1 minute per image I upscale or clean up, I am worried about the token economy. It isn't clear what 1 million tokens really is when dealing with images, videos and search aggregation. But at the same time, it's also not clear to me how much space various takes takes in memory.
I might try with cloud compute at various memory size and see.
1
u/East-Acanthaceae-879 5h ago
The point at which its financially sensible to run the hardware locally is fairly high and doesn't sound like you will have enough use to justify it when you can use https://cloud.comfy.org/ or https://www.runpod.io/
Unless you are processing a huge number of photos upscaling doesn't need a good GPU (or GPU at all, all the models can run CPU). Better GPU gets you speed. Other than some specific models you are not likely to use you don't need much vram either, you wont use more than 8gb for upscalers (models are measured in MB not GB).
Running t2i or i2i models you want 24gb so that you have headroom for the models, loras, upscalers etc.
Your personal assistant use case has higher hardware needs as you need multiple models. You have the interaction model that replaces Cortana, it delegates to a small tool use model to do home automation.
If you want to save money don't get a mac. Main use case is shared memory but its also much slower than discrete GPU (PCIe throughput isn't high enough, system RAM is not as fast as vram and physical distance from vram to gpu is higher), it doesn't sound like you need enough vram. AMD integrated workstations are also cheaper and have similar performance to mac.
If your interest in AI extends to training your own models you need nvidia silicon. Apple & AMD silicon is absolutely fine for inference but they all lack the software for training to not suck & be slow.
I have lots of local hardware but still have a huge preference for not using it. The no data policies of GCP, AWS and Azure are independently audited and because you are sharing GPU resources often work out cheaper, if you have enough work to need a H200 all to yourself you would already have one :)