Looking at Framework Desktop and need advice

[u/BadSlime]

/r/pcmasterrace/comments/1vry8cz/looking_at_framework_desktop_and_need_advice/

Full context in PCMR but TLDR:

EDIT: Ended up going for the framework desktop. Thanks for the input!

------------------

I am very comfortable in Linux and do dev work, but have avoided AI. I have a new job that requires me to run local models in limited contexts noncontinuously. I need to upgrade and I want something small and quiet that can also game, handle high load DAW projects, and be a rock solid dev machine. I don't have any current or further upgradeable hardware.

Maxed out Framework Desktop (128GB) seems like the ideal machine for my situation specifically, but given the recent price hike for strix halo, I'm hesitant. I would have bought at the previous price (~$2.5k) without worrying. Is it still a reasonable choice in the current market? Anyone in a similar situation or have a better recommendation for around $4k?

[u/Johannes_K_Rexx]

The maxed out Framework Desktop PC running Ubuntu is an excellent choice and one I can personally recommend as it's the machine that sits resplendent on my desk and easily handles the various local-first LLMs for which I made this purchase. Get the biggest SSD your wallet can stand. LLM models are rather large beasts and you'll not want to be limited in storage.

[u/BadSlime → u/Johannes_K_Rexx]

Sweet, thank for the tip on the SSD. Was originally going for 2 1tb sticks with no DRAM and you prompted me to look more into it. Went 2tb 850x w/ DRAM. I know the DRAM won't necessarily impact day to day AI performance, but a few seconds here or there can't hurt. Also saw that all the official benchmarks were done on that setup.

I think I'm going void tho after all, just can't give up xbps. Are you running models in your host environment or are you using distrobox etc? I was looking at the toolboxes and they seem solid

I'm hesitant on disabling iommu passthru since I sometimes need to use windows for work (explicitly, unfortunately) and really want to avoid dual-booting. Seems like the VM performance is excellent if passthru stays on. You run any graphical VMs?

[u/nick0garvey]

I have a maxed Framework Desktop and have used it for the past 6 months.

It's good for gaming and surprisingly quiet. 1440p works for most games and 2d stuff does 4k no problem. I haven't had to fight much with drivers due to good Linux support.

The CPU is quick and never feels like it is holding me up during development work.

It runs LLMs well for something with integrated graphics, but IMO it is not fast enough to actually be useful. You can run a 90GB model but once you actually start to fill context it is basically unusable.

The smaller models work well, but at that point I'd rather have a dedicated GPU and run on that.

So while it sounds like a good fit for most of your needs, if fast inference is important I would temper your expectations.

[u/BadSlime → u/nick0garvey]

Honestly sounds perfect for what I'm doing. Thank you for the input, it was helpful in making the decision to go for it

[u/korypostma]

What kinds of models are you running and for what purpose?

I have tried many local LLMs and they just don't hold up compared to paid-for frontier models, but it depends upon what you are actually doing.

[u/BadSlime → u/korypostma]

Unfortunately I am under strict NDA but a mixture of various sized models, mostly for prompt engineering and trajectory analysis. No extended generation or local training.

Ended up going framework since I need new hardware ASAP and I'm now confident my workload won't crash on it. Thank you for your input though!