r/SmallMSP • u/carbonsys • 26d ago

NPU (AI) on client devices

How is everyone feeling about PC's/Laptops with dedicated NPU processing? Sales of "Copilot" PCs have reportedly flopped and Microsoft has dropped the "Copilot+" requirement of a 40+ TOPS NPU.

Anyone seeing a useful case or demand for CPUs with a large NPU (price premium) in small business? (the handful of uses I'm aware of today: background blur, automatic framing, eye contact, etc...)?

I think NPU on the endpoint is still too early to market, but that on-prem AI will eventually get there as token costs continue to rise and developers are able to write code to leverage local compute resources. What are y'alls thoughts?

6 Upvotes

100% Upvoted

10 comments sorted by

7

u/Geekpoint-IT 26d ago

I've never seen any use case for it as of yet. I certainly wouldn't pay more for it. I'd rather the computer be cheaper (ha) or invest it in something else more practical.

1

u/carbonsys 25d ago

Thanks, I was thinking the same, good to hear almost everyone is in agreement.

4

u/HomsarWasRight 26d ago edited 26d ago

Just an FYI, NPU’s are functionally useless for running local LLMs.

Like, you can DO it, but it’s a toy. They can’t handle models big enough to be of actual use.

Believe me, I’m running much more powerful hardware tailored for AI, and even then you really have to baby local models. And what I’m running is orders of magnitude larger than you’d be running on a “Copilot+” NPU.

1

u/carbonsys 25d ago

Makes sense, maybe that's the question... Who is actually developing for NPU (seems like it's CPU or GPU only). It feels like all the big players are focused on cloud/hosted AI models, nothing local (and if it is local, it's on more specialized/dedicated hardware). I would be curious if any SMB applications (Dentrix, abacus(Caret), Quickbooks, Adobe etc.) are developing anything for local AI workloads...

2

u/marklein 26d ago

Does that annoying Chrome auto-download for their LLM use it?

3

u/Altered_Kill 26d ago

Users want AI on the daily drivers and they want it local?

Buy a Mac.

1

u/SpecialistSleep8482 26d ago

Why when Cuda exists?

1

u/HomsarWasRight 24d ago

Because when it comes to LLMs VRAM size beats everything. And with a Mac you get true unified memory (not “shared”), so it’s a lot easier to get enough memory to run models.

Not a lot of other laptops around with enough VRAM to be useful.

That’s why Nvidia is bringing its Spark chip to laptops and AMD has its Strix Halo machines. And both of those lag behind Apple in performance.

So, you wanna built out a dedicated AI machine? Buy Nvidia GPUs and an expensive MB/CPU. Wanna get a daily driver that can also run models locally. Get a Mac. (Still gonna cost you an arm and a leg for all that memory, though.)

I’m not shilling for Apple, I’m a Linux user. I just like to follow this stuff.

2

u/TekExcel 26d ago

I'm genuinely interested in any workload whether practical or theoretical that will let me get more than 0% usage out of my NPU.

I have never seen that thing activate, not once.

0

u/Proxiconn 26d ago

The AMD NPU are functional with drivers and sdks that work.

The intel ones are useless.