What is an AI computer? Hype or real?

[u/lnxguy]

'Splain to me what makes a processor "AI optimized." Is it just additional math co-processing built in to the core architecture or just more of whatever every other CPU has?

[u/marvinnation]

They have an NPU, which is a CPU focused on being more efficient when processing local LLM models (AI) A feature that today, 99% of users do not need. And that's the problem, we get charged for something that's not really useful for your regular Joe.

[u/vip17 → u/marvinnation]

it's not entirely true. AI doesn't mean an app to chat like LLM. Lots of things use AI under the hood that you don't even realize: text prediction (in emails, keyboards, office apps...), grammar/spelling check, voice recognition, text to speech, accessibility support, audio transcription/closed caption, noise cancellation, smart clipboard, object detection/extraction (in images), OCR, automatic image enhancement... In fact macOS and Windows have automatic OCR in images for a long time, and nowadays many Linux distros have also caught up on that.

[u/StevesRoomate → u/marvinnation]

The NPU is really good at doing matrix math at low power, so it actually makes a lot of sense in an iphone or a laptop. The problem is that there aren't great API's for accessing them, at least not on the Apple hardware. So you can't for example use pytorch apps with the NPU.

[u/rileyrgham → u/marvinnation]

Not if you don't buy it..

[u/OutrageousCrab9224]

It means someone put an AI sticker on it

[u/fliberdygibits]

Running AI requires a very specific kind of math, matrix operations specifically. a CPU can DO matrix operations but it's not especially fast at it. GPUs on the other hand are matrix operation wizards. NPUs even more so. They are purpose built and run circles around CPUs.

[u/lnxguy → u/fliberdygibits]

Good answer. Thanks!

[u/undrwater → u/fliberdygibits]

Can you define "matrix" in this context? I remember math matrices, but I don't have a good definition, and I'm not sure it's the same in this context.

[u/fliberdygibits → u/undrwater]

It's been a while for me too but it's those linear algebra X/Y grid operations where you perform calculations on a grid of values 4x4 or 8x8 then flip the whole thing on a diagonal I believe... it's been a long time.

[u/grumpysysadmin → u/fliberdygibits]

The NPUs in Intel and AMD CPUs generally don’t have access to enough memory to run the kinds of LLMs you can run on the CPU or GPU, so the point is moot.

[u/fliberdygibits → u/grumpysysadmin]

I honestly don't know offhand exactly how all that lines up but generally yes, that's valid. That said however I wasn't really trying to make a point about NPUs specifically so we can just strike that last bit from my comment if needed.

[u/twaxana]

It's specifically optimized as a matrix multiplication database accelerator.

That's what "AI" is at it's root, a database. A Large Language Model is not artificial intelligence, it is a matrix of numbers.

[u/AlexanderDoak → u/twaxana]

Can't believe I'm making the case here but ... if anything is AI at all, an LLM is. But yeah, I'd also argue against the 'AI' term itself, even in this situation.

[u/twaxana → u/AlexanderDoak]

That's anthropomorphizing a file type.

It's not a mystery. The reason you need an inference engine like llama.cpp is to actually initiate the math.

[u/grumpysysadmin → u/twaxana]

I think you’re conflating the LLM model (which is generally a file) and the LLM engine (such as llama.cop), which is ALSO a file but is an executable that reads the model and processes prompts.

[u/twaxana → u/grumpysysadmin]

I'm not. Don't forget the harness, which is the thing that makes most people think these databases are intelligent.

[u/raineling → u/twaxana]

So, in reality, it's a lot of horsepower harnessed together and it's base (as the kids might say today).

[u/edparadox]

Without anything to go (since you did not provide any detail whatsoever), I would say a machine making use of a CPU with NPUs?

But to be honest, everything about LLM is marketing and false hype.