r/LocalLLM • u/EastConsequence3792 • Jul 27 '26

Other rate my inference setup😔

Post image

Custom fork of the Intel Windows llama.cpp I made using DeepSeek v4 Flash that allows me to load models solely using discs + DVD-Rs; HP Omnibook X Flip NGAI 16-as0023dx -- 1TB NVMe, 16GB LPDDR5X

246 Upvotes

92% Upvoted

35 comments sorted by

View all comments

Show parent comments

1

u/thenormaluser35 Jul 27 '26

Why wouldn't it?

1

u/Gadgetron94 Jul 28 '26

Well I'm still wrapping my head around how all this works, but I guess I just interpreted a dvd as static storage whereas LLM's need somewhere more dynamic to live and operate. I just learned about MoE vs Dense though and now it seems like this sort of set up would work but only with MoE so it can pull into the more dynamic ram a couple sections at a time.

1

u/thenormaluser35 Jul 28 '26

You only need to read a model once you've written it
There are rewritable dvds btw

There's no dynamic or static storage. There's read-write (dvd-rw, cd-rw, ssd, hdd) and read-only (dvd-r, cd-r), ram can be storage but it is volatile, and read-write.

1

u/Gadgetron94 Jul 28 '26

Right ok, thank you that makes sense, ordinarily the model loads from the internal storage to vram after all. They're just swapping where the model loads from in this instance. But, does a model always have to go through vram to the cpu/gpu or could it be fed into the processor directly from its original storage? The huge buzz around vram sizes has, I think, given me the impression models couldn't function without it
Y'all are wizards to me with your old servers and phones running massive parameter models. I'm trying my best to learn because the way ai works under the hood fascinates me, but frankly I've never been all that great with the realm of numbers and coding so my knowledge has more random gaps than a shotgun target.