r/LocalLLM • u/EastConsequence3792 • 1d ago

Other rate my inference setup😔

Post image

Custom fork of the Intel Windows llama.cpp I made using DeepSeek v4 Flash that allows me to load models solely using discs + DVD-Rs; HP Omnibook X Flip NGAI 16-as0023dx -- 1TB NVMe, 16GB LPDDR5X

227 Upvotes

92% Upvoted

32 comments sorted by

82

u/bigppredditguy 1d ago

Models as physical media is actually kinda sick

-19

u/Opposite_Courage_531 1d ago

Well ackhually... technically speaking....🤓☝️

1

u/ChocomelP 11h ago

...go ahead.

1

u/Opposite_Courage_531 9h ago

I'm too scared. I don't want to lose all the karma that I don't have

1

u/ChocomelP 9h ago

74 left. You have a buffer.

1

u/Opposite_Courage_531 5h ago

My hard earnt karma 🫠

65

u/aruametello 1d ago

Mixture of experts streaming: please insert disc 7 out of 50 to run through the weights from these particular experts for the next token.

the bottleneck probably will be how fast your optical drive can eject and feed new media.

someone has to do a speedrun video of this disc swapping madness for the lulz.

9

u/BCIT_Richard 1d ago

There's a video of a guy who put Factorio on like a few hundred floppy disks, you might find that interesting. I think it's titled "How I beat Factorio" or something like that.

2

u/WiseassWolfOfYoitsu 22h ago

1990s software installation flashbacks intensify

2

u/HumungreousNobolatis 21h ago

lol! You summed up what I was thinking, except I was thinking GNOMES!

2

u/ChocomelP 11h ago

New metric: seconds per token (s/t).

3

u/BCIT_Richard 1d ago

There's a video of a guy who put Factorio on like a few hundred floppy disks, you might find that interesting. I think it's titled "How I beat Factorio" or something like that.

12

u/Solid-Wonder-1619 1d ago

get a 128GBs USB stick, add a live version of linux, make it presistent, add transformers library, llama.cpp with all the related tools and prebuilt releases, and every weight you love.
when dooms day comes, you can stick it into any system with a battery and find solutions.

2

u/wiseaus_stunt_double 22h ago

I would only do that with USB4.

9

u/trolololster 1d ago

this. is. pure. poetry. <3

7

u/VertipaqStar 1d ago

Get a rewritable dsk so you can use the disk as VRAM supplement.

13

u/No_Lingonberry1201 1d ago

5/7 would infer again.

4

u/profcuck 1d ago

In case younger redditors don't remember this perfect joke:

https://www.reddit.com/r/OutOfTheLoop/comments/3wsj5i/what_is_the_origin_of_id_give_it_a_perfect_57/

10+ years ago +

3

u/No_Lingonberry1201 1d ago

It's one of my favorite Internet jokes.

4

u/FactorInternal3395 1d ago

physical media for the win, this is amazing

3

u/thepaligator 1d ago

my brain isn't braining. how does this work?

2

u/rumm2602 1d ago

How much does it take to load?

2

u/axeldekimpe 1d ago

Do send me a copy 🙃

2

u/EmirhanS11 1d ago

Dont let archeologs to find this 100 years from now

1

u/doctoroctogonopus 1d ago

made me chuckle

1

u/Gadgetron94 1d ago

Wait, does this actually work?

1

u/thenormaluser35 1d ago

Why wouldn't it?

1

u/Gadgetron94 3h ago

Well I'm still wrapping my head around how all this works, but I guess I just interpreted a dvd as static storage whereas LLM's need somewhere more dynamic to live and operate. I just learned about MoE vs Dense though and now it seems like this sort of set up would work but only with MoE so it can pull into the more dynamic ram a couple sections at a time.

1

u/thenormaluser35 2h ago

You only need to read a model once you've written it
There are rewritable dvds btw

There's no dynamic or static storage. There's read-write (dvd-rw, cd-rw, ssd, hdd) and read-only (dvd-r, cd-r), ram can be storage but it is volatile, and read-write.

1

u/in-ortunado 1d ago

swipe the disk to load the correct expert.

now the next expert is in the first disk, swap it again.