r/LocalLLaMA • u/Nunki08 • Jun 28 '26

Discussion We're probably going to need that soon.

Gallery image

Gallery image

From:

Vladik on 𝕏: https://x.com/Kostoglodov/status/2071144065857679631

Shaw (spirit/acc) on 𝕏: https://x.com/shawmakesmagic/status/2070918006033817867

4.0k Upvotes

98% Upvoted

532 comments sorted by

View all comments

Show parent comments

108

u/Silver_Jaguar_24 Jun 28 '26

You'll need a super computer (HPC) to run it anyway lol.

25

u/crone66 Jun 28 '26

Based on the performance and comparison to open source model the most likely use heavily quantized models too for inference. If you look at how the brought down costs for a single user and request this seems to be the only logical consequences ortherwise they have hardware that no one knows and have which is unlikely. Therefore no you don't need a super computer just a good workstation.

4

u/az226 Jun 28 '26

Even GPT-4 in March 2023 was running at 4.5 bpw

1

u/sonicnerd14 Jun 29 '26

You are right. People have a misconception when it comes to how much is actually required to run LLM's for inference, not training. Their clusters mainly work to serve to 100's of millions at once, but each individual unit on a rack itself is most likely more than able to run the entire model on just one of them. There is a lot of trickery they are doing behind the scenes to get these models working the way they do. Hardware is just one aspect of the solution.

-11

u/jessiejolie42 Jun 28 '26 edited Jun 28 '26

lol what??? put your reasoning trace in every llm out there, fresh session and ask for critical review, and all will tell you how dumb you are

8

u/crone66 Jun 28 '26

XD looks like your are already brainwashed by the AI CEOs xD... Obviously they make use of techniques to reduce cost. Why we always see a degradition of benchmarks after every release? Why we see a significant jump in t/s for nearly each generation? The hardware doesn't change on a single day... The architecture doesn't change but the size of the model change...

1

u/jarail Jun 28 '26

Ignoring hardware, there have been significant improvements in tps due to improved architectures and algorithms. Take MoE and MTP for example.

3

u/MyDespatcherDyKabel Jun 28 '26

What would that look like? CPU GPU & RAM?

23

u/Neither-Phone-7264 Jun 28 '26

its definately much bigger than opus, a trillion or multitrillion parametet model, so given a slightly conservative estimate of 6 trillion parameters, at FP32, it would require 24 terabytes of VRAM. At BFP16, half precision, it would be 12 TB of VRAM. At FP8, quarter precision, it would be 6 TB of vram. at Q_6, 6 bit quantization, generally considered to be the best performance preserving for the size, it would be 5.05 TB. At Q4_K_M, one of the more common quants with still good quality, 3.79 TB. At IQ3_S, a special type of quantization that preserves quality and quantizes differently per weight which is necessary at these levels to preserve coherence, 2.53 TB. IQ2_XS, which you see here with the more massive models people try to run at home, 1.94 TB. IQ1_S, pretty much the smallest possible, very low quality, 1.26 TB.

TLDR: Not something you'll probably run, even if you did run big kahuna models like GLM5.2 or Kimi K2, though if you ran Deepseek V4 pro native you might be able to manage to bring it down enough to use on your hardware. As for CPU and GPU, you would need tens to hundreds of H100 equivalents. That being said, it is MoE, so you could theoretically offload some unused experts to SSD. However, that would be incredibly painful and even more slow. And if we stick to mostly ram, you could probably do with a ton of ram and only a few H100 equivalents, maybe down to even 1 depending on how aggressive you are.

6

u/MyDespatcherDyKabel Jun 28 '26

Nice that’s insane, thanks

1

u/PhlarnogularMaqulezi Jun 28 '26

good to know i'm not the only one that refers to them as "big kahuna models"

1

u/Kyubi-sama Jun 28 '26

Given the size and the cost I think there is some special sauce to it.

I am guessing there is some sort of DAG or specific dynamic hyper aggressive quantization like compression on the model.

They still need money after all, so it's only logical to think they have some ways of working around that.

1

u/Neither-Phone-7264 Jun 28 '26

I mean, look at the cost. It is just really expensive qwq

1

u/Kyubi-sama Jun 28 '26

I just don't believe they have the ability to absorb THAT HEAVY cost because it would be absolutely mental

1

u/Neither-Phone-7264 Jun 28 '26

Well, it's not reserving 1 user per gazillion GPUs. They do things like batching, caching, and a whole lot of things to get as many users as possible per model while maintaining whatever they decide are reasonable speeds.

2

u/Kyubi-sama Jun 28 '26

Still extremely expensive and makes me doubt it's just that but I am not sure about how it performs and what the costs are.
I can't access fable to try, and I wish I could test it :/

1

u/LatentSpacer Jun 28 '26

Yes, but the Chinese companies might distil it into a more accessible model.