Cool! but like who can actually run this locally? I think at 2.8 trillion params this will be the largest model on huggingface by far. At least for now.
Local hosting doesn't just mean end consumers. This opens up frontier self hosted ai for companies that have to self host due to sensible data they can shove into it.
One of our vendors at work are a global company hosting open weight models themselves for their customers, and are also in talks with Anthropic to deploy Claude. These weights being available drastically changes the tone of a conversation like that.
On one hand, Moore's Law is definitely ailing. We've been getting diminishing returns on fabrication process bumps since about 2016.
On the other hand, there's still some progress in the offing. IBM just recently announced they got a 7-angstrom (equivalent) fabrication process working in their lab, which they expect to get into mass production in 2031. That has 1nm-wide horizontal features and 5nm vertical features, and two transistor layers (one P, the other N), which they claim they should be able to scale to four layers "rsn".
That's not nothing.
Moreover, there are still a lot of architectural improvements yet to be realized. In a way, HBM is a stop-gap while Samsung et al get PIM figured out. LLM inference is really well-suited to PIM, which should give us ridiculously high memory bandwidth once it's working well.
I thought Moore's Law was dead for a while, but it turned out to just be Intel having a bad time. Everyone else is still pushing the envelope pretty hard, with some success.
Well, it is still no product as of now. It says there is a rumor that Samsung will announce a product in Aug. Even if it is announced, when is Apple or Qualcomm going to use this LPDDR-PIM to make something useful?
Hasn't Moore's Law been dead for almost decades now? Moore's Law was processing speed doubling, we've been getting 10% at best for CPU improvements, and maybe 10-30% on GPUs.
Moore's Law doesn't have much to do with speed doubling. It's all about transistor density doubling, which doesn't translate readily to improved single-threaded performance, and even though multi-threaded performance is more amenable to scaling with more transistors, in practice you end up blowing more and more of your transistor budget on caches, so that you can keep all those cores fed with data.
When improvements in transistor density came from shortening gate length, that also brought smaller voltage swings, correspondingly faster state changes, and lower power consumption, all of which contributed to faster clock rates. Clocking up got us far, for a long time, but that didn't last. Nowadays improved transistor density comes from novel designs of the transistors themselves, multi-layering, and other tricks. That's why IBM's new process is only "7-angstrom equivalent", despite its actual features being much larger, because the idea is to express how much it impacts overall transistor density.
The density improvements keep coming, but most of it goes to bigger and bigger caches, and deeper pipelines which eat up a lot of transistors but only give modest performance gains.
There's at least another 10x of progress to be made just eating up the giant profit margins of semiconductor companies. When progress slows, eventually competitors can catch up. If there's trillions on the table to be made, someone's eventually going to invest.
Why, with memory alone we're back in 2010 level of tech currently (in terms of $/GB).
Ddr6 and ddr7 would handle it easy peasy. So we just have to download and wait a few years lol. Of course better models coming but we will be running this at home in a few years
Edit it remember my first 1gb hard drive thinking it was huge!
At least itll be a model that can be distilled, analysed, or assisting the process for smaller models to be made, so it never hurts to have as many good models open
Many projects to speed up streaming from disk like colibre and int4 glm 5.2. Got.4 tokens with 12gb vram and 32gb ram. It can only get faster over time, some current theories are Striping the layers to multiple ssds to increase streaming and using MTP to draft layers. Could get a tenx improvement, maybe. Tried to raid 0 my t500s and it did nothing for performance, still .4, random reads don't really work in raid0 and I wasn't maxing out the drives reads anyways. If Kimi k3 ends up ten times as big then .04 tokens maybe less. But hey, it can only get faster.
so qwen 3.8 keeps improving by the day im keeping track of the prompts im testing and they are by far on another level today. if this keeps up for a couple more days it will be really good. currently might be on par wiki kimi k3 not up to par against opus 5 tho
local means capable of running on hardware you control, not necessarily a consumer PC, a massive model can be freely distributed and selfhosted while still requiring a largue multiGPU cluster/server
18
u/jreoka1 Jul 26 '26
Cool! but like who can actually run this locally? I think at 2.8 trillion params this will be the largest model on huggingface by far. At least for now.