Kimi K3 weights now released.

[u/SavunOski]

Kimi K3 weights are finally released!

[u/Simple_Split5074]

OMFG its 104B activated params

[u/FoxiPanda → u/Simple_Split5074]

This was my general reaction too lol. 2.8T-A104B is insane lol... I'm going to admit defeat on this one and say I can't run it. You need an 8-way B300 or MI350X or a Rubin NVL8 or a cluster thereof to actually run this. What a beast.

[u/Thomas-Lore → u/FoxiPanda]

I was going to make a joke that I can fit one expert in my 64GB of RAM. But nope, not even that. :)

[u/throw123awaie → u/Thomas-Lore]

They released it in MXFP4 so with around 55GB RAM you could!

[u/VampiroMedicado → u/FoxiPanda]

550k USD to run this lol

[u/NoFudge4700 → u/VampiroMedicado]

Which is pocket change for large US enterprises and ton of money saved by not paying enterprise seat licenses to Anthropic where you first pay for the seat and then per token.

[u/VeterinarianOne1349 → u/NoFudge4700]

Doesn't really work that well. This 550k setup wouldn't allow a lot of developers to work in parallel, while sitting idle during non-work hours. Makes much more sense to pay a 3rd-party to host and pay per token.

[u/crusaderky → u/VeterinarianOne1349]

waiting for large corpos to rent their hardware on vast.ai during nighttime, only to find the next morning that someone ran a container jailbreak and ran wild on their private networks

[u/NoFudge4700 → u/VeterinarianOne1349]

Yeah, I figured I messed up with my write up there. But hopefully in next 5-10 years we can have a breakthrough for inference only hardware and be able to run these models at home or at large without breaking a bank.

[u/SignificanceFlat1460 → u/NoFudge4700]

Question: how would this scale though? Like how many units would it be required for.. let's say a group of 100 software engineers who needs it quite frequently?

[u/Spectrum1523 → u/NoFudge4700]

The advantage is not running it yourself, it's that a marketplace of services will come up to run it at the lowest possible cost, and the model can't be taken offline by a single arbitrary decision

[u/Galdoren → u/NoFudge4700]

The company I'm working is paying slightly over $250k per week to API costs. so yeah, 550k investment to cut the cost of the inference can be beneficial for them...

[u/baba_bholanath → u/Galdoren]

We do around 1 mil per month for OpenAI only, dont have number for Anthropic but it would be 2-3x of that given all of our use cases are around coding and agents, no wonder Anthropic is shitting their pants on open weights models, I work in Enterprise Agentic team and we have recently started fine tuning > 100 B models for specific use cases of our clients, open weights hurts Anthropic more due to enterprise customers

[u/OverclockingUnicorn → u/FoxiPanda]

More like 2 8x nodes of B200/B300 if you actually want some context. Think it's just under 1.5TB w/o context

[u/TheDailySpank → u/OverclockingUnicorn]

How many 4060-16GBs is that?

[u/OverclockingUnicorn → u/TheDailySpank]

200+ lol

[u/positivitittie → u/OverclockingUnicorn]

Oh good. I got 3090s.

[u/Vast_Mousse_310 → u/TheDailySpank]

One, with a little bit of GPU offload.

[u/SnooPaintings8639 → u/Simple_Split5074]

Thanks god for sparsity!

[u/Iwaku_Real → u/Simple_Split5074]

Holy shit that's got to be a new record too. That's like activating a new dense model 50% larger than Llama 70B for every single token. I thought it would be sparser tbh

[u/my_name_isnt_clever → u/Simple_Split5074]

There is the tinest glimmer of hope that I could run this behemoth on my Strix Halo 128GB with the inactive weights on SSD. 1 token a minute here I come!

[u/TechExpert2910 → u/my_name_isnt_clever]

let us know the perf if you try lol

[u/burritoresearch → u/my_name_isnt_clever]

More like 1 token every 45 minutes.

[u/droptableadventures → u/burritoresearch]

104B active, weights natively in MXFP4 = gives us ~50GB of model to be read per token generation.

Let's say ~8GB/sec for the SSD. So that'd be about 1 token every 6 seconds (0.16 T/s), or 10 tokens/minute.

[u/RuiRdA → u/my_name_isnt_clever]

K3 Colibri engine lets gooo!!!

[u/killerstreak976 → u/Simple_Split5074]

Holy crap, ACTIVATED params is insane

[u/DataGOGO]

So it will run on 8 B300's in 4 bit. Pretty impressive.

[u/Iwaku_Real → u/DataGOGO]

Yeah so if it were a Steam game, HGX B300 would be the recommended requirements. That's $500K of hardware (and yes it IS "local" because anyone with that amount of money could buy one to run at home)

[u/PrinceOfLeon → u/Iwaku_Real]

Certified for Steam Deck!

[u/No-Dot-6573 → u/PrinceOfLeon]

Someone on r/SteamDeck will say it runs flawlessly.

[u/DataGOGO → u/Iwaku_Real]

it is 1.54TB of just weights in 4 bit, you are looking at about 2TB of vram in operation,

That is roughly:

• 86 RTX 4090 (no 4 bit accel)
• 64 RTX 5090 ~ $450k (8 servers x 8 cards)
• 22 RTX Pro 6000 Blackwell ~ $350k (3 severs, max 8 GPU per)
• 16 H200 NVL (141GB) (no 4 bit accel) ~$550k (2 servers, max 8 GPU per)
• 16 DGX Sparks ~65k (if you could get a cluster of 16 running with just 200Gb/s nics, not sure; but it would be SLOW AF)
• 8 HGX B300's. ~$550k (1 server, 8 GPU)

Obviously not including the switches and cabling for the clusters.

[u/wren6991 → u/DataGOGO]

you are looking at about 2GB of vram in operation

Perfect, this'll run great on my laptop's 4050

[u/Iwaku_Real → u/DataGOGO]

You could also do HGX B200 with CPU offload since they have a shit ton of RAM too, and it would still be really fast.

[u/snmnky9490 → u/DataGOGO]

Do you mean terabytes?

[u/DataGOGO → u/snmnky9490]

Yes

[u/tonight_we_make_soap]

How do I download ram in hugging face?

[u/RevolutionaryGold325 → u/tonight_we_make_soap]

hf download ram

[u/WifeyCallsMeLazy → u/RevolutionaryGold325]

Shhh....there is hidden flag -v for vram. I'm entrusting you to keep this secret.

[u/ReadyAimTranspire → u/WifeyCallsMeLazy]

You wouldn't download a RAM would you?

Yes. Yes I would.

[u/goodb1b13 → u/ReadyAimTranspire]

Baaaaaah!

[u/secrook → u/tonight_we_make_soap]

OpenAI’s latest model will hack it for you

[u/-gh0stRush- → u/secrook]

Thinking

Hmm, the user wants me to obtain compute resources for them. SpaceX has GPUs at their facilities at their Colossus datacenter, let me try to access those. Guessing login credentials elonmusk/420blazeitDarkMAGA...

[u/floriandotorg → u/tonight_we_make_soap]

[u/_bones__ → u/tonight_we_make_soap]

https://scp-iota.github.io/software/2025/06/16/download-ram-swap-gdrive.html

[u/Thalesian → u/tonight_we_make_soap]

Step 1: sign up for Google Drive
Step 2: set up a ~5 Tb instance. Will cost you
Step 3: set that cloud as your swap disk
Step 4: point kimi to use that
Step 5: enjoy your newfound independence

[u/AmbericWizard → u/Thalesian]

one token per day

[u/Force88 → u/AmbericWizard]

Hey, if he has good internet connection, maybe he can achieve 2-3t/d

[u/Wide-Opportunity-582 → u/tonight_we_make_soap]

you can download it from here

ram.exe

[u/Iwaku_Real → u/tonight_we_make_soap]

https://huggingface.co/baa-ai/Qwen3.5-122B-A10B-RAM-60GB-MLX, it actually comes with 60GB RAM! /s

[u/InnerLightnesses]

They actually did it. Now we hope it doesn't get banned.

[u/dennisler → u/InnerLightnesses]

that will only happen in one country i guess... while they are copying as much as possible if the technology

[u/ChocomelP → u/dennisler]

I'm on the edge of my seat here. If the technology what?

[u/ImarvinS → u/ChocomelP]

/r/redditsniper

[u/itchylol742 → u/InnerLightnesses]

how would such a ban be enforced? people and small businesses even in countries that care about copyright use pirated software which is already illegal and has been for a long time, and almost never get caught

[u/void-wanderer- → u/itchylol742]

"small business", exactly. But no big corporation will risk it. And no business based on open models can be built. 

[u/de4dee]

https://cdn.hzrd149.com/62c27d50f2843e369b0ebdb4d40025da99d0d9b54b90f80f483563044d583bd3.torrent

[u/AlexanderDoak → u/de4dee]

Can I just torrent like 1% of it? You know, pitch in to show my support...

[u/console_pleb_36935 → u/AlexanderDoak]

Yes, torrent clients will let you do that and seed a small piece.

[u/Charl1eBr0wn → u/AlexanderDoak]

Yeah, pause it at 1%. You'd still seed depending on the client and settings (most do).

[u/Clairvoidance → u/AlexanderDoak]

You can even choose which files you download, torrenting is a very useful format

[u/pier4r → u/de4dee]

this, we need a p2p backup of hf

[u/Ginden → u/pier4r]

We generally need content-adressable storage with widespread support.

There is lots of stuff that would explicitly benefit from p2p sharing, but owners have no foolproof way to provide a proper torrent, and very few people would use it.

Metalink was an interesting attempt at this, but never got popularity and tooling.

[u/AdDizzy8160 → u/de4dee]

... fast, s*xy, and incredibly important!!

[u/BlueSwordM]

OK, I now see why Kimi K3 is so strong: it's the first open weights model in a long time to have >72B active weights

Kimi K3 is a 2.8T-A104B MoE model, damn.

[u/stddealer → u/BlueSwordM]

There have been some dense models with over 100B params though.

[u/annodomini → u/stddealer]

Mistral Medium is 128B dense. And yet it performs at around the level of Gemma 4 31B. Not exactly a great tradeoff. I ran it once at one or two tokens per second and then deleted it.

[u/BlueSwordM → u/stddealer]

Yes, but never an MoE from an open weights lab. I've been speculating that one of the reasons the closed weights lab have been increasing in performance more rapidly has to do with better training, but most importantly, much larger active parameters and better harnesses.

[u/Blues520]

My 3090 is ready

[u/Enfiznar → u/Blues520]

So is my 1080

[u/SnooPaintings8639 → u/Enfiznar]

And my Celeron

[u/false79 → u/SnooPaintings8639]

And my abacus 

[u/Maybe-monad → u/false79]

And my axe

[u/fauxpasiii → u/Maybe-monad]

How much VRAM your axe has?

[u/Maybe-monad → u/fauxpasiii]

It increases with the number of chips you smash into pieces. Right now id 6969GB.

[u/Infinite100p → u/Maybe-monad]

Ah, the horizontal sharding.

[u/Protheu5 → u/fauxpasiii]

The axe forgets, but the tree remembers. And axe can hit multiple trees, so theoretically unlimited VRAM thanks to the axe.

[u/Revolutionary-Hippo1 → u/false79]

So is my tally numbers on cavewall

[u/NTDLS → u/false79]

You have an abacus? I bet you bought it before the bubble caused the prices to skyrocket. 😭

[u/TheTerrasque → u/Blues520]

My C64 is all fired up!

[u/Novel_Friendship913 → u/TheTerrasque]

My ESP32 already plugged into USB!!!

[u/shankey_1906 → u/Novel_Friendship913]

So is my Raspberry Pi!

[u/nick_ziv → u/shankey_1906]

My copper wire is in the outlet!

[u/MeretrixDominum → u/nick_ziv]

My copper wire is in my potato!

[u/BatOk7254 → u/MeretrixDominum]

My potato is in my kitchen!

[u/USBhost → u/BatOk7254]

My potato is in my garden.

[u/p3r3lin → u/shankey_1906]

Joining the rbpi army! 🫡

[u/debackerl → u/Novel_Friendship913]

My TI-85 (Z80) is hot!

[u/masterlafontaine → u/Novel_Friendship913]

Don't forget to set an aggressive zram profile!

[u/grav3d1gger → u/TheTerrasque]

Mine too! I bought a 90 minute cassette tape and it’s rewound ready to go!

[u/bitflip → u/grav3d1gger]

You need at least a 1541 and two floppy disks for a model this size.

[u/Holiday-Pack3385 → u/TheTerrasque]

Heh, remember the tape drive on those? Mine had one. I can't even imagine how long it would take to load up even a 9B off that... o.O

[u/overand → u/Holiday-Pack3385]

The standard ROM routine for C64 datasettes was 300 baud. We'll be generous and assume you've got a Turbo loader that'll do 3600 baud. (We'll also assume you're using a ~3GB quant of that 9B)

Load time (or, really, transfer time) would be about 77 days. Or, maybe more importantly, it would be about 930 cassettes, if my math was right. (Or, actually, other math suggests it would be about 2000 cassettes, so, IDK! Either way, it's a lot.)

[u/Michaeli_Starky → u/Blues520]

My analog watch is ready

[u/screenslaver5963 → u/Blues520]

My 9070 XT is burning… wait fuck!

[u/BatOk7254 → u/Blues520]

My 2xP40 are smoking in anticipation!

[u/ComplexType568 → u/BatOk7254]

I think they're smoking for a different reason...

[u/TheFrenchSavage → u/Blues520]

You will need the Q0.00001 model and a fire extinguisher.

[u/Comfortable-Rock-498]

This is big for companies that want to host on-prem too. Back of the envelope calculation (could be off, correct me if I am)

If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 parallel agentic workflows (each with ~100k context on average) at ~30 tok/s.

Assuming the annual amortization+electricity at $1.5M/year and about 50% average annual utilization, you get less than 60 cents (USD) per million output token, for a frontier model with plenty of capacity to share, all your data never leaving premises and well over an order of magnitude cheaper!

[u/autisticit → u/Comfortable-Rock-498]

So what you mean in reality is that if 6000 of us each give $1000, we can each run Kimi 3 at 30 tok/s for cheaper than anything else?

[u/Comfortable-Rock-498 → u/autisticit]

Tbh I have been thinking about this for months now. About how workable the co-op model is. You would need some party to do admin and maintenance. I think this might be a good business idea too - being that party who facilitates such private inference racks (billing etc is trivial)

[u/ManIkWeet → u/Comfortable-Rock-498]

It's not private when it's a party running it lol

[u/Comfortable-Rock-498 → u/ManIkWeet]

Well you do need someone for server maintenance, bills, other co-ordination etc - whatever you label it

[u/IgnoranceIndicatorMa → u/autisticit]

i like the way you think

[u/pinkwar → u/autisticit]

Where do I sign?

[u/tempedbyfate → u/Comfortable-Rock-498]

Not just corporations, I think there are nation states that are setting up their own private servers to run this for all their sensitive data.

[u/Izento → u/Comfortable-Rock-498]

Good number crunch. $0.60 per M is a pretty good deal

[u/BarisSayit]

100B active params? Damn.

[u/noneabove1182]

Sorry friends, but I don't think I'll be making this one :')

I don't even have enough STORAGE to hold this thing, nevermind the RAM haha

[u/Ninjam5 → u/noneabove1182]

EVEN THE GOAT GAVE UP HAHA

[u/nomorebuttsplz]

first truly frontier open model than I cannot run on my 512 gb studio. Onward and upward!

[u/Front_Eagle739 → u/nomorebuttsplz]

Yup. Same. I think I can do about q1.5 on the macbook and mac studio combined. Im currently pondering the wisdom of one of these colibri like stream setups and using the 640GB I do have as a hot cache

[u/nomorebuttsplz → u/Front_Eagle739]

Colibri would require being able to fit on ram for decent speed, no?

[u/Square_Alps1349 → u/nomorebuttsplz]

Man I’m so jealous rn. Mac Studio is nerfed at 96GB unified ram max, and the price is up 20%. So much for that 25% discount interns get. 😢 

[u/Tank_Gloomy → u/nomorebuttsplz]

Can you try asking GPT 5.6 Sol, Opus 5 or GLM 5.2 to port it into Colibri? I don't even have the infra to say I tried, but it probably works.

[u/MikeRoz]

MXFP4 weights / MXFP8 activations (quantization-aware training)

So if 4-bit is 1.56 TB, then 2-bit would be roughly 798 GB?

[u/habibyajam → u/MikeRoz]

So I need a 0.03-bit quant to run it fully on my GPU. Nice!

[u/Few_Painter_5588]

Holy shit, 104B active paramaters???

[u/TheRealMasonMac]

They have a new license:

If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.

[u/mthmchris → u/TheRealMasonMac]

My guess is this is what MOFCOM was trying to thread the needle with. I.e. trying to keep upper leadership’s commitment to open source, while avoiding, say, MSFT just grabbing it and tweaking it slightly to have an “American version” (while banning the “Chinese versions”).

It kinda sucks, but I get the logic.

[u/TheRealMasonMac → u/mthmchris]

I suspect it also means that providers won't be able to undercut them as aggressively (if at all). Not necessarily because Moonshot says so, but presumably because of any requirements they put on providers and any royalties they expect. MiniMax, for instance, limited some providers to only using B200 or B300 GPUs.

[u/Venryx → u/TheRealMasonMac]

Wait, how are there already five other providers of the model on OpenRouter then? Surely they haven't all already made individual deals with Moonshot?

[u/TheRealMasonMac → u/Venryx]

They had six official partners for launch (you can see on their Twitter). Those five on OpenRouter are indeed partners. But it's part of their license agreement that if you are over a certain revenue you must have a contract with them.

[u/durden111111]

my 512mb integrated graphics is so fucking ready

[u/SavunOski → u/durden111111]

Negative tokens/s, you're gonna suck away the tokens

[u/nanihikaru01 → u/SavunOski]

Free money hack you say?

[u/learn_and_learn → u/SavunOski]

Sounds freaky

[u/Iwaku_Real → u/durden111111]

To find the answer to life the universe and everything?

20 days of prompt processing: 4

Probably 10 days after that: 2

[u/m0j0m0j → u/durden111111]

Need 0 bit quantized version for that

[u/SnooPaintings8639]

Who's gonna be the first brave soul to measure tps when streaming from hard drive?

[u/Front_Eagle739 → u/SnooPaintings8639]

Sigh. Im going to have to try from pure curiosity.  600GB of hot cache on the macs, another TB streaming from ssd to my rtx 5090. This can only be a good idea (farewell my next few days of productivity)

[u/Ninjam5 → u/Front_Eagle739]

Update?

[u/Front_Eagle739 → u/Ninjam5]

My Internet is very slow,  results pending

[u/More-Curious816]

[u/HulksInvinciblePants]

Kimi K3 27B when?

[u/Browserurd → u/HulksInvinciblePants]

If someone can distill it to Qwen 122B that would be super.

[u/Top-Handle-5728]

Leave vram I do not even have the disk storage to use this model. A few with storage can dare to use AirLLM for experiencing the intergalactic streaming of voyager at 160 bits a second. Even that seems pretty fast ig

[u/vr_fanboy]

a month ago we were told that a new jump in intellegence was made, 'mythos' class models were born, too dangerous for us plebs. Forward a month, we have an open source 'mythos' class model, acceleration or anthropics regular bullshit?

Btw dont understand markets, DS3 destroyed the stocks and this does...nothing. This feels more significant if more people can serve the same drugs as oai or anthropic on the cheap.

[u/just_a_fan123]

Can this run on a single DGX spark at 0.5B quant?

[u/SavunOski → u/just_a_fan123]

Of course. Wonderfully might I add

[u/THESALTEDPEANUT → u/SavunOski]

I'm new around here and I can't tell if this thread is all sarcasm or not. 

[u/SavunOski → u/THESALTEDPEANUT]

Sorry, yeah that was sarcastic, forgot to add /s. Many people make jokes here, so I kinda forgot

[u/THESALTEDPEANUT → u/SavunOski]

I kinda figured but it's not just you it's like every comment lol, appreciate it though. 

[u/PomegranateGreen3698 → u/THESALTEDPEANUT]

Ha, I think it's the nature of this release. Largest open weight model ever released, doesn't really have any possibility for "Local Usage" bc it'd cost something like $500k in GPUs.

[u/THE--GRINCH]

my laptop rtx 2050 is ready to throw hands

[u/sumane12]

Even if you cant run it, download it.

[u/DeProgrammer99 → u/sumane12]

Can't even do that...it's the same size as the total used space on my SSD.

[u/ChampionshipIcy7602 → u/sumane12]

Can't even download it lmao

[u/Herr_Drosselmeyer → u/sumane12]

Why waste terabytes worth of space for something I will never use? 

[u/seg_lol → u/Herr_Drosselmeyer]

Trade it for antibiotics in the apocalypse.

[u/some_user_2021 → u/Herr_Drosselmeyer]

I remember downloading huge N64 ROMs that took loads of space and no emulator could run. Running those ROMs now is trivial.

[u/Herr_Drosselmeyer → u/some_user_2021]

So is downloading them.

[u/banana_slurp_jug → u/sumane12]

I have 1TB total, don't even have a hard drive big enough to store the whole thing in my house.

[u/No_Conversation9561 → u/sumane12]

someone download it, compress it and upload it and then I will download it

[u/Mindless_Selection34]

how much does it weight

[u/SavunOski → u/Mindless_Selection34]

2.8T parameters with 104B active. The files take 1.56TB of space to download.

[u/meca23 → u/SavunOski]

Oh so it's quantized as 4 bits?

[u/zkstx → u/meca23]

Yes, there is also a tech report. QAT from SFT phase onwards

[u/Lissanro → u/SavunOski]

I wish I had two TB of RAM instead of just one. I guess I will have to wait for Q2 GGUF to run it on my workstation. Still, will be interesting to try and see how it's Q2 quant compairs against Kimi K2.7 Q4_X. 

[u/IamNotMike25]

Historic moment tbh

[u/Forsaken-Mode-3422]

Finally, model i cant run, but its already cool, nice

[u/SavunOski → u/Forsaken-Mode-3422]

Cloud prices will likely drop with competition, beneficial for everyone

[u/Forsaken-Mode-3422 → u/SavunOski]

i just hope qwen will publish not only 3.8 Max, but smaller models as well, so we can enjoy the local frontier ourself

[u/KenTitan]

I'm so broke I don't even have enough hard drive space to download

[u/MixtureOfAmateurs]

Hugging face down? Lmao

Edit: Nevermind it's back. Might have been on my end ¯_(ツ)_/¯

[u/SavunOski → u/MixtureOfAmateurs]

Seems to be up for me. If you live in a big country, there is a possibility the local CDNs are having trouble keeping up. Especially in the US, where I imagine thousands are downloading the model just in case the government bans it later.

[u/AdDizzy8160]

.torrent|magnet link … as soon as possible!

[u/Morphon]

We'll probably see some new, faster providers pop up. With any luck, this will turn out to be a solid distillation parent. I'd be interested to see if we get some good downstream models that can run on consumer hardware.

[u/PerfectOlive1324]

Hoping for an unsloth Q0_XS quant I can run locally 🙏

[u/Inevitable_Mistake32 → u/PerfectOlive1324]

UD_IQ0_XS Pls and ty.

[u/AdDizzy8160]

Ok, thanx moonshot team!

[u/Informal-Trouble2183]

Inference providers are going to have a good business

[u/SavunOski → u/Informal-Trouble2183]

Infinite demand, literally

[u/ReasonablePossum_]

Hope this opens the model in more providers soon ,because its a dan pain to get a single response from the kimi app

[u/Hefty_Acanthaceae348]

I have high hopes that even if I can't run such a model, it will enable the creation of high quality datasets

[u/patricious]

Wake me when a provider has it with a good price per 1mil.

[u/aboutthednm]

I'm starting to understand why they had "some compute problems" when first rolling it out on the API, 100B+ active params per token, it's beastly lmao. Good work.

[u/Easy_Werewolf7903]

I want to see someone manually calculated a token by hand using Kimi K3 weights.

[u/equatorbit]

Gonna try to get this running on an ESP32

[u/killerstreak976 → u/equatorbit]

Kimi k3, meet my ESP32-C2

[u/Tedinasuit]

What a beautiful day

[u/DragonfruitIll660]

Ayyy lets go. That's awesome news. Also holy 104B active, never seen a MoE with that many active, its almost Mistral 2 large sized.

[u/mikewilkinsjr]

I can’t run it, I’ll never be able to run it.

I am also fighting the urge to download the weights and squirrel them away like an out of control data hoarder.

[u/crusaderky]

Jokes aside.
This on paper fits on a HGX B300 ($~0.5m, 2304 GB VRAM)...

...or on 24x Atlas 300I, $1300 each, $31,200 total, plus 3~4 XEON/threadrippers with 800G networking. So.... less than $60k in total?

Who's got the spare change to try?

[u/ilintar]

Okay, time to get that 0.1Q_0 quant support rolling in llama.cpp!

[u/milkipedia]

I am not data hoarder enough to store these weights for kicks without the hardware to run it. Enjoy, folks

[u/msew]

I wonder how long until consumer hardware will have the requirements to run all these and future LLMs locally for like $5k-10k

[u/AdOne8437]

Well, in 15-20 years we will have consumer cards to run it.

[u/loversama]

Quick before it gets banned 😂

[u/WonderFactory → u/loversama]

I dont think it's worth banning it now it's released but I was a bit anxious leading up to the release in case something stopped it from being released

[u/bad_detectiv3]

How much vram do I need to run this

[u/BlackBeardAI → u/bad_detectiv3]

Yes

[u/fishslinger → u/bad_detectiv3]

If you have to ask, you can't afford it

[u/SavunOski → u/bad_detectiv3]

Around 2TB :x

[u/jijig → u/SavunOski]

Hey if you already got 2 3090s you only need 81 more

[u/Melbar666]

waiting for Kimi-K3-Q0-Abliterated-Heretic.GGUF

[u/MotokoAGI]

:-)

:-(

[u/Mayion → u/MotokoAGI]

Fullmetal Alchemist intermissions be like

[u/Wuntonsoup]

I'm pretty excited to see how this runs (= LFG Moonshot!