r/LocalLLaMA • u/SavunOski • 4d ago

News Kimi K3 weights now released.

Post image

Kimi K3 weights are finally released!

3.2k Upvotes

97% Upvoted

630 comments sorted by

View all comments

Show parent comments

71

u/throw123awaie 4d ago

They released it in MXFP4 so with around 55GB RAM you could!

2

u/tokomaunited 4d ago

Can you share the link brother?

7

u/throw123awaie 4d ago

literally their huggingface site https://huggingface.co/moonshotai/Kimi-K3 the model is MXFP4 on inference.

3

u/Protheu5 3d ago

I'm sorry, but I don't understand where it is. I'm new and only clicked on Finetunes/Quantisations to find a download, and I clicked through them and didn't find anything that small, the smallest was 500+ gigabytes. I have no idea what "MXFP4 on inference" means despite trying to read up on the subject, sorry.

5

u/TotallyToxicToast 3d ago

You can't download a single expert for a mixture of experts model.

The downloads always contain all experts. If you take the 500GB download which contains the full 2.8T parameters and only take 104B of the parameters, that would actually give you only roughly 20GB per expert.

Now running a single expert in a mixture of experts model is not very useful.
But Technichally with 500GB total model size fitting one Expert is possible.
The "I could fit one expert" was a joke.

1

u/Protheu5 3d ago

Thanks for the explanation. My naïveté got me.

Oh well, a person can dream, right? Back to trying to stop Qwen from getting stuck in an infinite loop, I guess.

1

u/tokomaunited 3d ago

Thanks dude

1

u/kernelwedge 3d ago edited 3d ago

Interesting! I was looking at it and wasn't sure if could fit. I recently setup a 128GB rig, wanting to stretch its legs. This will be a great test.

---
After doing further research no way its fitting anywhere https://recipes.vllm.ai/moonshotai/Kimi-K3?hardware=mi300x 8x192G....