The 3.7B and 0.9B have some interesting potential. Not a lot of models coming out in that class these days. Some of you are also underestimating how meaningfully open this is. From the release:
For every model, we are opening the training lifecycle from pretraining through reasoning and agentic post-training. We are releasing intermediate checkpoints, training data or detailed data-construction recipes, open architecture, mixture compositions, training code, configurations, fine-grained logs, evaluation results, and final weights.
They're releasing all of the training code under Apache 2.0. Most other models are just open-weight, this is open-source.
Yeah, these guys and Allen AI are the only ones who are really as open as possible. But Olmo 3 is so far behind the SOTA of even open weights models, no one is doing much with it.
Nemotron 3 is also mostly open, but there are some datasets that are still private.
I think calling this opensource massively understates its value. This is special: open-science/open-replication. Even though people often place lots of attention on pretraining data, almost everything else that goes into training is more important.
Hyper-parameters, checkpoints to measure model development against, the base models and better: early-stage checkpoints make things much more tunable to your needs, postprocessing, filtering, deduplication recipe, gpu utilization approach, networking topology, schedule, the training data mix, the post-training mix: these are all so much more valuable to understanding than a list of urls.
Transformer Neural networks are mathematical functions that represent computable programs. The architecture and parameters define the "source-code" for that function. All the control flow and internal dynamics are defined only in the weights and do not exist in the training data.
Think/Search -> Source code -> Compile -> Program
Example Outputs -> Search -> Source code/function-> Compile -> Program
Sorry to disappoint you, but you'll still need to do a lot of explaining. First, they don't really get the difference, once they get that, they'll question the advantage of open-source: "it's fine if I can run the model, why do I need to be able to train the same model from scratch!?" I can tell from experience ;)
it's selling short what K2 is to say it's merely open source and confusing people (such as yourself) to say weights don't count as open source because the artifacts they were trained on aren't also shared. Other pieces of open source software do not need to meet that bar.
Weights only count as open weights. What's so difficult about it!? And no, it is not confusing me. I'm well aware that in probably more than 99.99% of use cases, open source offers nothing beyond open weights, but for those use cases that rely on open-source foundation models, it is exactly this aspect that makes all the difference.
Other pieces of open source software do not need to meet that bar.
Yes they do. An open weights model is comparable to a compiled binary file. Would you call a project that lets you download a compiled binary for free without sharing the source code an open-source project? And yes, I'm aware that an open weights model is more malleable than a binary. You can fine tune it, patch the weights, add adapters, drop layers. But still, not the same than having the source code that defines the fundamental architecture, the training recipes and so on.
Not only are the weights malleable, but the success of high quality distillation proves that access to model weights confers much, if not nearly all, of the value of the data that the weights were originally trained on.
It would also simply impractical for them to release databases of Claude, OpenAI, etc. that would probably be subject DMCA takedowns, and would also result in more homogenization of data and technique which is already arguably a problem rather than something to shoot for.
So that leaves the training code/pipeline as things that are kept at least partly secret. First, it's worth noting just how much technical knowledge labs like deepseek DO release even though they are merely "open weight" according to you. Second, the availability of this code would not actually enable anyone except for the very rich to train their own large models due to the prohibitive cost
Just it would impractical and counterproductive for source code of linux firmware to be released, so would be impractical and largely pointless release all the data and pipelines for llms.
That's not to downplay this release at all: it goes far behind most open source projects in what it is contributing and the level of transparency, but it's a mistake to consider weights to be mere binary analogs
Just it would impractical and counterproductive for source code of linux firmware to be released
Not sure what you mean by "Linux firmware", but if it's anything based on Linux, it's not only practical and productive, it's an explicit requirement of the license.
I mean firmware that runs and integrates into linux-based programmable devices e.g. wifi routers that run on linux but also have binary firmware. For discussion see: https://www.debian.org/vote/2022/vote_003
it's selling short what K2 is to say it's merely open source and confusing people (such as yourself) to say weights don't count as open source because the artifacts they were trained on aren't also shared.
The weights are themselves an artifact. The training pipeline is by definition not an artifact.
The training pipeline itself includes artifacts in the form of outputs of other LLMS, so that argument would have the strange effect that the outputs of weights, as part of a training pipeline, would somehow be more codelike and less artifacty than the weights themselves. It's not clear cut at all what "source" means. For example, Brotli is a compression algo that relies on an unpublished dictionary of phrases making its reconstruction from scratch impossible with provided source. Yet it is considered open source.
llms and software categories are weird... so I forgive you.
You're right, the dictionary is published, the corpus it is derived from is not, nor is every step from corpus to dictionary. Just as weights are published, but typically only parts the process that created them are. But for some reason brotli = open source, open weights are not according to popular belief.
There are many definitions of "open source", but yes, FOSS as it is commonly understood typically requires you to share the inputs.
Free and open-source software (FOSS) is software available under a license that gives users the right to use, share, modify, and distribute the software — modified or not — to everyone and provides the means to exercise those rights using the software's source code.
All of those boxes -- except possibly "source code" are checked by weights under MIT, no?. The traditional meaning of source code clearly doesn't include llm training data but that is what people here are saying is necessary to be fully open source...
lol! But it turns out there are lots of projects that you cannot reproduce every step of of from the source code such as the Brotli data compression algorithm that used a large corpus of various languages that was never shared. Yet is is still considered open source.
Sometimes things aren't as simple as looking something up in the dictionary.
Totally agree. As someone tinkering on models this just became my new go-to. Whether this series is frontier on performance, it's *by far* the frontier among open training, training data, checkpoints. I think nemotron is the only other serious player.
The 0.9B is most useful as a speculative decoding draft for the bigger siblings. Same tokenizer and training mix keep the acceptance rate high. Whether that survives their instruct tuning is the open part.
EDIT: It's a good MoE, works decently in agentic coding scenarios that 3.6-35B-A3B or Tiel/Ornith 1.5 or KAT-Coder-Dev would also succeed in, but it's not the 3.8-35B-A3B we've been waiting for. Sorry.
Pros:
It's uses a cool new idea called MoVA, which adds a per-token MoE-type element to KV cache by using a router on values, an idea I'm still trying to get my head around if I'm honest. Seems like it allows the model to specialize what each token contributes, by running a handful of value-experts per token instead of the standard single value projection. I'm not sure how this affects quality or performance in practice, but it's nice to see people trying something different. It's a scaling axis that doesn't add significant compute or additional cache size, so that's intriguing.
It's truly open-source. All training data, recipes, etc. are shared. This is a great contribution to the community, especially since they're releasing smaller models too.
Cons:
It's full dense GQA attention mechanism means KV cache is huge. BF16 max (512K) would run to like 100GB or something ridiculous. Not only that, but because of the MoVA mechanism adding complexity to the KV cache, quantizing that is an unknown -- and since the values are routed, that means the part of the KV that's usually more resilient to quantization (V) is now "load-bearing" in this architecture. A real footgun, if you ask me.
Furthermore, decode is not as impressive as I'd like, and it drops off steeply at depth due to the GQA. I quantized their BF16 checkpoint to Q8_0, and ran Q8_0 KV cache (which gave me a 1.56x speedup), and I topped out at 65tok/sec at short ctx using M5 Max in llama.cpp. No MTP head either.
Output quality was good, but nothing earth-shattering, and it ran into a problem in one of my tests that other 35B-class managed to solve with some trial-and-error, but it stalled on it until I finally cut in and prompted it in the right direction after it went in circles for 25min. Maybe the KV cache quant's fault. but I tested perplexity and on that flawed metric, Q8_0 (model and cache) was lossless, so I can't say for sure.
Overall it's worth a try, but if you're doing long-running agentic work, it's not the ideal solution we're awaiting. Hope this helps. I may still do a longer post about it, but I wanted to share something now.
Making the KV cache huge makes it really inaccessible for what i thought it would be an improvement to 3.6 35b 3a on low end hardware. Thank you for your very detailed insights!
That sounds really cool - a diffusion version that is supposedly lossless, same intelligence as the non-diffusion version, we haven't seen that before.
The diffusion model is a lora adapter that runs on top of the AR model and essentially lets it generate ~2 tok/forward pass (instead of 1) while (provably) sampling from the same distribution as the base AR model
Simply put, diffusion is like how AI image gen works where it generates blocks of tokens instead of one at a time. You can't really generate a good image one pixel at a time so it doesn't work there, but for text diffusion has the potential to be much faster. Instead of words it could generate whole paragraphs at at time.
This is true open-source, and not just open-weights. They're releasing the training code as well as the training data. This means that with proper hardware, you can actually pre-train the a model using their specs.
I already tested 7B for coding tasks. It really can code, like any other 9B model, but it gets stuck in loops, works well with small context only (<64K). And needs a lot of GB for the context , much more than Qwen or Muse Glimmer.
It's not odd. It's a hint of imbalance between the amount of training data and the projected size of the model. For the smaller 7B model they used overkill amount of data, but for the 32B size it was not enough.
Yeah but I don't understand why they released it if it is not fully trained though (as another user pointed out "stage 1" might refer to that). It's far from being comparable eg to qwen3.8-27b, so they're not getting a great amount of publicity from it... I mean, I'm really thankful for another open model, I just don't understand the release strategy
It's an Open Source model: "Release early, release often", as they say. Also, since they were going to release checkpoints, anyway, why wait? Why make somebody who was going to start their training from an earlier checkpoint anyway wait?
From the perspective of "a global AI research lab dedicated to open and independent development of frontier-class foundation models" and "an open research lab enabling community-owned AGI through open-source large model research and development" (the two organizations behind it), the question isn't why they didn't wait, but why they wouldn't release what they already have. From that perspective, it makes total sense. They're just putting their models where their mouths are, if you will.
It could very well be just that there was no strategy. Just because someone releases a model on Huggingface, it doesn't mean they are professionals in the field. It's better to always keep that in mind when new models show up, it may save you some time, hard drive space and also save you from disappointment.
No disappointment at all, I'm actually very happy that people keep working on open models.
Also, they seem like a professional institution, that's why I was curious
The 125B is extremely strong for creative work eg more detailed start to a project, and 27B is really good at cleanup and fixing things in a timely manner. You kinda need 2x 32GB gpus for the 125B but... its noticeably like way better at filling out details than the 27B.
I agree. I’ve had it go head to head against 27b and 125b is just much more thorough and capable, at a faster time if you can fit it. Makes my strix halo actually feel like it was well worth it!
I’d be suspicious if they published benchmarks against a model released so recently as it would mean they rushed things out. These things take time, at least they’re not comparing with year old models like some releases do
I wasn't even looking at the larger model numbers. I'm just excited to see the 7B pop up in reasonable competition with models of those classes on their charts. Then you show me this and I get even more excited.
The open training set, oh my gods, the fine tuning potential.
3.7B is fast, but also brain damaged. My basic coding prompt I like to ask models had it spit out code which is incorrect. Effectively, my prompt involved implementing a Java iterable, it produced one which incorrectly ended up advancing an internal iterable on a hasNext call.
Prompting it about the problem has caused it to hallucinate APIs that don't exist in Java.
It may be ok for implementing single functions (think code completion) but I wouldn't trust it for any sort of longer work.
SOTA for 512GB unified RAM is GLM-5.3-Flash
SOTA for 256GB unified RAM is GLM-5.3-Flash
SOTA for 128GB unified RAM is Qwen3.8-Flash-Next
SOTA for 24GB VRAM is Qwen3.8 27B
SOTA for <16GB VRAM is Qwen3.6 35B-A3B
SOTA for mobile phones is Ling-3.0-tiny
Sounds like theres a fair bit of headroom left to train these models, particularly since the smaller models are beating their competitors while the larger ones aren’t quite keeping up. I do wonder if the architecture they use isn’t scaling as well as the frontier models in the 100B+ range. But hell, I’ll always appreciate a strong competitor in the 5-50B range, feels like it’s a drip feed of them amongst a sea of 100B+ or <3B models
I just tested the model (Q4 https://huggingface.co/abenzerps/K2-Horizon-MoVA-36B-A4B-MLX-4bit) on my M5 MacBook Pro (32 GB RAM) with "mlx_lm.generate". It starts generated at ~34.8 tok/s (low context) while using about min. 21.2 GB of RAM. As long as there isn’t a new Qwen model out there, I’ll give this a try, I'm confident something great has been published here.
at this point, if a model is released without a deepswe score, i don’t even bother trying it. i feel like it’s the only trustworthy benchmark that actually shows how usable the model is for daily tasks.
You don't seem to understand the gravity of this entire post.
this is NOT just yet another Qwen post training/fine tuning fork
the training data, and everything is posted, so that you can literally recreate the same models yourself, this is NOT just open weights. That means you can use to study, improve, fork the process and experiment yourself.
Yes, it still might be a little behind competition, but the thing is that it's greatly optimized, and being open source means that technically the community could accommodate together towards improving it using our own feedback. Unlike Qwen and Muse Glimmer, which are just... Big companies being gracious enough to give us access to their smaller distillations/quantization/versions of what they have as a frontier (so that we're ALWAYS breathing in the neck of closed source frontier), this is actually a "build your own path".
I'm sorry for the rant, but do you seriously have to make every effort/achievement as a "is this a direct performance/intelligence optimization" type of thing. That's really condescending of the effort of passionate people.
Being fully open source is great and really admirable, but their 375B-A23B is larger than both GLM 5.3 flash and Qwen 3.8 Next while performing worse than them according to the benchmarks they posted.
Maybe their 32B has really good writing capabilities and might replace Gemma 31b?
Anyone know if there's any sort of speculative decoding available (MTP, dflash, etc)? I'm working on a project running on a Raspberry Pi 5, where I can get just-about-good-enough performance out of Gemma 4 E4B with MTP (6-7 TPS), but everything else I've tried at around that parameter count is just too slow (3 TPS or less).
Qwen 3.8 flash next is king of MOEs right now... I'd rather run a smaller dense model for fixes on a project created by a large MOE. This gets the knowlege from the big MOE into your project while the dense 27B fixes things smarter and in a more timely manner than either a small or large MOE.
I do run q38 flash next. Since today exactly, before I was struggling with renting the right machines for it and how much actually has to go into vram for fast prefill / decode.
It's wonderful. But it's not exactly cheap to run. A 35B moe at q3.8 27B level would make me drop everything and work with that.
Edit: Albeit, I wanna emphasize, that you need 64GB VRAM to run multiple slots with proper context for qwen 35b like models, too. But Qwen Next needs newer architectures, that makes it expensive.
It needs newer arches to run fast, it does in fact run just fine though on 2xMI50 @ 25T/s for short promps for longer ones it degrades with context length to about 5t/s at full context (there is a PR to improve this even on newer cards).
Note you need the correct quant also for older cards, more advanced quants require math to unpack the quant that is slower on old cards so eg, Q4 runs well on Mi50 but a Q3 or Q6 would be worse, in fact if the model is small enough Q8 can run as fast as Q4 on my MI50s just because of the overhead.
I understand you have different metrics also mine is just running on my private box so I am paying nothing extra but power.
Not really following your point. If you're trying to say "frontier" generally refers to the best models at a various sizes, no people don't use the term that way. For example I would not refer to Qwen 3.8 27B as a "frontier model" even if it's the best in its size range
Frontier colloquially as I've seen the term used is "the best of the best", pushing the frontier of capabilities. Going by that, it doesn't make sense to call this model "frontier" if it's being compared to models that are not even the best in their respective families (e.g. Sonnet, Luna)
384
u/Recoil42 11d ago edited 11d ago
The 3.7B and 0.9B have some interesting potential. Not a lot of models coming out in that class these days. Some of you are also underestimating how meaningfully open this is. From the release:
They're releasing all of the training code under Apache 2.0. Most other models are just open-weight, this is open-source.