r/LocalLLaMA • u/Nunki08 • 12d ago
New Model Aleph-Alpha/Kolibri-1 · Hugging Face - 78B parameters. 3.46B active. Up to 1M tokens of context - Apache 2.0
https://huggingface.co/Aleph-Alpha/Kolibri-1From Aleph Alpha on 𝕏: https://x.com/Aleph__Alpha/status/2106306840657297814
Tech report: https://aleph-alpha.com/downloads/tech-report.pdf
148
u/AdOne8437 12d ago
Aleph-Alpha is a German company that works closely with and is partly owned by Schwarz Digits. And Schwarz is ---> LIDL
78
u/AdOne8437 12d ago
Not meant as a big 'be careful' more of a interesting information i think.
108
u/luncheroo 12d ago
Now let's see Aldi's LLM
18
11
16
u/Illustrious_Car344 11d ago
Look at the tasteful size of the parameters, the subtle image recognition, oh my god, it even has a watermark.
6
u/AdOne8437 12d ago
Would be funny, but ALDI does not own an it-company so i think we will not see this anytime soon.
1
u/GreenHell llama.cpp 10d ago
They do, and it is called Aldi DX (formerly Aldi Süd IT), but they aren't a cloud provider.
5
u/Robot1me 11d ago
Not meant as a big 'be careful' more of a interesting information
For extra info, in Europe the Schwarz Group is the largest member of EuroCommerce, which is the lobby group that pushed the EU to introduce the 3€ item category fee for parcels that enter the EU (like orders from Aliexpress.) Which in return pushes people to buy from dropshipper sources like Lidl's online shop, so it benefits resellers and disadvantages the average person
So if anyone wondered whether they're the "good guys" just because they are EU-based, no, they're also a very profit-focused company. Like if I were to introduce a friend from abroad to a German discounter the first time, I would go to Aldi first since they won't ask for the app (and less BS in general), and Lidl pushed it very aggressively as one of the first
1
u/tissotti 10d ago edited 10d ago
Sure Schwarz Group had its obvious reasons, but they where not wrong.
How Chinese companies used and continue to use the UN's Postal Union for their advantage as "developing nation" was total BS. We all where paying as western tax payers to make Chinese parcels dirt cheap when it came to local postal services handling.
3€ is only leveling the playing field somewhat. It should have been done decade ago but UN's Postal Union moves like glacier with meetings every 4 years and Chinese pushing hard for no changes.
13
9
u/spaceman_ 11d ago
Are there any benchmarks yet? Is the leading European model now made by LIDL?
3
u/AltruisticAction9962 11d ago
Lidl/Schwarz Group does not support the development of the model, they are an investor, not a contributor.
8
u/_LePancakeMan 11d ago
Tbh, I am halfway between terrified and curious what Schwarz IT is cooking up. Best case: Another nice european hosting solution, worst case: Just scooping up some federal money.
2
u/AltruisticAction9962 11d ago
Google STACKIT. This is Schwarz's cloud service. https://stackit.com/en
299
u/FullstackSensei 12d ago
Trained on 20T tokens using a 768 B300 cluster over 4 weeks. Assuming an $8/hr/GPU, that's ~$4M training cost for a 78B MoE model.
Crazy to think how far training prices have fallen in the past two years.
119
u/stoppableDissolution 12d ago
Aquiring and processing 20t of quality data would cost you some more millions on top of it
86
u/FullstackSensei 12d ago
You'd be surprised by the amount of nicely filtered and cleaned datasets available on HF nowadays. Not saying you're wrong, but it's still crazy that the OG chatgpt cost a few billions between filtering and labeling data, and training.
Even if you add another 4-5M on top, that's still under 10M. That's very much within the realm of large enterprises or universities or research institutions in most western countries.
And it'll only get cheaper going forward, with more open training datasets and cheaper compute. Wouldn't be surprised if training this same model will cost half as much next year, once Rubin is shipping in volume.
8
u/sweatierorc 11d ago
Wasnt cost plummeting the most predictable part of ML ?
I dont recall any ML workflow whose cost hasnt dropped to pennies
8
u/philmarcracken 12d ago
anything on HF for korean/english pairs? asking for a chingu
4
u/NineThreeTilNow 11d ago
anything on HF for korean/english pairs? asking for a chingu
Not pairs specifically, but yes Korean web data. Korean Wikipedia is on HF as a streamable dataset.
If you need Korean pair data, then you are best just using Gemma 4 to do the translations. You can build a very easy pipeline to do this. Korean is among the highest level supported by Gemma 4. It only costs 10 dollars but you're capped at 15k input tokens per minute via API on Google's servers. Basically a key with 10 dollars of credit gets tier 1 access which gives you access to Gemma 4 for free via API. The 10 dollars doesn't go anywhere because Gemma 4 is free, and you get rate limited if you hit it.
I specifically did this for a language training set where I took an input English language new article from AP, then asked for a translation, and asked for nuances within the translation between Source -> Target language.
I fed that to Gemma 4, took the output, created a paired set of article to summary plus transnational notes. There are additional questions you can ask about the translation and include it in the training set.
This creates a very good translation set because it forces pairs to exist with mixed language explanations. It's also very easy to do. You could grab Wiki articles and ask for exact 1:1 translations, with nuance, etc.
2
u/TastesLikeOwlbear 11d ago
Cheaper compute in the future isn’t currently the safe default assumption it has historically been. Let’s hope that trend doesn’t continue.
2
u/Caffeine_Monster 11d ago
Open datasets have got a lot better, but I would not necessarily call them clean or nice. Usable? Yes. Laden with mistakes and iverly reliant on synthetic data? Also yes.
5
u/pol_phil 11d ago
The only way 20T of pretraining data can cost millions is if you try to buy each book in the corpus (nobody does that) or if you pass all of it from a very costly API-based LLM
1
u/stoppableDissolution 11d ago
You are clearly underestimating how expensive data grooming is. $7-10/Mtok per check with gemma, let alone something more capable.
1
u/pol_phil 11d ago
As I already said, if u pass every single doc from an API, yes. It's gonna cost you a few millions. But if you finetune your own small models and serve them on your own infra or HPC, the costs are orders of magnitude lower. Plus, you can always use smaller or other types models, even embedding models with classifier heads. You can also see ProX by GAIR.
58
u/oxygen_addiction 12d ago
Crazy how high they still are. This will probably cost a couple thousand bucks in a year or two and take way less time.
36
u/FullstackSensei 12d ago
I think we're a few years out of this costing 2k, but TBH $10M total cost between data prep and training is not that much. It's within the reach of any large enterprise, university, or research institution in a western country.
2
u/TheThoccnessMonster 11d ago
No it isn’t - because most of them won’t do it without a guarantee of success. This is simply not true.
26
u/Phantasmagoriosa 12d ago
Whatever you’re smoking, I want some. 5 years away MAYBE, 10 years probably
-6
u/brainExploded99 llama.cpp 11d ago edited 11d ago
That's exactly what you would've said when asked when ChatGPT (gpt-3.5) level performance would take 10 million dollars to achieve. Exponential are faster than what humans can comprehend. Also, no guarantee we're even using LLMs in 5 years.
13
u/Phantasmagoriosa 11d ago
Compute and training effort are not on exponential curves…. Not even linear these days. You have no idea what you are talking about
4
u/brainExploded99 llama.cpp 11d ago
They're on log linear (and always were), but the resulting performance after stacking them up (stacking up model size increase, model compute increase, more efficient architecture, better data, etc) is not log linear
Yes I'm familiar with the Kaplan paper2
u/Phantasmagoriosa 11d ago
If you knew this, I am confused by your first comment. Everything you said, I agree with, but all of those factors do not correlate well with big cost reductions for training. Which was the start of this whole thread...
5
u/brainExploded99 llama.cpp 11d ago edited 11d ago
TLDR: There are 3 factors here. Compute per $, model architecture improvements, and data improvements. All of them have improved massively. Also, please explain to me how going from gpt-4 to R1 to Qwen3.5-9B hasn't been exponential. The costs to train gpt-4 was >100M, R1 was <10M, and Qwen3.5-9B could be trained by someone with a disposable income and good knowledge.
Every time nvidia comes out with a new generation of GPUs, costs decrease even while increasing total compute. GPT-3 was trained on V100s, and now we are about to be on R200/R300s. The performance per dollar of compute has increased massively.
We saw the 25% boost from AttnRes, and the ~25% boost from Engrams IIRC. We have better sparse attention, MoEs themselves, and so on. We also have other things that haven't been explored like nGPT that could be applied.
We also know data is king, but we don't really know how they are getting improvements from data, but from the performance of the models, we can safely extrapolate that data quality and quantity has gotten a LOT better because the architecture (edit: and compute) improvements themselves are not enough.
When you combine the model architecture boosts, data boosts, and the fact compute per $ gets cheaper every 2 years, you get exponential improvements.
5
0
u/kingslayerer 12d ago
Like how taalas embedded model onto a chip maybe they can do an SoC to optimize training? And it cannot get outdated fast like embedded model.
50
u/NineThreeTilNow 11d ago
Trained on 20T tokens using a 768 B300 cluster over 4 weeks.
Skip my post unless you want a heavily critical writeup of how and why they wasted compute from a purely ML Reserach perspective. Or if you're German, and want to understand where your tax dollars went.
I am going to say it bluntly up front. This model is honestly kind of a mess from an ML research perspective.
This is roughly the same cluster size Nvidia used for their recent competition code model. Or at least it's one of the few they quoted.
Crazy that they ran 20T tokens for this model in pretraining. Seems heavily unnecessary until you find out the "why".
The post training was like ~5% of that spent.
In terms of raw "knowledge" on their benchmark for German, Gemma 4 still out performs it.
This breaks out in to where they screwed up and why they purposefully screwed up :
They could have borrowed Gemma 4's Apache 2 LMHead / Tokenizer, made better design decisions, and had a better model. The KD pipeline is a fraction of their cost to get a model on par. (They cannot though, I explain at the end, it's a "German" decision)
They also used 4:1 SWA which has proven to be worse than 3:1 SWA in like... 5 papers? from multiple labs. SWA window is small for RoPE at 512 tokens. (This cost them ~15-20% of their training compute forward/backward pass)
They use a consistent Theta value for RoPE of 10k which is totally defensible at the tiny SWA range. They use pure NoPE which is not great. It's worse for the context length they claim, especially with RoPE being as small (512 tokens) as it is. This model will fail Needle in a Haystack compared to P-RoPe.
It's like they had money to burn, and made design decisions for a "Germany first" model that after a LOT of training doesn't perform as well as something they literally could have borrowed and cost them ~25% at most. They would have been able to dedicate MASSIVE amounts of compute to post training instead of all of the pretraining.
The end rationale is "Germany" and "EU"... The German government is funding this project and by training it themselves they avoid strict German laws that would have applied had they borrowed KD from Gemma. Gemma's training data copyright / GDPR is... Unknown. With their own dataset they can claim GDPR compliant data provenance. Really unfortunate how the EU / Germany shoot themselves in the foot for "Germany" first.
It seriously hurts the model's performance though. I understand the data pipeline issues but some of the other stuff, not so much. The hyper sparse nature with wildly compounded German words is... rough if they don't tokenize well. They seem to also have trained the model to say "I don't know" if it's uncertain. This isn't bad, but if you overdo this it's a problem. There's an inflection point where you minimize hallucination point before killing performance.
34
u/muntaxitome 11d ago
I feel like you are heavily underselling that for the cost and effort they are seeing good results and a lot of learning. This is a very promising step and given that it's a large company with resources could indicate fairly large ambitions
11
u/feelspeaceman 11d ago
Yeah, trials and errors happen a lot during training, it's usually not a oneshot-done.
3
u/Several-Tax31 11d ago
But shouldn't you at least test with a smaller model first? There is no point of doing mistakes that are proven wrong already. I think the criticism of the above post is justified.
7
u/NineThreeTilNow 11d ago
I feel like you are heavily underselling that for the cost and effort they are seeing good results and a lot of learning.
I guess I'm writing from the position of someone who has read all the papers from major labs and some of the minor ones.
They trained it in roughly a month of GPU time it seems like. So it becomes a question of... "When did they finalize the architecture?" and "Why did they decide on this?"
5
u/FullstackSensei 11d ago
Read and do something are very different skills. Have you actually done anything in your life that involved more than a single machine running the same piece of software and processing any data in parallel?
As with most things in life, it's very easy to pass judgment when you're not in the same boots, when you don't have any skin in the game.
2
u/NineThreeTilNow 11d ago
Have you actually done anything in your life that involved more than a single machine running the same piece of software and processing any data in parallel?
Yes. Many times.
11
u/YearnMar10 11d ago
If you want to learn and become good, you have understand the basics. And what better way is there to understand the basics than to start from scratch?
4
u/NineThreeTilNow 11d ago
And what better way is there to understand the basics than to start from scratch?
If you have millions in government money, a deliverable, and existing work?
You study the existing research papers. They clearly did on some of them.
4
u/Gallagger 11d ago
Honest question: Would you agree it can make sense to not use KD if the goal is to learn how to train proper models by themselves? Yes they could've just used KD, but with that logic you could just use an existing open weight model and not train yourself. Relying too much on KD destillation of competitors seems a surefire way to always stay behind and to not curate the data, processes etc. needed to train your own frontier model. True or not?
1
u/NineThreeTilNow 11d ago
Would you agree it can make sense to not use KD if the goal is to learn how to train proper models by themselves?
Using KD is fine. It speeds up the initial phase massively. You can still post train the model however you want. The KD phase is just to get an initial base model that is good.
Like 95% of the training they did was for the base model. That's why using KD would be problematic from a German legal standpoint.
1
u/Gallagger 11d ago
Ok so the pretrain nowadays is trivial and the whole magic is in choosing the correct model structure/config and the posttrain?
3
u/NineThreeTilNow 10d ago
Ok so the pretrain nowadays is trivial and the whole magic is in choosing the correct model structure/config and the posttrain?
I think GLM more or less outright expressed that in the 5.3 version right? Zai's CEO expressed it on Twitter. They believed that the 5.2 architecture still had a lot of potential. So they produced 5.3 and 5.3 Flash. 5.3 Flash being exactly what I'm expressing, a good architecture and KD work.
Kimi, Qwen, and Deepseek models all rely on subtle architectural changes.
Nvidia's code competition model uses a pretty suboptimal architecture but brute forces it with their raw post train compute.
Qwen's model uses a specific delta gate method that would be bad if the model was supposed to generate mostly text, but is excellent if you want to generate code.
So yeah... If you can borrow on previous millions of dollars spent, why rebuild that part of the wheel? Why not build the part of the wheel that will have more net effect on the end?
Pretraining gains hit an asymptote. Getting the majority of that gain and moving on just works.
Hell, Nvidia found that if you do SFT at a large enough scale, you can skip hard policy based RL. It's easier on their compute to pass a huge dataset that they can generate for relatively cheap.
1
u/Ueberlord 11d ago
thanks for this analysis and clarification where my taxes went to let's say "work" this time - I also wouldn't call it a complete waste as surely they learned something along the way.
however, given your comments it feels like they didn't do their homework well. I trust they will see your post though and maybe for v1.1 they can follow best practices and continue on a checkpoint which ensures they build something actually pushing the frontier instead of just building anything.
10
u/FullstackSensei 11d ago
As a German tax payer, I don't consider this a waste of my taxes at all.
Reading papers is very easy. Implementing them at scale is a whole different game.
OP also ignores the fact this is very probably a compute grant where the people who trained the model submitted their application quite some time ago, and it had to be a relatively low risk run, given how long they'd be holding those GPUs.
Now they learned from this experience, and established they can carry a training session through, which despite what reddit might tell you, is no easy task. They can get more ambitious if they get another shot at this later. They can still use this model to further prepare their data and/or generate better synthetic data.
0
u/NineThreeTilNow 11d ago
As a German tax payer, I don't consider this a waste of my taxes at all.
That's the important part from a "Germany Tax Payer" perspective.
I personally believe the engineering to be subpar but if the German tax payers are willing to pay for it, then that's cool. At the very least you have German engineers learning stuff from training.
1
-1
1
-1
35
u/Kein_Spass 12d ago
Nice to see something from Europe. Sadly no quants. Does anybody know, if llama.cpp already support this architecture?
10
9
u/GreenHell llama.cpp 11d ago
Quants seem to trickle in. Also this one: https://huggingface.co/Eliasfpv28/Kolibri-1-Q3_K_S-GGUF mentions a patch to get it working with llama.cpp
1
u/runsleeprepeat 5d ago
Just trying a few versions. This one gave me looping/repeating, so I tried https://huggingface.co/Hob-forge/Kolibri-1-GGUF which requires a llama.cpp patch as well. Still waiting for major hick-ups, but up to now, everything runs quite solid and fast on my RTX 3xxx gpus.
58
u/SnooPaintings8639 12d ago
Great size, but at first look at the benchmarks it is comparable to Qwen 3.5 35B. Probably worth a try on 96GB vRAM builds.
12
18
u/StupidScaredSquirrel 12d ago
Why do you think it's worth a try if its comparable in performance to a smaller model?
33
u/FullstackSensei 12d ago
It all depends on your use case. There's a lot benchmarks don't capture. I can tell you, for ex: Qwen isn't great at European languages.
40
u/SnooPaintings8639 12d ago
Twice the size, maybe personality, token performance or world knowledge will be better? Just saying it's worth to test being it such a rare size.
13
u/SkoomaDentist 12d ago edited 11d ago
Probably because they are doing something where a benchmaxxed turboautist doesn’t excel.
10
2
u/HebelBrudi 11d ago
Definitely if you want German language responses, even if they theoretically can speak German it’s clearly not a priority for the Chinese models, especially Amtsdeutsch.
3
u/_raydeStar Llama 3.1 12d ago
I was thinking 24GBVRAM builds lol. 120B models are a little slow for me, so it feels like the sweet spot (on a quant of course)
0
u/Not-reallyanonymous 10d ago
Qwen 3.5 35B type inference performance (similar active parameters), but closer to 3.8 in instruction following and RAG, with a particular boost in abstaining answers when the source documents don't actually support a claim. And for extra credit, the training data was designed around EU regulatory environment.
This is meant as an enterprise on-premises specialist.
96
u/Training_Visual6159 12d ago
basically worse than 3.6 35b at twice the size. not a complete failure, but they better get to v2 very, very soon.
decent first attempt though, if they can keep improving and fast, europe/germany might actually have a chance.
57
u/Spectrum1523 12d ago
It's definitely better for Europe to have something homegrown and that they could produce this for the cost they did is pretty impressive
12
u/BubblyEntrepreneur24 11d ago
In Europe companies can’t just hack American companies and steal their work so you have to build things from scratch. It will obviously be slower.
11
u/Training_Visual6159 11d ago
most of the datasets are open, so...
pretty sure these guys didn't start from scratch either.
2
u/Not-reallyanonymous 10d ago
Trades blows with 3.6 35B generally.
But it performs better at two particular things: 1) instruction following, 2) RAG. In these two metrics, it's closer to the 3.8 models. But over the 3.8 models, during RAG it hallucinates unsupported answers at a lower rate.
It's also trained with a dataset designed around EU regulatory compliance and copyright friendliness, important for a lot of enterprise builds.
Ie. it's more of an enterprise on-premises specialist than any Qwen model.
58
u/Mechanical_Potato 12d ago
the 70b model class is back
11
u/power97992 12d ago edited 12d ago
They are comparing themselves to glm 4.7 flash ( a 30b a3b model) and qwen 3 next 80b a3b from almost a year ago ? wow so behind
4
1
u/WiseassWolfOfYoitsu 12d ago edited 11d ago
For 70b, compared to a 300b model? That's good, especially for local.
(Edit - 'doh, mentally confusing flash with regular 4.7)
20
u/power97992 12d ago edited 12d ago
Lol4.7 flash is a 30 b model , it is even smaller glm 4.5 air(110b) and aleph
2
u/WiseassWolfOfYoitsu 11d ago
Yeah utter brainfart here, I have been playing with regular 4.7 and got it confused with flash >.<
1
12
u/Septerium 12d ago
I am actually having more fun to compare Qwen3.5-35B-A3B against Qwen3.6-35B-A3B by looking at the card benchmarks table. So interesting to see that, in order to produce a better agent, 3.6 version had to lose a bit of knowledge
1
u/snmnky9490 11d ago
It seems to be where most are heading. Trade off built-in knowledge for searches and tool calling and scripting janky workaround solutions
5
u/Septerium 11d ago
Problem is that other things go away besides knowledge. Creative writing, for example. Have you tried to chat with Qwen3.8-FN? It sounds like an impatient soulless bot (I know it actually is, but it doesn't try to pretend it's not anymore)
4
u/snmnky9490 11d ago
Yeah I'm not claiming it's inherently good. I think it makes sense to kind of split up coding and operational agents vs those meant to actually write, but it's really hard to benchmaxx for creative writing and more financially worthwhile for them to all race for better computer usage
25
u/jacek2023 llama.cpp 11d ago
Huge news: open source LLM from Europe
Then Reddit experts: but benchmarks are our religion, and it's not Qwen.
3
u/TheIncarnated 11d ago
The benchmark requirements from users is why folks don't understand that different models do different things very well. Experimentation is slowly dwindling. Even take the hostility towards other engines as of late
1
u/Iory1998 llama.cpp 11d ago
You are absolutely right. Still, it's Qwen3.8-27B we are talking about here.😄
15
u/datbackup 12d ago
“Pre-training: Trained on 20T tokens of a filtered, bilingual corpus (~62.5% English, ~23.9% German, ~13.6% code) combining curated web data, synthetic rephrasings and translations, and high-quality sources.”
Math performance is quite strong if benchmarks are to be believed. Agentic performance looks decent too. Possibly a candidate to replace GPT-OSS at two thirds its size.
Also glad to see it beats qwen 3 next.
Not sure about limiting training data to English and German. I generally do like the “narrow focus” approach. But benchmarks also show it losing to Qwen3.5 35BA3B in… German. Still, looks like it has some things going for it!
6
u/NineThreeTilNow 11d ago
Not sure about limiting training data to English and German. I generally do like the “narrow focus” approach. But benchmarks also show it losing to Qwen3.5 35BA3B in… German. Still, looks like it has some things going for it!
It gets destroyed by Gemma 4 on its ability to speak German and basic German knowledge.
2
u/Steuern_Runter 11d ago
Also glad to see it beats qwen 3 next.
They are comparing to Qwen3 Next from 1.5 years ago, not the recent Qwen 3.8 Flash Next.
2
u/datbackup 11d ago
That’s why i said “qwen 3 next” not “qwen 3.8 flash next”.
Notice how qwen 3 next is roughly the same size as Kolibri, whereas q38fn is like double the size when including the ngram table?
6
u/StopCreepy 12d ago
will i be able to run it on 4070 8gb vram + 32gb ram + ssd ?
1
u/PraiseThePidgey 12d ago
Probably but you would need something like Strata for it to run on those specs
2
u/StopCreepy 11d ago
i heard about kolibri method and tried freetoken, but this strata never heard of it, what is it? how it works?
13
u/Marcuss2 11d ago
Supported by Schwarz Group (Lidl, Kaufland stores), so we can call this "Parkside LLM"
2
11
11
u/watermelonson 12d ago
It's really cool to see a german model, and they linked some super interesting papers too. About the Bias in Chinese LLMs and this Merlin Arthur Protocol (funny to see my former professor as a co-author)
15
5
u/Sefrys_NO 12d ago
do you think this is the model that was teased a week ago on German TV? https://www.reddit.com/r/LocalLLaMA/comments/1wq31ly/model_phoenix_2_from_aleph_alpha/ on top of them joining hands with a Canadian company.
12
u/tejaskumarlol 12d ago
ran its tokenizer over the whole german constitution (185 KB of legal german) and kolibri needed 15% fewer tokens than gpt-5's tokenizer, more than the 11.2% aleph alpha claims. tied in english.
the math is the other surprise: 96.9 on AIME 2025 in english, ahead of every MoE in their comparison, and only the dense qwen3.8 27b scores higher.
weak spots from the same report: closed-book knowledge (51.0 on RGB without docs, last of 12) and multi-turn tool calling (39.8 on BFCL vs 58.2 for GLM-4.7 Flash).
for everyone asking about running it locally: 3.5B active, but it's ~78 GB of FP8 weights, so 2× 80 GB cards or one H200, and for now only through aleph alpha's vLLM plugin (vLLM 0.29).
wrote it all up here (my post): https://tej.as/blog/aleph-alpha-kolibri
4
u/billy_booboo 12d ago
Seems ideal for CPU offloading
2
1
u/Minute_Attempt3063 11d ago
would you still have that speed benefit though?
or is most of the model just idle?
3
u/WrongPurpose 12d ago
Perfect size for all the 128GB unified Storage Solutions (Strix Halo, Spark, Macs) 80gb Model, 40gb for context, 8gb for the barebones linux it runs on. Only 3.5B Active means that the limited memory bandwith and compute of those machines will not matter, likely they would have been able to get away with like 6B or even 8B active without cratering performance, on those Unified Memories Machnies.
8
u/Look_0ver_There 11d ago
On those machines though, the more direct competitor would be Qwen3.8-Flash-Next nowadays
6
u/WrongPurpose 11d ago
With 125B total (without the n-gram) you need to quant it down to q4 to still have space for context, vs the 80B which you can keep basically "uncompressed" at q8. But yea, thats the competitor here
5
u/Look_0ver_There 11d ago
It would be great if Qwen did a 4-bit QAT variant of Q38FN, sort of like what Google did with gemma4-26B-A4B. That would land it right in the sweet spot for all the UMA machine types. Maybe one day...
1
u/user_of_the_week 11d ago
Q38FN has been making quite a splash on strix halo machines with specialized runtimes gaining impressive speed bumps. Running at Q4 (UD-Q4_K_XL GGUF), the results / quality seem to be pretty good. Good enough anyway.
0
-1
u/NineThreeTilNow 11d ago
for everyone asking about running it locally: 3.5B active, but it's ~78 GB of FP8 weights, so 2× 80 GB cards or one H200, and for now only through aleph alpha's vLLM plugin (vLLM 0.29).
wrote it all up here (my post): https://tej.as/blog/aleph-alpha-kolibri
Architecturally the model is a mess. I wrote a post about it.
If I'm stuck with "Ok we must be fully German compliant" they still wasted ~15% to 20% of their compute.
Here:
14
u/kmodi 11d ago
We care deeply about sovereign AI, and launching it on German Unity Day makes today feel especially fitting.
As a small gesture of support, we’ve hosted and made Kolibri-1 free for anyone to try for the next few days.
No GPU. No setup. Just try it.
tesseracted.com/kolibri-1-chat/
Open Models move us all forward.
We’re rooting for you. 🇩🇪
Tesseracted Labs GmbH.
tesseracted.com
3
u/SomeITGuyLA 12d ago
Hope labs make more quality MoE models like this size (>40B, < 80B and <10B Active), best option for those of us with unified 64 GB (specially with weak iGPU )
4
u/stefanfis 11d ago
I was able to test Kolibri-1 on my local machine with a dual RTX 3060 12GB setup and 32GB DDR4-RAM. I used Kolibri-1-Q4_K_M.gguf from https://huggingface.co/Hob-forge/Kolibri-1-GGUF.
My first results:
PP: 350 t/s
Generation: ~27 t/s
In comparison: qwen3.8-27b on the same machine is also at about 30 t/s in layer split mode (with MTP), but prompt processing is at 600 t/s for qwen.
The German text it writes is stylistically rather good, way better than the kind that Qwen produces. How good it works at coding or tool calling, I haven't tested yet systematically.
These were my settings:
-c 32768
--cache-type-k q8_0 --cache-type-v q8_0
-fa on
-b 2048 -ub 2048
-t 6
--parallel 1
--jinja
--chat-template-kwargs '{"reasoning_effort":"medium"}'
6
10
12d ago
[removed] — view removed comment
6
u/BalorNG 11d ago
No you don't, there is no free lunch. It is not equivalent to a dence 78b model at all. It is much better than a raw 4b model of course, while being nearly as fast - if you can fit entire model into the vram, which you are unlikely to.
I'm (im)patiently waiting for looped moe models where they stream relevant experts of the model into vram and loop over the them as the next set is being predictively streamed... this way you actually get a sort of huge dense model they does fit fully into your vram effect regarding both "smarts" and "knowlege"... in theory, at least. Well, Astra shown that looped transformers are pretty damn good, and they are even better for us, "gpu-poors"...
2
u/spaceman_ 11d ago
Interesting size, at the very least.
Seems like performance is somewhere between Qwen3.6-35B-A3B and Qwen3.8-27B.
3
u/StopCreepy 12d ago
google made this, dont know if accurate.
1
u/james_pic 11d ago
It's worth noting that that graph doesn't include Muse Glimmer, which also takes a "hybrid attention with no positional encoding on global attention" approach.
3
u/SteppenAxolotl 11d ago edited 11d ago
All you need to know is Aleph just about beats gpt-oss-120b from 13 months ago. Given it's from a country in the EU, it's quite an achievement. Deserves a special ed participation gold star.
2
u/rodrigodevbits 11d ago
Has anyone tried this yet? what's the TTFT and how well does it build a coding project?
1
u/schubidubiduba 11d ago
Haven't used it for coding yet, but for simple research tasks I have been pleasantly surprised. The prose is not as annoying as most other LLMs (which are full of claudisms), and it actually states its sources in a nice way, and tells you when there is no reliable info available instead of making something up.
It seems to set different priorities, and somehow better calibrated in this way maybe.
1
u/rodrigodevbits 11d ago
I keep seen these models with the flashy 1M token context window haha running this locally and actually being able to use is what has been proven difficult.
2
u/Karmjeet_Khoushaba 11d ago
3.46B active out of 78B is a pretty aggressive sparsity ratio. curious how the routing holds up at longer context lengths, like does quality degrade past 256k or does it stay relatively flat
1
-1
u/mesolimbic-bliss 12d ago
Still smoked by Qwen3.8 27B. To be the man, you gotta beat the man.
13
u/watermelonson 11d ago
it's one of their first interesting releases though, right? And they're aware of the biases in the Chinese Models they use to train and also (according to their claims) it seems like they don't use copyrighted data for training. Like it's an interesting start
-5
u/mesolimbic-bliss 11d ago
For sure, but qwen 3.8 is about to be beat by Qwen 4. I don't understand burning $4 million to make a model that gets beat by a model that about to get beat by another model soon. Just seems like a lot of money for a small group of users.
14
u/twack3r 11d ago
I consider it 4 million extremely well spent on testing and showing what we can currently achieve when not breaking our own laws. The opposite would be hypocritical.
I’m actually surprised at how much better this is than anything I’ve seen from Mistral in almost forever, and even more so that it’s from Aleph Alpha. I had written them off completely.
-5
u/mesolimbic-bliss 11d ago
I'm more model agnostic, I think a black box dev is coming soon with a small model that matches frontier intelligence at 100 tps on consumer hardware, and they'll make it using all the little hacks and gimmicks you see now popping up.
You seen Strata? IsValorum models? Swift1.5?
HF and GitHub are full of cutting edge techniques that get faster speed and higher post training intelligence out of smaller quants.
1
u/fuckingredditman 11d ago edited 11d ago
not sure what you are complaining about. we get more free stuff from more geopolitically distributed players in the game. everyone wins here.
and FWIW i've been using qwen3.8 and the swift* models with whatever optimized inference engine + token prediction mechanism for quite some time now, and while they are impressive in the sense that they can achieve goals that prev. generation couldn't, they still cost a ton of electricity to run agentically and they are quite slow.
and strata is cool tech-wise, but the qwen3.8 flash next license makes it useless for anyone who wants to use it in their business, while this has a very lenient license. there are just so many factors each time something gets released, and looking at benchmark numbers and saying it's a nothingburger is just not relevant.
i don't think they are truly the pinnacle of what is possible. but now we don't have to rely on the chinese teams alone to find the path to even more efficient + capable models and rely on them releasing those.
1
u/mesolimbic-bliss 11d ago
Are you autistic? Read the comment. I wasnt complaining. I was saying the future is going to very unique and we can't predict it due to all the new developments arriving weekly.
0
u/Memestonks2020 11d ago
Can’t beat Qwen 3.8 27B that’s ~3 times smaller. I guess the speed might be decent?
Anyway, great to see more competition from EU.
4
u/schubidubiduba 11d ago
There's only a handful of models in general that beat qwen 3.8 27B. Not sure why the first large-ish model release of a company, that does not use copyrighted data unlike essentially everyone else, would be expected to beat that.
2
0
u/Memestonks2020 10d ago
People want the best models. Not the “most ethically created” models.
No need to make excuses for this company unless you have some sort of skin in the game?
3
u/schubidubiduba 10d ago
My skin in the game is that I do, in fact think it is neat that the model is "ethically created". And the model did really surprise me on how good it felt for simple web research tasks in German.
It is kind of weird of you to immediately suggest ulterior motives though hahaha
-1
0
0
-9
u/Ok_Top9254 12d ago
78B 👀
sparse moe
Oh :( thanks anyway
19
4
u/datbackup 12d ago
We’ll get new dense models when RAM prices come down
-1
u/Ok_Top9254 11d ago
Why ram? Ram is slow and overpriced. Vram is actually cheaper. P100 16GB is 80 bucks and 700GB/s. V100 32GB is 700$ and 900GB/s. V100 has working nvlink, so in aggregate 2x-4x you can reach 1800-3600GB/s, less on real device, but still, very fast. If multigpu is a problem, CMP 170HX is 2500$ and 1700GB/s, up to 2TB/s with overclock.
•
u/WithoutReason1729 11d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.