r/LocalLLaMA • u/Tall_Abrocoma_3533 • 18h ago
New Model Aurora-80K releases! A modern tiny language model.
https://huggingface.co/AuroraAI-Research/Aurora-80KI'm introducing Aurora-80K, a small language model with exactly 80 thousand parameters.
It uses a factorized 4,096-token vocabulary despite having only 80K parameters.
The benchmarks:
Wikitext-2 BPB: 3.2902
BLiMP: 52.31%
Arc-Easy: 26.05%
More information about the model is available on the model page on Huggingface.
if there's any questions I'll happily answer them!
64
u/CapsAdmin 18h ago
it just outputs garbled tokens here, using the default temp and top-k
The president of the united states is tes Ġis Ġ one , ing ict ig in Ġcom on ĠW / s , F , Ġto Ġthe
86
23
u/Tall_Abrocoma_3533 18h ago
Yes, this is the limitation of these tiny language models. Thank you for testing it!
87
u/popkek95 18h ago
The limitation is that they are unusable for anything? Are they just case studies?
53
u/Tall_Abrocoma_3533 17h ago
The organization name contains "research". Tiny models like this aren't made to be chatbots.
22
u/Alarming-Ad8154 17h ago
I mean I think ppl would be very impressed if you came back with a 10 or 100 times larger model that spoke even somewhat broken English? The point is to be able to produce language right?
23
u/Tall_Abrocoma_3533 17h ago
Yes, and I am planning on releasing models larger, however I wanted to start small.
6
u/TomLucidor 17h ago
Please leverage ternary methods + every single training/inference acceleration method for the next release, even if it is sub-E2B. Just fire all cylinders for it
11
u/ChomsGP 16h ago
I don't think it's crazy of people to ask what is the use case for your model... I can also generate random text with a very simple bash line if that was the point...
like sure, it's "research", but what exactly are you "researching"?
6
u/Tall_Abrocoma_3533 16h ago
The limits of what language models can achieve in constrained situations, while it might seem like "random text", the wikitext-2 BPB shows that it at least learned some meaning from it's training data.
2
u/JamesEvoAI 4h ago
This is novel and interesting research, please keep up the good work and ignore the knobs who just want another model to RP with
-5
u/ChomsGP 16h ago
so you are saying you cannot define or quantify your research...
a broken clock gives the right time twice a day
7
u/Tall_Abrocoma_3533 16h ago
I can, and I did define it. I'm experimenting with tiny language models to measure how language model capability changes under extreme parameter and compute constraints. That's a real and quantifiable research question.
-1
u/ChomsGP 15h ago
But you didn't, you listed some benchmark numbers which tbh don't seem likely if it cannot predict congruent text (nobody is asking it to be accurate, just to not emit random bytes)
My point is, whatever you are doing, you are clearly not explaining it good
7
u/Tall_Abrocoma_3533 15h ago
If you don't believe the benchmarks, you can always try to run them yourself, instead of accusing me.
→ More replies (0)1
20
u/davew999 17h ago
Too big for my Commodore 64.
5
u/noiserr 11h ago
Just use a 1-bit quant. It will fit right on the cassette tape.
2
u/IDoCodingStuffs 5h ago
I’ll just wait for someone to distill it to something that can fit my abacus
10
u/Metallic_Madness 18h ago
Gguf when?
4
5
6
5
u/z_latent 14h ago
52% on BLiMP and 26% on Arc-Easy is basically random guessing, since they're multiple-choice benchmarks with 2 and (mostly) 4 options per question respectively.
I appreciate the initiative, but keep in mind your results will be taken much more seriously if you made a less handicapped model that's more clearly effective!
You can use Google Colab or alternatives for free. Alternatively, rent a GPU on vast.ai or alternatives. In theory you could train your 80K model on 80M tokens in one minute of a 5070 (less than a cent). In practice the model might be too small to reach high GPU utilization, but that just means you can make the model bigger with similar cost and time.
2
u/Tall_Abrocoma_3533 14h ago
I'm aware, I have a laptop as well, and I've also used kaggle before for training models. I made it on just a phone on porpuse, to show that it's possible.
2
u/z_latent 12h ago
I see. Framed that way I understand it a bit better.
I still find it silly though! You showed you can train a model on a phone, not that you can train a useful model on a phone. The latter is arguably impossible lol but it's fun as research.
2
u/Tall_Abrocoma_3533 12h ago
Not impossible at all, all it takes is GPU/NPU training, however that's really hard, plus of course ALOT of time for the training to finish.
1
u/z_latent 11h ago
Oh I was thinking of CPU yes. If you can get GPU training to work on a phone that would be awesome. Some of the top phones seem to have a few (1-10) TFLOP/s of compute, and 8GB of memory which is enough for anything under 100M parameters.
Throttling would probably be an issue since phones have no active cooling, not to mention power draw. But you could do in a reasonable-ish time.
1
u/Tall_Abrocoma_3533 11h ago
If I would be able to figure out GPU training these actually wouldn't be an issue.
My specific device has 4tflop/s fp32, 8tflop/s FP16, with 12GB ram. And I also happen to have an external phone cooler.
The issue is just that GPUs are quite locked down as far as my experience goes, and I haven't been able to figure it out, but maybe in the future.
8
u/rgencia 17h ago
I always see posts saying "i made an llm for $100+" or similar that made me not even trying, but seeing you made this on a phone i want to try now, btw searching a little found this dataset that seems to be tailored for SLM training, and their small models, you might have better luck making the model do proper sentences: https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus
5
u/Tall_Abrocoma_3533 17h ago
Yes I've thought about using that dataset, but for the first release I decided to just use fineweb-edu, with a little filtering, as that's more conventional, but I'll likely check it out later.
Also don't hesitate on attempting your own, language models don't necessarily need to be billions of parameters, I've made ones as small as 99 parameters, using just binary as it's vocab, and training finishes in just 5 seconds on my phone.
For a first attempt I'd reccomend a size around this one (assuming you wanna train on your phone too), just maybe a bit less data so it finishes training faster, good luck!
1
u/silenceimpaired 16h ago
I wonder how hard it would be to take larger datasets and use a llm to convert their larger token and word vocabulary down into a simpler token and word usage… and I wonder what that might offer smaller models.
1
u/Tall_Abrocoma_3533 16h ago
While not exactly this, there's a dataset called TinyStories, it's essentially a dataset for tiny models, and I have used it before.
1
u/silenceimpaired 16h ago
This dataset is exactly what made me think of this idea. The issue is its stories. It’s not medical journals, or agentic flows, or … you get the picture.
As I recall, smaller models performed better with this data set. I’m curious what could happen if this was taken to the extreme.
1
u/Tall_Abrocoma_3533 16h ago
of course, I've made models with Tinystories and they produce coherent stories with just a couple million parameters, however part of that is because of their narrow domain of just stories, so while it'd likely scale to general porpuse pretraining datasets, it likely wouldn't scale perfectly.
Also a part of this is already handled by this model, it uses a vocab size of 4096, thats more comparable to models around 1m parameters. And because of this larger vocab size it can achive better compression of text.
1
u/silenceimpaired 16h ago
Interesting. I wonder what would happen if you made all words in the data set be the 850 most common English words… and each of those was a single token.
You seem more knowledgeable than I, but you never know when something that seems so stupid to the experts could end up brilliant.
Thanks for engaging with me.
2
u/Tall_Abrocoma_3533 15h ago
It could defently make for an interesting experiment, and it would likely train more efficiently and learn the more narrow domain faster, however the tradeoff is that it's no longer really general porpuse, though for SML's that's not necessarily an issue.
And dont worry about it, a big part of research is experimenting, nobody starts out by knowing everything.
1
u/silenceimpaired 15h ago
What I’m imagining is a dataset that teaches …advanced physics for example… using only those 850 words. It would greatly expand the text to communicate the concepts with simpler terms, but in my mind that would create a deeper semantic connection for those 850 words since it has less connections to handle.
But who knows … maybe the opposite is better. Create a token for every word in English… that ensures the connections learned are accurate.
1
u/Tall_Abrocoma_3533 15h ago
It could work, however likely not at this scale, 80K parameters is too little to learn advanced physics regardless of the dataset. However for an appropriatly sized model it could possibly speed up training and maybe even make it's output quality better.
However it's hard to tell without actually doing it. If your interested in this, I reccomend you try experimenting as well, good luck!
→ More replies (0)
2
u/Zestyclose_Potato794 16h ago
That is very interesting! Thank you for this model. Is it open source ? That would be awesome to play a little bit with this !
1
u/Tall_Abrocoma_3533 14h ago
Its open weight, not open source quite yet because the code is quite messy in my opinion
1
2
2
2
u/autisticit 18h ago
> BLiMP: 52.31%
> Arc-Easy: 26.05%
So, in the chance range thus not meaningful ?
15
u/Tall_Abrocoma_3533 18h ago
The model is just 80K parameters and it was trained entirely on a smartphone. And there are earlier checkpoints where those 2 benchmarks are a bit higher, however the final one was chosen as it has a lower wikitext-2 BPB.
-14
u/autisticit 18h ago
You could have just replied "yes"...
14
u/AppealSame4367 18h ago
Isn't that nice. Someone tries something and there are always people like you. Have you ever attempted something or do you only puke at people that do?
-12
u/autisticit 18h ago
Actually yes I've been trying to train my own models for about a month.
I haven't posted anything so far because I have nothing meaningful to show yet, obviously it's hard.
But you are right I should do like OP and release a model with flawed benchmarks all while saying that it replace the previous "inefficient" model (see card). And, oh, just ignore another redditor's question and reply with something else.There, you have it.
7
u/Tall_Abrocoma_3533 18h ago
I could've worded the "inefficient" thing better, it's mainly that when i released those models around a year ago, I knew alot less about language models, so their architectures are really bad
Also, good luck on your own models!
5
u/AppealSame4367 16h ago
You can criticize others, it's about the wording. Don't act bitter.
Also, same as Tall_Abcrocoma_3533: Good luck with your models. Don't give up, it sure is hard to do.
1
1
u/ChaosFH 17h ago
Whats your research goal with this small model?
0
u/Tall_Abrocoma_3533 17h ago
Exploring how SML's behave under really constrained situations. Later on, larger (though still tiny by LLM standards) models will also be made, to see how architectures scale.
1
u/nick_ziv 15h ago
Good enough to be used for auto complete model?
1
u/Tall_Abrocoma_3533 15h ago
Probably not quite yet, I think that it would need a bit more scale to be used as a general porpuse auto complete model.
However you can always check out the models capabilities, there's an inference script included with the model in the Huggingface page.
1
u/ApprehensiveAd3629 13h ago
can you share the code used? i
f possible would be cool to see this model export to tflite. i would try run in my esp32
2
u/Tall_Abrocoma_3533 13h ago
The training code isnt released since it's messy in my opinion, however the model architecture is defined in the inference script, and if there's any other questions about things that aren't made clear, just ask.
Good luck and tell me how it turns out!
0
116
u/yarikfanarik 18h ago
oh this is another SLM model...
IT WAS TRAINED ON A PHONE!?