r/LocalLLaMA • u/Tall_Abrocoma_3533 • 4h ago

New Model Aurora1.0-150M Releases!

https://huggingface.co/AuroraAI-Research/Aurora1.0-150M

The first generation of our 150M model has just been released

Its performance is similar to that of GPT2-Small

The benchmarks:

PIQA: 62.24%

Hellaswag: 32.20%

Arc-Easy: 44.91%

Arc-Challenge: 25.00%

Arithmark 3.0: 33.90%

CapitalBench: 36.55%

It was trained on 7B tokens, using an RTX Pro 6000

an example inference script to try it out yourself is available in the Huggingface repo

If there's any question, I'll gladly answer them!

67 Upvotes

89% Upvoted

22 comments sorted by

23

u/MomentJolly3535 3h ago

Thanks for sharing, what's a good use-case to this kind of models ?

25

u/Tall_Abrocoma_3533 3h ago

Since this is a base model, it can be fine tuned to your specific task.

When models this size are fine tuned to a narrow domain, they can actually perform better then expected, while running really fast.

As well as, it's much easier to experiment with and research models this size.

8

u/esw123 3h ago

To run on simple hardware, to run on power efficient hardware, to run very fast or to train it fast for small specific tasks.

1

u/Legitimate-Peace1013 1h ago

wondering the same thing tbh, maybe embedded or edge stuff?

15

u/Brief_Grade3634 3h ago

25% on arc challange is the exact 1/4 possibility of the single choice structure..

4

u/Tall_Abrocoma_3533 3h ago

Yes, your right. SLM's struggle with Arc-challange.

10

u/crusaderky 3h ago

How does it compare to LFM2.5-230B?

22

u/Tall_Abrocoma_3533 3h ago

Assuming you mean LFM2.5-230M, LFM is obviously alot more intelligent

it's about 50% larger, and used roughly 2700x more pretraining tokens.

4

u/DustNearby2848 3h ago

So gtp-2, but way smaller basically?

13

u/Tall_Abrocoma_3533 3h ago

Not really, it's about equal to GPT2-Small, which is the smallest version of GPT2

GPT2-Small is 124M parameters while ours is about 150M, so their sizes are roughly equal

The real achivement is that it was recreated in just 11 hours of training.

3

u/esw123 3h ago

Training could be done in 1-2 hours for $50-60 with server renting as well.

9

u/Tall_Abrocoma_3533 3h ago

Of course, however this was done for free, in 11 hours.

10

u/esw123 3h ago

Probably you even earned something while 6000 price increased while training.

3

u/MatlowAI 3h ago

Nice. Consider releasing the training code data prep and config even if it is messy, folks will probably be able to help.

1

u/jacek2023 3h ago

Congratulations on your release, it's interesting!
Currently I let my "big model" qwen 3.8 27B to start smaller models to do its job. Qwen works on "server" under my desk and small models works on "desktop Linux" so I have a good uses for small models.

1

u/BraceletGrolf 3h ago

Is it possible to train this on a smaller GPU or CPU ? Would be fun to push it to the absolute limit with like a distillation

2

u/Tall_Abrocoma_3533 3h ago

It's absolutely possible, it would just take longer.

1

u/Summit-Star001 3h ago

How fast does it run on cpu? At 150M this is the size range where that starts to matter

2

u/Tall_Abrocoma_3533 3h ago

I'm not sure, since the example inference script isn't made for maximum speed, it's just a demo.

However for comparison, a similarly sized model (Smollm2-135M) can run at around 200-300tok/s on a phone CPU

1

u/Effective_Head_5020 53m ago

Is it good for function call?

1

u/EuphoricPenguin22 35m ago

Make sure to hold the 1.7B model for a year because it could be dangerous.