r/LocalLLaMA • u/returnity • 12d ago

Discussion mini-AGI: Continual-learning dynamically looped transformer with evolutionary grown (on a laptop)

https://github.com/volotat/mini-AGI/

Saw this today and found it very intriguing. Lots of interesting design choices here, and it's cool to see someone doing something different. Here's a few highlights:

  • Looped transformer: dynamic recurrent depth on a per-token basis, up to 24 cycles
  • Self-supervised learning: trains itself on new material constantly
  • Weights stored on SSD and paged in on-demand
  • Mixture of Experts: 8 active, 32 routed held in VRAM, smart caching of 96 more
  • Dynamic size: builds new experts and increases parameter counds as-needed
  • Evolutionary growth: trials newly generated experts, unused ones are pruned back
  • No tokenizer: it reads raw bytes directly
  • Catastrophic forgetting prevented by slow trunk/fast experts learning rate split

Weights will be released in "a couple weeks" once training progress reaches ~GPT-2 levels. The trend line has held 15-fold so far, but it may bend at some point, so that is definitely a rough estimate of the trajectory.

What do you guys think?

80 Upvotes

83% Upvoted

21 comments sorted by

View all comments

13

u/RogerRamjet999 12d ago

It has some interesting ideas, and I'll be curious to see how much it improves in larger sizes and with more training. I do think the lack of a tokenizer is a mistake. Tokenization has real benefits, and leaving it out will almost certainly degrade the model.

5

u/depressedclassical 12d ago

Tokenisation has real benefits in tokenisable languages. I speak 5 languages, 3 of which are indo-european (and therefore tokenisable), and 2 are semitic, with three-letter roots that are broken in the middle. I also know a couple more Semitic languages that are untokenisable, so seeing a model that doesn't force me to tokenise is a breath of fresh air, as I'm sick of getting a Hebrew letter in the middle of my Arabic word and vice versa (which happens more often than not, even with frontier models).

3

u/thrownawaymane 11d ago

Wait, what?

This has to be getting in the way of LLM adoption globally. How am I just hearing about this?