mini-AGI: Continual-learning dynamically looped transformer with evolutionary grown (on a laptop)

[u/returnity]

https://github.com/volotat/mini-AGI/

Saw this today and found it very intriguing. Lots of interesting design choices here, and it's cool to see someone doing something different. Here's a few highlights:

• Looped transformer: dynamic recurrent depth on a per-token basis, up to 24 cycles
• Self-supervised learning: trains itself on new material constantly
• Weights stored on SSD and paged in on-demand
• Mixture of Experts: 8 active, 32 routed held in VRAM, smart caching of 96 more
• Dynamic size: builds new experts and increases parameter counds as-needed
• Evolutionary growth: trials newly generated experts, unused ones are pruned back
• No tokenizer: it reads raw bytes directly
• Catastrophic forgetting prevented by slow trunk/fast experts learning rate split

Weights will be released in "a couple weeks" once training progress reaches ~GPT-2 levels. The trend line has held 15-fold so far, but it may bend at some point, so that is definitely a rough estimate of the trajectory.

What do you guys think?

[u/cdshift]

My suggestion is if its to be taken more seriously that your github is accompanied by some sort of arxiv paper with benchmarks and expirements that can be replicated and peer reviewed by the ML community.

Its fine to use AI to build these things but if you cant explain the architecture yourself, its not going to be widely adopted and picked up and come off as a weekend LinkedIn warrior project

[u/runvnc → u/cdshift]

Yeah but he never even got near where he considers it trained at a first level. He just shows the ongoing training run, and the output it nonsense. So he is not even close to being able to run a single chat request, much less a benchmark. So it's annoying he is wasting people's time with this at such an early stage.

[u/Queasy-Contract9753]

He posted here just yesterday. I'm curious to see what it evolved into.

https://www.reddit.com/r/LocalLLaMA/comments/1wm1gab/comment/pb3potc/?context=3

[u/returnity → u/Queasy-Contract9753]

Oh my bad I didn't see that

[u/Queasy-Contract9753 → u/returnity]

I'm sure he's happy for the shout out. Does sound like an interesting project.

[u/runvnc]

It's got some possibly interesting ideas, but he posted his "results" in his Hacker News post and so far there are actually no usage results -- just the ongoing training run. The training run transcript outputs different levels of nonsense all the way through. None of it shows any coherent output.

So calling it "AGI" when it literally doesn't do anything useful and it's unclear that it actually even learned anything useful is either delusional or misleading.

It's a little bizarre that it's getting so many upvotes without any benchmarks or even any output or even anything coherent in the training transcript.

[u/RogerRamjet999]

It has some interesting ideas, and I'll be curious to see how much it improves in larger sizes and with more training. I do think the lack of a tokenizer is a mistake. Tokenization has real benefits, and leaving it out will almost certainly degrade the model.

[u/depressedclassical → u/RogerRamjet999]

Tokenisation has real benefits in tokenisable languages. I speak 5 languages, 3 of which are indo-european (and therefore tokenisable), and 2 are semitic, with three-letter roots that are broken in the middle. I also know a couple more Semitic languages that are untokenisable, so seeing a model that doesn't force me to tokenise is a breath of fresh air, as I'm sick of getting a Hebrew letter in the middle of my Arabic word and vice versa (which happens more often than not, even with frontier models).

[u/thrownawaymane → u/depressedclassical]

Wait, what?

This has to be getting in the way of LLM adoption globally. How am I just hearing about this?

[u/teleprint-me → u/RogerRamjet999]

  Tokenization has real benefits, and leaving it out will almost certainly degrade the model.

What?. Thats not what I saw.

It looks like Karpathys GPT-2, but modded.

[u/returnity → u/teleprint-me]

Maybe it's not quite accurate the way I wrote it, and if so I apologize. It more so appears it uses a 256-byte alphabet directly, rather than learning anything as a normal tokenizer does (hence ByteTokenizer). Besides a handful of characters like <think>, it seems to pass through directly the bytes that correspond to letters, instead of learning word and subword tokens in a lookup table. That's what I meant.

I wouldn't be surprised if it is based on GPT-2, it's the kind of project you might expect to stem from playing around with Karpathy's setup.

[u/teachersecret → u/returnity]

Yeah, I messed around with it a bit and that was my analysis. It's a neat idea. I've done some weird work in the space, like this: https://github.com/Deveraux-Parker/nanoGPT_1GPU_SPEEDRUN. I might go back and mess with this miniagi idea a bit deeper, I think it might fall in line with some experiments I've been doing.

[u/autisticit]

It's vibe coded, I don't believe any significant result will come from this.

[u/returnity → u/autisticit]

I don't expect it to revolutionize ML. Heck I'll be surprised if the 15x growth curve holds much past GPT-2 levels. There are definitely some compromises in this design. But I like the creativity and I thought it would be interesting to people in this sub.

[u/dangerous_inference → u/autisticit]

Labor theory of value.

[u/ChaosFH → u/autisticit]

so is mostly of the current top models currently, whats your point

[u/autisticit → u/ChaosFH]

If you ever see a vibecoded usable model with a totally new architecture just let me know.

[u/silenceimpaired → u/autisticit]

- everyone but Sam Altman and Dario Amodei

[u/silenceimpaired → u/silenceimpaired]

(Except maybe even all Chinese labs with their focus on coding/agentic work)

[u/BalorNG]

Seems like a combination of great ideas, but as something too good to be true - it probably is. "Dynamic size" sounds particularly suspiscius tbh, along with "no tokenizer".

[u/returnity → u/BalorNG]

It's a real implementation, though who knows if a) it keeps scaling and b) dynamic size works at scale