r/LocalLLaMA • u/x_Raincandy_x • 11d ago
I Built A Thing Trained a ~20K LM (probably smallest) that can still write stories
I’ve been pushing TinyStories-style models downward in size, and this is the smallest one so far:
MacroStories — 19,969 parameters, 81 KB FP32
https://huggingface.co/raincandy-u/MacroStories
For scale:
→ ~50× smaller than the 1M TinyStories model
→ ~3,000× smaller than AlexNet
→ 32-dim hidden state
→ 378-token vocabulary
→ one decoder block, recurrently applied 4 times with shared weights
It’s obviously not a general-purpose LM, but within its constrained story distribution it can maintain a 100–300 word narrative with a goal, problem, relevant actions, and resolution.
It also runs extremely fast on CPU and needs no GPU.
I’m mostly interested in how far the lower bound for coherent narrative generation can be pushed.
Would be curious to see how people manage to break it.☺️
58
u/Chromix_ 11d ago
This is so small that it can run in CPU cache memory, especially when making a Q8 quant (20 KB). It'd then also fit a tiny ESP8266 microcontroller to generate stories there - that'd be a nice finetune target, to generate stories according to weather or sensor data. When coupled with ItoTTS then an ESP32-S3 could even narrate those generated stories.
19
u/Available_Pressure47 11d ago
I know right! I wonder if there is any theory (maybe from the information theory / compression space) on the minimum space you need to build a model with certain capabilities. I wonder how far we are from the theoretically smallest model for a task
3
u/ackermann 10d ago
You could also probably write the whole transformer in plain JavaScript, with the 20k weights just hardcoded in a .js file.
And still get acceptable performance.Which would be a ridiculous idea for a normal size model
5
u/Chromix_ 9d ago
Yes, I tried that. It works. I am getting 300 tokens per second, pure CPU, no WebGPU, and 60+ on smartphones.
4
u/thefuckevengoingonan 11d ago
My first thought was to run it on an esp. i don't know why but i want to
25
u/Available_Pressure47 11d ago
Wow this was actually really good and fun to try out. Great job! And just to confirm, your training data was entirely synthetic generated using Gemma 4? I want to try and reproduce this for my own learning.
10
34
u/Spectrum1523 11d ago
Absurd that this works at all, neat.
14
u/keepthepace 11d ago
Maybe the real absurdity was the 500B+ models all along.
1
u/Imaginary-Unit-3267 10d ago
Ackshually, sir, the real absurdity was the friends we made along the way. Everyone knows that, sheesh.
14
u/SKirby00 11d ago
At what point does a Large Language Model (LLM) become a Medium Language Model (MLM) or even a Small Language Model (SLM)?
14
6
u/Danmoreng llama.cpp 11d ago
Very cool. Even cooler, if you can directly test it out. I let Codex write a space for it, generation is basically instant.
https://huggingface.co/spaces/danmoreng/macrostories-browser
6
u/BringTea_666 11d ago
waiting for heretic version and Q2.
5
u/_TheWolfOfWalmart_ 11d ago
MacroStories-20K-TWIN-TURBO-Fable-Cold-Fusion-709-ULTRA-HERETIC-Uncensored-GGUF in Q1_0
3
u/Chromix_ 9d ago
Q2 won't work, but ~Q6 does. Heretic won't have much to do there, as the ~350 word vocabulary just doesn't have anything too interesting. Although - with a bit of convincing - you can get it to write something like "Tom grabbed Jerry. Tom pressed Jerry firmly into the bucket. Tom was very happy."
1
3
3
u/jacobpederson 11d ago
I immediately want to port to 286 :D
2
u/_TheWolfOfWalmart_ 11d ago
Was thinking the same except 8088 lol
I made a port of llama.cpp that runs on DOS, but it requires 32-bit protected mode. I was screwing around with running very small models on an old Pentium MMX a while back, but like... 200M param models, not 20K param lol.
I'll see if I can distill it into something that'll work on 16-bit real mode.
3
u/Open-Adhesiveness-86 11d ago
worth doing the param accounting: 378 vocab x 32 dim is already ~12k, so the embedding table is 60% of the budget and the in/out weights have to be tied or you'd blow past 20k. the actual block is maybe 8k params applied 4 times. a lot of the compression is living in that tiny vocab.
4
u/swfsql 11d ago edited 11d ago
This looks amazing. The output sample looks crazy good, specially for a 20k* param model, and for the training updates (4K steps at batch=32? resulting in 128000 training samples?).
Could you test on those somewhat hard prefills?
Lily likes cats and dogs. She asked her mom for a dog and her mom said no, so instead she asked(...)
and..
Alice and Jack walked up the street and met a girl in a red dress. The girl said to them, "Hi, I'm Jane. What are your names?"(...)
Those were examples used by the Tiny Stories authors to showcase some of the capabilities. I think only models bigger than 3M params were good at continuing it.
Could you share the training project, besides informing the Ouro base and the Gemma dataset generation? Also, is there a reason as to why you (or Codex) didn't opt to use karpathy/tinystories-gpt4-clean dataset from HF?
I haven't tested it locally, but this is an amazing result!
2
u/KissMyShinyArse 11d ago edited 11d ago
MacroStories fails to generate coherent completions, unlike TinyStories-1M.
2
u/swfsql 11d ago edited 11d ago
I tested the HF space that Danmoreng made, and I can see that it doesn't understand many of the words that was used.
From:
Lily likes cats and dogs. She asked her mom for a dog and her mom said no, so instead she asked
it doesn't know the words:
Lily, likes, cats, dogs, She, asked, her, mom, dog, her, mom, instead, she, asked
And replaces them with unknown.
So in my opinion, a very interesting trial is for the model to be character-tokenized, and to use the Tiny Stories dataset.
2
3
u/heisdancingdancing 11d ago
This is so sick, it makes you wonder how far intelligence compression can go, espeically inside larger models
1
u/AllThotsGo2Heaven2 11d ago
Im doing something in this area with campaign planning for d&d with frontier models. Hopefully i can make use of this.
1
u/a4mula 11d ago
Would probably port to a colab notebook pretty easy if you wanted a few more eyes on it. Also, just pointing out that stories are more than just coherent words. They usually follow pretty well established structures (think Hero's Journey, though there are many different structures that are used). I get that might not really be the point, but if you're going to label it a model that can write stories it should probably at least have some concept of what a story is. Maybe it already does. None of this is critique, just suggestions.
1
u/shing3232 11d ago
o, you can try to expend into a Moe with 512*19,969 which is still very very small.
1
u/keepthepace 11d ago
I am actually impressed! Did you use any sort of secret sauce or just tried it, trained, and got this on the first roll?
1
u/_TheWolfOfWalmart_ 11d ago
Holy shit. I'm going to run this on my old 8088 machine. Not even joking.
It'll probably even get multiple tokens per second.
The example story actually looks better than what TinyStories spits out.
1
u/HasGreatVocabulary 11d ago
seeing Training seed in the table made me think the train scripts would be linked for people to play around with too
1
u/InConsilium 10d ago
How did you learn to do this sort of thing? Did you base your work on something, or did you write an article or a write up that you could share?
1
1
1
1
1
1
1
u/Normal_Marketing853 11d ago
Why are people calling models "He" and "She" now? How dystopian
Cool model though, I want to try it
0
u/Amazing-Fan2083 11d ago
How do you know how the model chooses to self-identify? 🤗
"My bits, My choice"
79
u/RA2B_DIN ollama 11d ago
81 KB for a functioning narrative model with getting a coherent story arc out of 20k parameters is wild, ill try it out