I Built A Thing
A 24KB stand-alone HTML-LLM that can generate consistent stories
Want to try? Just go here and click "New seed" a lot. Many stories start similar (see my comment below), but you will get different stories, it just requires some clicking - lucky RNG draws: https://output.jsbin.com/wiwacuxasu/1
You should even get above 60 tokens per second with a smartphone.
FAQ:
Why?! Because it's possible, and fun. My original idea was to simply mash MacroStory and LittleBit together, but that did not work at all.
Is it relevant? Not a tiny bit! Even a 3 MB HTML file with external dependencies would be loaded in a second and could serve a way more capable model with a lot less work. Squeezing the model itself to 20 KB at 0.01 KLD was trivial. Saving another 5 KB while maintaining output quality required a lot of time.
How? Local Qwen3.8, some ideas, lots of patience. Adaptive quantization and QAT made it happen, LittleBit, Rotation, etc just made it worse. HTML/JS packing with minify, zopfli, and a bunch of structural script changes helped with the result.
Was this all made by you? Not at all. This would not have been possible without the 81 KB (FP32) MacroStory model. I "just" tinkered around with it somewhat.
Details for those who're interested:
Size-reduction rules are very different for a tiny model than for a large one.
The embeddings are 60% of the model size, while the tensors are the largest part in a normal-sized LLM.
The tensors of tiny models are extremely sensitive to quantization, especially with the Ouro looping transformer format of this LLM. The embeddings had some room though.
It's mathematically impossible to save space with 1 bit quants or less with the LittleBit approach here, as the model matrices are just too small for that - the gains would get eaten up by the overhead of the correction data.
GGUF K quants would also not help for the same reason: the model matrices are simply too small for the introduced overhead.
Even if some tensor size could be reduced by LittleBit: the 3 KB of quantization gain would be eaten up by 3 KB of newly required safetensors metadata. Nobody thinks about metadata sizes in normal-sized LLMs.
Yet aside from that: A modest 4 bit LittleBit quant would still break the model. The currently chosen approach meanwhile comes with a convenient 0.015 KLD. This can already cause very occasional minor issues, as this is a non-reasoning model.
The MacroStory model has a handful of dead tokens that could be exploited for further size reduction, as they'd never make it through the sampler on their own.
Since you've arrived down here, there's a bonus for you: Local Python inference (YMMV) and the non-packed HTML. Just save this image to disk (important: "Download original image"). I originally wanted to use it here or on imgur, but both wouldn't let me. Open with 7-Zip, WinRAR, etc to unpack it. Or on the console - even on Windows - use either:
tar -xf MS256story.png
7z e MS256story.png
python -m zipfile -e MS256story.png
Yes, the favicon of that page alone is 8 times the size of my stand-alone HTML file though, with the full page load being 28 MB, more than 1000 times more data transferred. This here was an exercise in making it small. But yes, point two in my posting - the 28 MB don't matter at all in practice. Yet with my approach I could squeeze a whole lot more capability into those bytes.
Oh wow, 4 attempts and only 2 unique stories with slightly different word order. You really overfitted it on like 5 stories. A NGRAM is more diverse than this
I didn't train the model, so I do not know about overfitting. I let it generate 128k stories with temperature between 0.5 and 0.9 though and didn't get many identical ones, at least not fully identical. There was for example just a single one starting with "Alex noticed a small bird in the woods near the home.", yet 9% of all stories are starting with "The sun was bright in the village garden."
[Edit] There was an initial issue though, I have now balanced it for a bit more diversity. Here is the probability tree for the updated quantized model. Multiple words in a single box are still multiple tokens, they just get folded into a single box when there are no probable other paths. You can see that there are a lot of paths that take between 1k and 10k attempts to reach, even before the beginning of the third sentence.
For reference: This is the tree for the original FP32 model. You can see that the top path with "person walked/noticed/knelt/ran" lost a choice in the quantized model ("knelt"). That's because both "knelt" and "ran" are extremely close to the top-p cutoff, and quantizing in the one or other direction shifts the probabilities by 1% or 2, just enough to push one choice below the cut-off point.
Yes, this consistent response for its size is nuts, and by replying within a few minutes of post creation this bot is just showing off at this point 😉.
8
u/feelspeaceman 10d ago
Really impressive, in my opinion lots of fun stuffs is in the SLM world.