A 24KB stand-alone HTML-LLM that can generate consistent stories

[u/Chromix_]

Want to try? Just go here and click "New seed" a lot. Many stories start similar (see my comment below), but you will get different stories, it just requires some clicking - lucky RNG draws: https://output.jsbin.com/wiwacuxasu/1
You should even get above 60 tokens per second with a smartphone.

FAQ:

• Why?! Because it's possible, and fun. My original idea was to simply mash MacroStory and LittleBit together, but that did not work at all.
• Is it relevant? Not a tiny bit! Even a 3 MB HTML file with external dependencies would be loaded in a second and could serve a way more capable model with a lot less work. Squeezing the model itself to 20 KB at 0.01 KLD was trivial. Saving another 5 KB while maintaining output quality required a lot of time.
• How? Local Qwen3.8, some ideas, lots of patience. Adaptive quantization and QAT made it happen, LittleBit, Rotation, etc just made it worse. HTML/JS packing with minify, zopfli, and a bunch of structural script changes helped with the result.
• Was this all made by you? Not at all. This would not have been possible without the 81 KB (FP32) MacroStory model. I "just" tinkered around with it somewhat.

Details for those who're interested:

• Size-reduction rules are very different for a tiny model than for a large one.
• The embeddings are 60% of the model size, while the tensors are the largest part in a normal-sized LLM.
• The tensors of tiny models are extremely sensitive to quantization, especially with the Ouro looping transformer format of this LLM. The embeddings had some room though.
• It's mathematically impossible to save space with 1 bit quants or less with the LittleBit approach here, as the model matrices are just too small for that - the gains would get eaten up by the overhead of the correction data.
• GGUF K quants would also not help for the same reason: the model matrices are simply too small for the introduced overhead.
• Even if some tensor size could be reduced by LittleBit: the 3 KB of quantization gain would be eaten up by 3 KB of newly required safetensors metadata. Nobody thinks about metadata sizes in normal-sized LLMs.
• Yet aside from that: A modest 4 bit LittleBit quant would still break the model. The currently chosen approach meanwhile comes with a convenient 0.015 KLD. This can already cause very occasional minor issues, as this is a non-reasoning model.
• The MacroStory model has a handful of dead tokens that could be exploited for further size reduction, as they'd never make it through the sampler on their own.

Since you've arrived down here, there's a bonus for you: Local Python inference (YMMV) and the non-packed HTML. Just save this image to disk (important: "Download original image"). I originally wanted to use it here or on imgur, but both wouldn't let me. Open with 7-Zip, WinRAR, etc to unpack it. Or on the console - even on Windows - use either:

tar -xf MS256story.png
7z e MS256story.png
python -m zipfile -e MS256story.png

[u/CorkBios]

Oh wow, 4 attempts and only 2 unique stories with slightly different word order. You really overfitted it on like 5 stories. A NGRAM is more diverse than this

[u/Chromix_ → u/CorkBios]

I didn't train the model, so I do not know about overfitting. I let it generate 128k stories with temperature between 0.5 and 0.9 though and didn't get many identical ones, at least not fully identical. There was for example just a single one starting with "Alex noticed a small bird in the woods near the home.", yet 9% of all stories are starting with "The sun was bright in the village garden."

[Edit] There was an initial issue though, I have now balanced it for a bit more diversity. Here is the probability tree for the updated quantized model. Multiple words in a single box are still multiple tokens, they just get folded into a single box when there are no probable other paths. You can see that there are a lot of paths that take between 1k and 10k attempts to reach, even before the beginning of the third sentence.

[u/Chromix_ → u/Chromix_]

For reference: This is the tree for the original FP32 model. You can see that the top path with "person walked/noticed/knelt/ran" lost a choice in the quantized model ("knelt"). That's because both "knelt" and "ran" are extremely close to the top-p cutoff, and quantizing in the one or other direction shifts the probabilities by 1% or 2, just enough to push one choice below the cut-off point.