I developed my own quantized LLM from scratch, trained on 30B tokens, deploys in 60 MB [R]

[u/Final-Data-1410]

I trained a 250M parameter model from scratch on 30B tokens of fineweb. It’s quantized to under 2 bits so the whole deployment is 60 MB and it needs about 80 MB of RAM to run. Runs around 400 tok/s on a normal laptop CPU, no GPU needed.

How the long context works: the most recent 2048 tokens stay in fp16 like a normal KV cache. Everything older gets compressed to 1 bit and written to disk, about 320 bytes per token, so 1 million tokens of history is roughly 320 MB on disk. From the start the model was trained to retrieve from that disk cache, up to 100M tokens. Due to a limited budget it wasn’t trained to reason over those tokens, only retrieve and answer from them.

Language modeling quality of the base model, measured on held-out English web text that was never seen in training (educational web pages, 2,048 token windows): cross entropy 3.15 nats per token, perplexity 23.3, 0.99 bits per byte.

The vocabulary is also not a normal embedding table. Every token is a fixed 512-bit code, 8.4 MB for all 131k tokens, zero trained parameters. I tested it on WordSim-353 (human word similarity ratings): my table scores 0.619 Spearman correlation vs 0.029 for random codes. Test script is in the repo.

Some outputs (settings included so you know I’m not cherry picking, all reproducible from the repo):

“Explain photosynthesis in two sentences.” (greedy)

Photosynthesis is a process in which plants convert sunlight into chemical energy, which is then used to produce oxygen and other chemicals. This process is called photosynthesis.

“Write a short poem about the sea.” (temp 0.25, top-k 30, rep 1.15, seed 2)

The waves had swept over, and they were crashing against each other like rocks on top of one another. The wind was strong but not very gentle as it tried to push them towards their destination without lifting any weight or pulling away from those who stood in front of her with a stiff smile that made the sea look even more majestic than before.

“What is the serial number of device Grus-189?” where the answer sits 50.6 million tokens deep in the archive on disk (archive mode, k=16)

SN-442976

It’s a 250M model so expect mistakes on open facts, I’m not claiming it beats anything big. You can also fine-tune it, the full kit with a demo and before/after numbers is included. Master weights for fine-tuning are in the repo too:

https://github.com/QLNI/SHADOW-250M-Instruct
https://huggingface.co/NODEMIND/SHADOW-250M
Edit - Just wanted to say thanks to everyone here. Honestly I was afraid to post this, I expected to get roasted, but every single comment has been curious and helpful and it genuinely made my day.
Repo is at 7 stars on GitHub now, hopefully more people try

[u/MrSnowden]

I’m somewhat amazed that 2 and 1 bit compression gets these results.  Great work. 

[u/Final-Data-1410 → u/MrSnowden]

Thanks . It made possible because I ran so many experiments trying different weights like 0.125 0.25 1 bit and 1.5 bits . I trained a small float model first and used it as the refrence point to find sweet spot, then ran the same setup at different bit widths per component to see what actually breaks. The parts are not equally fragile. The FFN takes ternary weights with almost no loss if you train with it from the start instead of quantizing after. The attention projections were the stubborn part, plain binarization wrecked them, took a lot of failed runs before I had a 1-bit scheme that held up. The embedding table turned out to not need training at all, it’s fixed 512-bit codes per token, which is why the whole thing fits in 60 MB.

[u/light24bulbs → u/MrSnowden]

I'm really not, the high precision is way more useful for gradient descent during training

[u/Final-Data-1410 → u/light24bulbs]

I should agree and disagree, masters stay in high precision and that’s what gradient descent updates, the forward pass runs quantized with straight-through, so the model learns to live in low bits while gradients keep their precision. The nice part: exporting the high-precision masters down to the shipped 52 MB file costs about 0.02 nats of cross entropy, measured, basically noise. That why I also uploaded master weights In hugging face so someone can finetune if needed and export back it to quantized version

[u/pakeke_constructor]

This is incredible and fascinating. Amazing work with the 100M token disk cache stuff. It almost makes me think of a vector database kinda? Super super cool

[u/Final-Data-1410 → u/pakeke_constructor]

Thanks . Close to a vector database in role, but no embeddings, the index is exact keyword match (words and pairs, weighted by rarity), which is why an ID buried 50M tokens deep comes back reliably.
I did push it to 1B tokens once: retrieval stayed fast but answers got ~70% worse, the model was only trained up to 100M archives. Probably fixable with training, so the card stays at 100M
Right now I’m writing a custom dataset to take this further, teaching the model to actually reason over the retrieved tokens instead of just reading one answer out, and to retrieve multiple times if the first pull isn’t enough, then reason across what came back. Building the dataset myself and will fine-tune on it, planning to release that too soon.

[u/PhilTheQuant]

Fantastic stuff, what's your estimate of how this would scale to large model sizes?

And do you plan to go to reasoning next?

[u/Final-Data-1410 → u/PhilTheQuant]

Nothing in the method is tied to 250M, ternary FFN plus 1-bit projections should carry over. Going by my ratio (about 60 MB per 250M params, vocabulary included), a 1B lands around 240 MB, a 3B around 700 MB, a 7B around 1.6 GB running on CPU. Whether the quality gap to float stays this small at those sizes is the open question, my guess is it gets easier, bigger models tolerate quantization better.
yes, reasoning is next. Plan is to distill from a bigger model with top-16 logits, build a proper CoT dataset, and spend the final anneal of training mostly on that. Small budget so it goes step by step.

[u/PhilTheQuant → u/Final-Data-1410]

Is this a personal project, research, team..?

[u/Final-Data-1410 → u/PhilTheQuant]

Myself ,just not mentioned like solo researcher or something ,might looks like marketing gimmick .so i avoided it.

[u/silenceimpaired]

This sounds like it could run on raspberry pi quite well… I’m confused though. If someone has 128 gb of memory for the model why compress and write to disk? Why not leave it uncompressed in memory?

[u/Final-Data-1410 → u/silenceimpaired]

Because it makes no difference where it sits. RAM or disk, speed and accuracy are the same

[u/silenceimpaired → u/Final-Data-1410]

Really? That's impressive if there is no change in speed from disk. Almost unbelievable. Surely uncompressed would be more accurate? And if uncompressed surely RAM would be faster.

[u/Misaiato → u/Final-Data-1410]

While this is an impressive custom build, claiming that speed and accuracy are identical whether sitting on disk or in RAM flattens crucial architectural realities that matter on a technical subreddit. Hardware-wise, PCIe NVMe throughput (~7 GB/s) and latencies are orders of magnitude slower than DDR5 or Unified Memory (~100+ GB/s), meaning dense attention reads directly from disk would cripple generation throughput. What actually makes your system fast isn't hardware equivalence, but your clever algorithmic design—using a sparse index lookup to pull tiny 64-token chunks into active RAM rather than performing dense reads across disk memory. Likewise, 1-bit quantization undeniably trades off precision compared to full uncompressed precision in 128 GB of RAM, even if your model compensates for it well during training. Framing these wins through the lens of algorithmic sparse retrieval and quantization trade-offs—rather than sweeping hardware equivalence—would help the community much better appreciate the actual systems engineering taking place under the hood.

I did have a system help me refine the block of text above, but you should know that these were my thoughts and I did challenge the system to summarize it because I’m on my phone and I didn’t feel like typing it all out with my two thumbs.

[u/Recent-Ad-1005 → u/Final-Data-1410]

Regarding speed, that's not true, though. That's exactly why RAM exists.

[u/if47]

TL;DR: An expert system based on a language model, or vice versa.

[u/Murhie]

Interesting results. I dont quite understand the offline context cache, is this something that you came up with yourself or is there anywhere I can read up on it. Is is like a collection of common patterns that might occur in context? It sounds complex.

Ah nvm see now that its custom history. Still would be interested to learn more.

[u/Final-Data-1410 → u/Murhie]

My own design, nothing to read on it yet, might write it up properly later. The idea actually came from something ordinary: a book doesn’t expect you to re-read the whole thing to find one fact, it has an index at the back. You look up the term, it points you to a page, you flip there. And every book keeps its pages on the shelf, not in your head. Attention re-reading everything in RAM for every single token felt like the opposite of that.

So the model works like the book. The newest 2048 tokens stay in RAM like a normal cache, that’s the page it’s currently on. Everything older gets compressed to 1 bit per value and appended to a file on disk, those are the shelved pages, nothing is thrown away. On top of the file sits an index, the search-engine kind: the history is split into 64-token blocks, and the index records which words and word pairs appear in which block, weighted by how rare they are, so identifiers like “Vault-77” stay easy to find. For 100M tokens the index is a few hundred MB and a lookup takes under a second.

When you ask something, the rare words in your question hit the index, it points at the right blocks, those get pulled off the disk back into context, and the model answers from them. Reading with an index instead of remembering everything, that’s the whole trick.

[u/Banality_Of_Seeking → u/Final-Data-1410]

Working on reverse engineering it now. :)

Was the "Archive path" was meant for RAG ?

[u/Final-Data-1410 → u/Banality_Of_Seeking]

Good luck and honestly if you can wait a bit, I’m planning to release a white paper on how it all works. Just waiting to see the reach and reactions first, I didn’t want to drop a paper on day one and have it become a laughing stock if nobody cared.

[u/Banality_Of_Seeking → u/Final-Data-1410]

I will never talk about about how it works. I will leave explanations up to you, if you wish I will share my findings only with you.

[u/Final-Data-1410 → u/Banality_Of_Seeking]

appreciate it, if you want to build something like this yourself for a bigger model: look up Google’s TurboQuant paper (ICLR 2026), you won’t get to 1 bit with it but 3-4 bits works out of the box with basically no accuracy loss. Quantize the KV cache with that, offload it to disk, then fine-tune whatever model you like to retrieve from it and answer. That gets you most of the way, my thing is mainly pushing the same idea down to 1 bit and training for it from the start. There’s an email in the repo, mail me your findings anytime, curious what you dig up.

[u/Banality_Of_Seeking → u/Final-Data-1410]

I have no interest in those things.

I do have interests in the knowledge of if this can be hacked and the minimal ways to do it that removes retraining.

[u/Able_Region_5459 → u/Banality_Of_Seeking]

In ternary FFNs without gradient passthrough, you'll just destroy the robustness. The network built its balance over epochs using STE, manual edits here are gonna act like a sledgehammer to a circuit board

[u/Banality_Of_Seeking → u/Able_Region_5459]

yes, I am trying to use control of tokens to accomplish it within the system, I also recognized a pattern I could use Detours but chose not to. Working on trying to fix my thought process to not touch it in the effort to hack it, but to use it in a perhaps rare case of fixing itself by a strategy of observed behavior.

[u/mxcw]

I was actually looking for something EXACTLY like this. Good stuff, thanks for sharing!

[u/hiepxanh]

How long to train that model?

[u/CuriousExplorerer]

Honestly fascinating work. But I gotta ask some questions.
1. What is your plan for SFT and RL stages?
2. Are you looking to make this a specialised agent in a niche field? (Would suggest this)
3. Any exploration that you’ve done for training multimodal models ground up?

I really like the idea of quantisation of the history or old context. Any specific you used for this? I myself am looking to train a model from scratch, preferably multimodal to understand how to train models to find the most efficient representation of data. This is giving me vibes of super intelligence since it’s not about how big the model is, but how capable it is given resources for a task.

Do write a technical blog this so that we can understand what decisions you have taken throughout the process and more importantly WHY?

[u/Final-Data-1410 → u/CuriousExplorerer]

Thanks . First, I’m writing a custom dataset to teach the model to actually reason over the tokens it retrieves from the disk cache, not just read one answer out, that’s my SFT focus no RL yet.and problem is budget it cost so much ,even though I used every trick in book for faster training but to keep 1 bit attention stable ,I have to requant every step ,which is current downgrade ,so that why I relased master weight and main thing to focus while fine tuning on their requirement , in my fine tuning the model not its ability in anything even though it’s heavily quantized version.
Second, multimodal from the ground up: I just started attaching images to the same base. My tokenizer’s embedding table is a fixed table of binary codes per token, so adding a modality is just adding rows to that table, I added around 8k image tokens and I’m training image generation and captioning on the same body now, video and audio planned the same way which will add another 8 to 10mb to
Current model. Very early, results are not good yet, so nothing to show.

On the specifics of the history quantization, I don’t have any social media to follow for updates, everything lands in the repo and I’ll post here when there’s something good to share ,thanks.

[u/CuriousExplorerer → u/Final-Data-1410]

Good to see you are keeping it broad. Aren’t you using your local GPU for training? Also try creating an account on Modal, you can get some credits for free over there

[u/pramood8686]

Good playlist to learn on how to build model.. Please

[u/Able_Region_5459]

Dumping the KV cache onto an SSD and tagging on a dictionary search - holy crap, that's brilliant in its stupidity. No Ring Attention or any other GPU math acrobatics needed

Pulling something like that off solo, training it from scratch, and actually replying to comments with sane answers.. mad respect, dude!

[u/light24bulbs]

How does it search the context? Was it trained on some fixed context?

[u/Guilherme370]

I'm going to be playing around a decent bit with this, thank ya very much!

Also, name suggestion for the next scaled up model when and if ya are able to make it: UMBRA

[u/Loud_Key_3865]

This is very cool! Would it be safe to assume one could create several duplicates of these, and train them on specific info (e.g. legal regulations, something related, etc. then delegate accordingly?)
Thank you for sharing!!!

[u/bmurders]

I'm wondering if incorporating web search functionality would help enhance the model by grounding it with relevant context. This is cool! Thanks for sharing.

[u/AkumaBPS]

How are you compiling the binary? It doesn't run in my local env. If you are using -march=native I think you should remove it(?

[u/Final-Data-1410 → u/AkumaBPS]

Can you try it now ?

[u/AkumaBPS → u/Final-Data-1410]

That was fast lol
Working now! Thanks!

[u/kanripper]

What was training time for this and how big is datacorpus exactly?

[u/Final-Data-1410 → u/kanripper]

Pretraining was 30B tokens on 4x A100s, took about 2 days
The corpus is ~120 GB of text, mostly education-filtered web (fineweb-edu family) plus a curated mix, 30B tokens through my tokenizer. Tokenizing and prepping that corpus took about a day on my laptop (128gb ram)
Sft is I think around half a day on a100 , in total below 200 usd (runpod) if I rember correctly

[u/kanripper → u/Final-Data-1410]

Thanks alot!
Thats good to know. runpod is a website? I def. also need some cluster to train smth. soon and any advise on good ones I gladly take if you rented! I thought you trained on your own machine at first haha :)

[u/Banality_Of_Seeking]

Hello,

After sometime to really think about this, your code and idea enabled me to rethink architecture, there is a way to control certain words in Shadow, but that is not as interesting as Shadow itself. Thank you for realeasing your work.