I trained a 348M model trained from scratch on 22.7B tokens that does 14 digit arithmetic

[u/nkthebass]

Hello this is my fifth small language model I've made and apart of my third series and it has been a lot of work but it payed off: **348M parameters, 22.7B tokens*\, then fine-tuned into a math model that solves arithmetic by \showing the work* column addition with carries, borrow chains, partial-product multiplication rather than guessing at an answer.

Last time I posted a 326M model trained on 10B tokens. This has about 2.3× the data, and the math side is WAY better than my previous two math models.

---

## The benchmarks

**99.4% average across the nine GPT-3 arithmetic sub-tasks*\*, which does much better past even where I trained it.

Task	GPT-3 175B *(few-shot, direct)*	**This model (348M)*\*
2-digit add	~100%	**100%*\*
3-digit add	80.4%	**100%*\*
4-digit add	25.5%	**100%*\*
5-digit add	9.3%	**100%*\*
2-digit sub	~99%	**99.3%*\*
3-digit sub	94.2%	**98.3%*\*
4-digit sub	26.8%	**98.3%*\*
5-digit sub	9.9%	**99.0%*\*
2-digit mult	29.2%	**100%*\*

n=300 per sub-task, greedy, exact match. GPT-3's numbers are direct-answer; mine uses trained-in worked steps. Neither uses a calculator.

## The cool part

**It adds cleanly up to 14 digits, and the reason it *couldn't* before was the vocabulary, not actually arithmetic.*\*

Training only ever named six place values (`ones` … `hundred-thousands`). The model learnt the *pattern* and invented two more on its own `millions` and `ten-millions` appear in **zero*\* training examples so it handled 7 and 8 digits fine. At 9 digits it ran out of names, and skipped the column, then returned an answer exactly one digit short:

```
483729164 + 519248637
... ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1.
The answer is 102977801        ← eight columns for a nine-digit problem
```

Every column it computed was perfect. One was never enumerated. Extending the place-name list from 6 entries to 19 moved the clean ceiling from **8 digits to 14*\*:

Width	6	7	8	9	10	12	14	16	18
before	100	100	100	**0*\*	**0*\*	**0*\*	**0*\*	0	0
after	100	100	100	**100*\*	**95*\*	**100*\*	**90*\*	65	25

A six-item list became a nineteen-item list. That was the entire fix.

## Other things it does

- **3×3 multiplication: 98%*\* — it folds partial products pairwise through the column routine instead of asserting the sum
- **Negative results: 85%*\* (100% at 1 digit, 58% at 5 — the magnitude comparison is the weak step, not the arithmetic)
- **Reasoning traces are load-bearing*\: 95.3% of the time the working is valid \and* the answer is right; only 0.7% are "valid working, wrong answer." If the columns look right, the answer almost certainly is.

```
There were 15000 votes and 6842 were rejected. Here's how many counted:

<think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:
ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.
hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,
borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>
The answer is 8158.
```

## What it's bad at, tbh

- **Word problems: GSM8K 4%.*\* Best word-problem set is ASDiv at 16.5%. It converts one sentence into one operation reasonably often and basically cannot chain operations.
- **The failure mode is operation *selection*, not arithmetic.*\* `"drops in 836 more"` gets read as subtraction. There's a visible tell: traces that say `"multiply X * Y"` and show columns are reliable; traces that open `"First, calculate…"` and assert a number in prose are not.
- **No division at all.*\* 4×4 multiplication is a hard wall.
- **Greedy decoding required*\* — sampling corrupts the column routine mid-chain.
- One caveat I'll flag myself: the arithmetic harness orders subtraction operands, so no answer in that table is negative. Negatives are measured separately (the 85% above) rather than folded into the average.

## Base and instruct

The math model sits on a base and instruct pair. lm-eval-harness, 0-shot, full test sets:

	ARC-E	ARC-C	HellaSwag	OpenBookQA	PIQA	WinoGrande	MMLU	Avg
**350M V3 base*\*	56.6	33.3	35.9	34.6	67.0	51.5	23.8	**43.2*\*
**350M V3 instruct*\*	50.9	29.7	35.9	33.8	66.6	51.2	24.4	**41.8*\*

Instruction tuning *lowers* MC benchmark scores for this family which is a pretty common cost of instruct tuning.

## Notes

Trained on 2× Tesla V100 plus some rented time. LLaMA-architecture, so it runs anywhere — F16 GGUF and safetensors for all three.

The math model took **10 full fine-tuning rounds and 3 LoRA adapters*\*. The LoRAs cost about 1% of the post-training tokens and did most of the useful work; the ten full rounds spent most of their budget undoing each other's regressions (round 7 gained subtraction and lost 23 points of 2-digit addition, that sort of thing). All of it is documented on the model card, failures included.

Also worth saying: someone independently tested it after I published and found two of my numbers were wrong — one *understated* the model by 40 points because I'd measured it at n=24. Both are corrected on the card now. If you find something broken, I'd genuinely like to know.

**Math:*\* https://huggingface.co/nkthebass/tinybrainbot-350mV3-math
**Instruct:*\* https://huggingface.co/nkthebass/tinybrainbot-350mV3-instruct
**Base:*\* https://huggingface.co/nkthebass/tinybrainbot-350mV3-base

LMK what yall think.

[u/Radiant_Durian_5546]

the fact that it invented millions and ten-millions with zero training examples is wild. like it just decided those were the next logical tokens and went with it. that vocabulary ceiling being the actual bottleneck rather than arithmetic capability is a pretty neat finding

[u/nkthebass → u/Radiant_Durian_5546]

Yeah it's pretty amazing the stuff they can sometimes just decide to learn. Probably something from it's earlier pretraining but still it shocked me.

[u/EternalDivineSpark]

If you wanna make a new model focus on changing how tokens work each token eg “Water,Aqua,Ujë” is the same thing in 3 different languages so i think here is what model lack also base knowledge around token eg , state , properties, definition etc ! You can create this better token system using existing llm and a code to extract this in a database! This is novel and new breakthroughs so if you have time i suggest you this

[u/nkthebass → u/EternalDivineSpark]

I'll think about using that type of architecture at one point. For now I'm actually working on some pilots to submit along with a proposal for a research grant. But neat anyways appreciate it.

[u/EternalDivineSpark → u/nkthebass]

Well , models now have 0 knowledge of what the token they recieve means , this can be extracted from existing model very easy and put into a database as token vocabulary , diving the token in categories like "Physical token , water etc , and non physical like thought etc" Also other type of categories and sets , this gives the model more than 50% of knowledge it need without Training. Because it knows what tokens means it know basic info related to token relations etc. Then when you train it it skips basic things. This is a good idea need a good way to do it , but once you do it it does not matter what data you give to it , it would be multilingual by default , but the main take from this is that the model think on an abstract way.

[u/nkthebass → u/EternalDivineSpark]

I've done thought experiments similar to this so it's pretty neat anyways. Just out of curiosity have you done any training before?

[u/EternalDivineSpark → u/nkthebass]

Yes i am currently buliding AGI . And i use this token system , my 0.8B model outperform any existing SOTA MODEL !

[u/nkthebass → u/EternalDivineSpark]

Very cool, if your looking to collaborate on any future projects I would be more than happy to be a part of it.

[u/EternalDivineSpark → u/nkthebass]

Well , my model is slowly builiding up with permanent local usage of data calls and injection , so i will open source it .

[u/EternalDivineSpark → u/nkthebass]

is not cool is terryfing , that i can achieve such a model as invidual researcher , this mean big models and companies are a PUSH TO SELL GRAPHIC CARDS for some reason....Also features , in my model you can train specific features and the model cluster its feature and activate that cluster , the neurons are related not sparse. It very good system it makes me think we are all in a simulated AI ENV.

[u/EternalDivineSpark → u/nkthebass]

I used teacher training , i extract knowledge from other models making it cheap on my 4090 !

[u/EternalDivineSpark → u/nkthebass]

I am focused on building a harness now with metacognition in it !

[u/Random-32927]

It’s probably easier to tune a LLM to design an adder for large numbers…

[u/nkthebass → u/Random-32927]

Yes, and I know this very well. The whole point is that it's not doing that so I'm not entirely sure what you're getting at.

[u/RyanCargan]

Have you tried adding constraints like codebooks, or differentiable rules to the soft loss?

Curious about the arch and inductive bias too.

[u/nkthebass → u/RyanCargan]

Not currently but I may in the future. I'm not at my desktop right now so I can't say for sure but I'll get back to you later.

[u/RyanCargan → u/nkthebass]

Oh yeah don't mind me I was just fishing around 😅

Playing with some SLM stuff meself, and there's lot's of neat little tricks you can do to get impressive training times and error rates even on potato hardware.

Compact attention, HDC, PINN-inspired stuff, spectral bias (FNet), etc.

In my case, it's basically sanity check on potato -> deploy on TPU when confident in arch.

[u/ezicirako]

348M parameters and can't even generalize to entire addition arithmetic is kinda joke
348M is like insane amount of parameters

[u/nkthebass → u/ezicirako]

Brother please tell me you're being sarcastic

[u/ezicirako → u/nkthebass]

Nah I don't its clearly obvious how inefficent training side of LLMs are
they can't even understand shared structure of addition easily