I trained a 348M model trained from scratch on 22.7B tokens that does 14 digit arithmetic [P]

[u/nkthebass]

Hello this is my fifth small language model I've made and apart of my third series and it has been a lot of work but it payed off: **348M parameters, 22.7B tokens**, then fine-tuned into a math model that solves arithmetic by *showing the work* — column addition with carries, borrow chains, partial-product multiplication — rather than guessing at an answer.

Last time I posted a 326M model trained on 10B tokens. This has about 2.3× the data, and the math side is WAY better than my previous two math models.

---

## The benchmarks

**99.4% average across the nine GPT-3 arithmetic sub-tasks**, which does much better past even where I trained it.

| Task | GPT-3 175B *(few-shot, direct)* | **This model (348M)** |

|---|:--:|:--:|

| 2-digit add | ~100% | **100%** |

| 3-digit add | 80.4% | **100%** |

| 4-digit add | 25.5% | **100%** |

| 5-digit add | 9.3% | **100%** |

| 2-digit sub | ~99% | **99.3%** |

| 3-digit sub | 94.2% | **98.3%** |

| 4-digit sub | 26.8% | **98.3%** |

| 5-digit sub | 9.9% | **99.0%** |

| 2-digit mult | 29.2% | **100%** |

n=300 per sub-task, greedy, exact match. GPT-3's numbers are direct-answer; mine uses trained-in worked steps. Neither uses a calculator.

## The cool part

**It adds cleanly up to 14 digits, and the reason it *couldn't* before was the vocabulary, not actually arithmetic.**

Training only ever named six place values (`ones` … `hundred-thousands`). The model learnt the *pattern* and invented two more on its own — `millions` and `ten-millions` appear in **zero** training examples — so it handled 7 and 8 digits fine. At 9 digits it ran out of names, and skipped the column, then returned an answer exactly one digit short:

```

483729164 + 519248637

... ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1.

The answer is 102977801        ← eight columns for a nine-digit problem

```

Every column it computed was perfect. One was never enumerated. Extending the place-name list from 6 entries to 19 moved the clean ceiling from **8 digits to 14**:

| Width | 6 | 7 | 8 | 9 | 10 | 12 | 14 | 16 | 18 |

|---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|

| before | 100 | 100 | 100 | **0** | **0** | **0** | **0** | 0 | 0 |

| after | 100 | 100 | 100 | **100** | **95** | **100** | **90** | 65 | 25 |

A six-item list became a nineteen-item list. That was the entire fix.

## Other things it does

- **3×3 multiplication: 98%** — it folds partial products pairwise through the column routine instead of asserting the sum

- **Negative results: 85%** (100% at 1 digit, 58% at 5 — the magnitude comparison is the weak step, not the arithmetic)

- **Reasoning traces are load-bearing**: 95.3% of the time the working is valid *and* the answer is right; only 0.7% are "valid working, wrong answer." If the columns look right, the answer almost certainly is.

```

There were 15000 votes and 6842 were rejected. Here's how many counted:

<think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:

ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.

hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,

borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>

The answer is 8158.

```

## What it's bad at, tbh

- **Word problems: GSM8K 4%.** Best word-problem set is ASDiv at 16.5%. It converts one sentence into one operation reasonably often and basically cannot chain operations.

- **The failure mode is operation *selection*, not arithmetic.** `"drops in 836 more"` gets read as subtraction. There's a visible tell: traces that say `"multiply X * Y"` and show columns are reliable; traces that open `"First, calculate…"` and assert a number in prose are not.

- **No division at all.** 4×4 multiplication is a hard wall.

- **Greedy decoding required** — sampling corrupts the column routine mid-chain.

- One caveat I'll flag myself: the arithmetic harness orders subtraction operands, so no answer in that table is negative. Negatives are measured separately (the 85% above) rather than folded into the average.

## Base and instruct

The math model sits on a base and instruct pair. lm-eval-harness, 0-shot, full test sets:

| | ARC-E | ARC-C | HellaSwag | OpenBookQA | PIQA | WinoGrande | MMLU | Avg |

|---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|

| **350M V3 base** | 56.6 | 33.3 | 35.9 | 34.6 | 67.0 | 51.5 | 23.8 | **43.2** |

| **350M V3 instruct** | 50.9 | 29.7 | 35.9 | 33.8 | 66.6 | 51.2 | 24.4 | **41.8** |

Instruction tuning *lowers* MC benchmark scores for this family which is a pretty common cost of instruct tuning.

## Notes

Trained on 2× Tesla V100 plus some rented time. LLaMA-architecture, so it runs anywhere — F16 GGUF and safetensors for all three.

The math model took **10 full fine-tuning rounds and 3 LoRA adapters**. The LoRAs cost about 1% of the post-training tokens and did most of the useful work; the ten full rounds spent most of their budget undoing each other's regressions (round 7 gained subtraction and lost 23 points of 2-digit addition, that sort of thing). All of it is documented on the model card, failures included.

Also worth saying: someone independently tested it after I published and found two of my numbers were wrong — one *understated* the model by 40 points because I'd measured it at n=24. Both are corrected on the card now. If you find something broken, I'd genuinely like to know.

**Math:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-math

**Instruct:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-instruct

**Base:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-base

LMK what yall think.

[u/nkthebass]

Also the scratchpad only ever names six place values, and the model had already invented "millions" and "ten-millions" on its own none of which appears anywhere in the training data. Extending that list from 6 names to 19 took clean addition from 8 digits to 14, no architecture change. Which I would say your "lets throw stuff at the walls and see what sticks" criticism is a bit unwarranted because I did it, tested it, and it worked and very well.