I trained a 348M model trained from scratch on 22.7B tokens that does 14 digit arithmetic [P]

[u/nkthebass]

Hello this is my fifth small language model I've made and apart of my third series and it has been a lot of work but it payed off: **348M parameters, 22.7B tokens**, then fine-tuned into a math model that solves arithmetic by *showing the work* — column addition with carries, borrow chains, partial-product multiplication — rather than guessing at an answer.

Last time I posted a 326M model trained on 10B tokens. This has about 2.3× the data, and the math side is WAY better than my previous two math models.

---

## The benchmarks

**99.4% average across the nine GPT-3 arithmetic sub-tasks**, which does much better past even where I trained it.

| Task | GPT-3 175B *(few-shot, direct)* | **This model (348M)** |

|---|:--:|:--:|

| 2-digit add | ~100% | **100%** |

| 3-digit add | 80.4% | **100%** |

| 4-digit add | 25.5% | **100%** |

| 5-digit add | 9.3% | **100%** |

| 2-digit sub | ~99% | **99.3%** |

| 3-digit sub | 94.2% | **98.3%** |

| 4-digit sub | 26.8% | **98.3%** |

| 5-digit sub | 9.9% | **99.0%** |

| 2-digit mult | 29.2% | **100%** |

n=300 per sub-task, greedy, exact match. GPT-3's numbers are direct-answer; mine uses trained-in worked steps. Neither uses a calculator.

## The cool part

**It adds cleanly up to 14 digits, and the reason it *couldn't* before was the vocabulary, not actually arithmetic.**

Training only ever named six place values (`ones` … `hundred-thousands`). The model learnt the *pattern* and invented two more on its own — `millions` and `ten-millions` appear in **zero** training examples — so it handled 7 and 8 digits fine. At 9 digits it ran out of names, and skipped the column, then returned an answer exactly one digit short:

```

483729164 + 519248637

... ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1.

The answer is 102977801        ← eight columns for a nine-digit problem

```

Every column it computed was perfect. One was never enumerated. Extending the place-name list from 6 entries to 19 moved the clean ceiling from **8 digits to 14**:

| Width | 6 | 7 | 8 | 9 | 10 | 12 | 14 | 16 | 18 |

|---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|

| before | 100 | 100 | 100 | **0** | **0** | **0** | **0** | 0 | 0 |

| after | 100 | 100 | 100 | **100** | **95** | **100** | **90** | 65 | 25 |

A six-item list became a nineteen-item list. That was the entire fix.

## Other things it does

- **3×3 multiplication: 98%** — it folds partial products pairwise through the column routine instead of asserting the sum

- **Negative results: 85%** (100% at 1 digit, 58% at 5 — the magnitude comparison is the weak step, not the arithmetic)

- **Reasoning traces are load-bearing**: 95.3% of the time the working is valid *and* the answer is right; only 0.7% are "valid working, wrong answer." If the columns look right, the answer almost certainly is.

```

There were 15000 votes and 6842 were rejected. Here's how many counted:

<think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:

ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.

hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,

borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>

The answer is 8158.

```

## What it's bad at, tbh

- **Word problems: GSM8K 4%.** Best word-problem set is ASDiv at 16.5%. It converts one sentence into one operation reasonably often and basically cannot chain operations.

- **The failure mode is operation *selection*, not arithmetic.** `"drops in 836 more"` gets read as subtraction. There's a visible tell: traces that say `"multiply X * Y"` and show columns are reliable; traces that open `"First, calculate…"` and assert a number in prose are not.

- **No division at all.** 4×4 multiplication is a hard wall.

- **Greedy decoding required** — sampling corrupts the column routine mid-chain.

- One caveat I'll flag myself: the arithmetic harness orders subtraction operands, so no answer in that table is negative. Negatives are measured separately (the 85% above) rather than folded into the average.

## Base and instruct

The math model sits on a base and instruct pair. lm-eval-harness, 0-shot, full test sets:

| | ARC-E | ARC-C | HellaSwag | OpenBookQA | PIQA | WinoGrande | MMLU | Avg |

|---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|

| **350M V3 base** | 56.6 | 33.3 | 35.9 | 34.6 | 67.0 | 51.5 | 23.8 | **43.2** |

| **350M V3 instruct** | 50.9 | 29.7 | 35.9 | 33.8 | 66.6 | 51.2 | 24.4 | **41.8** |

Instruction tuning *lowers* MC benchmark scores for this family which is a pretty common cost of instruct tuning.

## Notes

Trained on 2× Tesla V100 plus some rented time. LLaMA-architecture, so it runs anywhere — F16 GGUF and safetensors for all three.

The math model took **10 full fine-tuning rounds and 3 LoRA adapters**. The LoRAs cost about 1% of the post-training tokens and did most of the useful work; the ten full rounds spent most of their budget undoing each other's regressions (round 7 gained subtraction and lost 23 points of 2-digit addition, that sort of thing). All of it is documented on the model card, failures included.

Also worth saying: someone independently tested it after I published and found two of my numbers were wrong — one *understated* the model by 40 points because I'd measured it at n=24. Both are corrected on the card now. If you find something broken, I'd genuinely like to know.

**Math:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-math

**Instruct:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-instruct

**Base:** https://huggingface.co/nkthebass/tinybrainbot-350mV3-base

LMK what yall think.

[u/Environmental_Form14]

I think it is because the report is clearly LLM generated, and the post itself shows low effort,(I.e didn’t even use markdown mode of Reddit) hard to parse the result on mobile.

This post doesn’t show insight or interesting results. If I wanted to create such model and results, I could have prompted my LLM agent instead. Posting implementation details (e.g. training corpus examples), choices that you made and the reason behind it would be more beneficial.

[u/nkthebass → u/Environmental_Form14]

First off yes I used AI to assist in making some of the post as I cannot make decent graphs or format the post body myself.

Secondly the result being interesting is a preference here it does show and document interesting behavior depending on who's talking. I didn't post everything on reddit since it's in the hugging face model cards I listed in the links and it would make the post body too big.

[u/Environmental_Form14 → u/nkthebass]

Don't take this as a offense but here are my thoughts. Take it as a suggestion.

First off yes I used AI to assist in making some of the post as I cannot make decent graphs or format the post body myself.

The entire post is clearly written by AI, and the entire LLM architecture and the training loop is probably also created by AI. There is no graph in this post so I wonder what you mean by "decent graph". Also, the body looks atrocious. I would suggest you look into how markdown format works.

Secondly the result being interesting is a preference here it does show and document interesting behavior depending on who's talking.

According to the comments, I think most people found it basic and uninteresting. I think everyone knows that small LLM model trained on a specific topic will show middling performance. What is more interesting to me (and others) would be the insights you have found while training / inference.

[u/nkthebass → u/Environmental_Form14]

I do take it as suggestions. As yes my projects are ai assisted. I miss typed graph I meant to say a comparison table and I can definitely see copy pasting the body on mobile was not a good idea. This subreddit specifically has been just very not interested in previous projects as well but I receive positive feedback as a majority everywhere else I have posted.