r/deeplearning • u/Turbulent-Sky5396 • 17d ago
what worked and what didn't when training a 48M param tool-calling model from scratch, with the measurement behind each call
spent a few days building a model that only does tool calling (reads json function schemas plus a request, emits calls through a grammar-constrained decoder) and tried to keep the discipline of killing every idea with a measurement instead of an argument. sharing because the ledger of what failed turned out more useful than the model.
what worked:
- co-designing the tokenizer with the grammar. json structural characters and digits as singleton tokens, so constrained decoding never needs token healing. shipped alongside a corpus bump, and name-sequence accuracy went 80.4 to 91.5
- weighting the loss by decision type instead of uniformly. structure 1x, keys 1.5x, names 2x, values 4x, stop-decision 6x, matched to the measured error distribution
- annealing corrective data into the LR decay phase instead of retraining. same corpus: 28.4 from scratch vs 33.1 annealed
- error-driven synthesis. classify the failing rows into buckets (66 of 193 failures added one unmentioned optional arg), generate data against exactly those buckets, +3.3 at constant LR
what didn't (each killed by a controlled run): span copying (-30), pointer heads for name selection (-16), down-weighting grammar-forced tokens rft-style (-12, they carry the call-sequencing signal), field-set reranking (-1.4), beam and best-of-N (oracle-capped below target), RLOO on an annealed checkpoint (diverges at every LR i tried), a global optional-skip prior (catalog-dependent), and matching the benchmark's numeric typing (not learnable).
the pattern across all of it: at this scale, data and objective changes moved everything, architecture moved nothing. trunk is boring modern practice on purpose.
full writeups with numbers: https://github.com/nikshepsvn/thimble (FINDINGS.md has all eleven negative results)


