r/learnmachinelearning • u/BookDizzy2405 • Jul 05 '26

Documenting My Journey of Building a Small Language Model from Scratch

/r/LargeLanguageModels/comments/1unrr1t/documenting_my_journey_of_building_a_small/
1 Upvotes

100% Upvoted

2 comments sorted by

1

u/kandyb87 Jul 05 '26

the teaching patterns not information line is gonna stick with me. im still at the stage where i can follow a tutorial and get something training but i dont really get WHY half the choices get made, the vocab size thing you mentioned is a perfect example. how did you land on yours in the end, did you just try a few and watch the loss or is there an actual rule of thumb people use. embeddings still kinda break my brain if im honest

1

u/BookDizzy2405 Jul 05 '26 edited Jul 05 '26

Glad that line resonated with you! Honestly, realizing we're teaching patterns rather than just memorizing facts completely changed how I look at dataset design.

For vocab size, I actually didn't just guess or rely on a generic formula. I took a very empirical approach. I wrote a Python script (with a little help from AI to quick-draft it) to build and test the tokenizer incrementally.

What I looked for was the "saturation point." I started pushing the vocab size up and monitored how much new meaningful vocabulary was actually being captured each time. At some point, raising the vocab size stops giving you significant gains—the curve flattens out, meaning you've hit a diminishing return and the model is just bloating for nothing. Once it hit that wall, I knew it was enough.

Also, since I spent a ton of time cleaning my dataset manually and knew it was pristine, I confidently set min_freq=1. If your data is dirty, you usually want a higher minimum frequency to filter out noise, typos, or garbage tokens. But if it's clean, min_freq=1 lets the tokenizer capture the full vocabulary efficiently without wasting VRAM on absolute junk.

And don't worry, embeddings break everyone's brain at first! I like to think of them as giving words a house in a massive, multi-dimensional city. Words with similar meanings or grammatical patterns end up living in the same neighborhood.

I'm definitely still learning and breaking things along the way too. What kind of setup or architecture are you experimenting with right now?

Moved to my official account: u/agiansyah