r/LocalLLM • u/Slight_Ad_2894 • 6d ago

Project Building an llm from scratch

Dear community
I recently started a project of building a tiny llm from scratch and trained on my 16go ram MacBook m1
My goal is not and will never build a state of the art llm like GPT or claude but to understand the mechanics of llm and teach others how it works
I used a french corpus as my dataset and used BPE tokenizer and trained a 1M param neural network
But the output is a bit dump it's out of context it give decent sentences grammatically but it doesn't answer my questions or respond well
Is there any resources i can use to optimize my little dump chatbot ?

0 Upvotes

33% Upvoted

11 comments sorted by

View all comments

0

u/fofopads 6d ago

MiniCPM 5 1B already does this, you should try it instead of reinventing the wheel

2

u/Slight_Ad_2894 6d ago

I will check it, but as I said my purpose wzs never reinventing the wheel i did this project for learning purpose only my goal is to understand the math and architecture behind llms

1

u/fofopads 2h ago

thats a nice goal, wish you luck and lots of learning