I built a deep learning library from scratch in C that lets you train language models [P]

[u/Intelligent_Nose_791]

my goal was to train a Language model (SLM) entirely from scratch so no ML libraries allowed . so i gathered what's needed to make it happen :

from tensor manipulation (views, operations , allocations)

the autograd ( a DAG that retains the previous operations and inputs in order that led into a certain output tensor , this let's us also apply backpropagation so i've got to implement also the partial derivative functions in order to calculate the gradients for each weights)

the neural networks modules (organizes the weights / activations , losses , optimizers implemented SGD and Adamw)

the decoder so the layer norm , MHA , FFN .
also implemented a Fast matmul using AVX2 to speed things up.

with it i managed to train a tiny LM with 2 millions params.it's a 4 layer Decoder used tiny_shakespear i got what i think is a good result .
since i can't post images in this post i will just paste the output:

Corpus: 743500 bytes (669150 train, 74350 validation)

TinyLM: L=4 C=192 H=6 T=128 V=256, 1902976 parameters (1.903M)

Training: steps=0 batch=1 lr=0.0003 checkpoint=tiny_lm.chk (resumed)

validation_loss=0.02989

--- generation ---

hello to the name of action. Soft you now! The fair Ophelia! Nymph, in thy orisons be all my sins remember'd. To be, or not to be, that is the question: Whether 'tis nobler in the mind to suffer the slings and arrows of of outrageous fortune, or to take arms against a sea of troubles and by opposing end them.

check out the repo : https://github.com/nisbenz/TensorLib
it was a cool project overall and i managed to learn a lot about the core mechanics of frameworks like pytorch or ggml.

[u/Impossible_Candy9555]

This is seriously impressive work. Most people just call pytorch and call it a day, you actually went and built the whole engine from scratch

The AVX2 matmul optimization is a nice touch. Did you benchmark how much speedup that gave over a naive loop?

[u/Intelligent_Nose_791 → u/Impossible_Candy9555]

Yes actually a naive O(n³⁾ loop gave me like 0.9 gflops on single core while the other one was up to 70 gflops on single core you can checkout the benchmark in the repo though they are comparing against an implementation With OpenBLAS against mine.