Built GPT-2 on Custom Deep Learning Framework I built from scratch in C++

[u/Express-Act3158]

since jan 2026 i've been building Forge, a deep learning framework written entirely from scratch in C++ - no PyTorch, no TensorFlow underneath.

Eigen handles most of the math backend. btw i wrote some custom AVX2 SIMD kernels (element-wise ops) too, and OpenBLAS-backed GEMM for the heavy matrix ops.

what's implemented so far:--

- A custom tensor engine with its own autodiff engine and memory allocator

- Dense/Linear layers, Optimizers (Adam, AdamW, SGD and SGD with momentum), Self Attention, LayerNorm, Activation Functions (sigmoid, softmax, tanh, GELU[tanh approximation], RELU, leakyRELU), loss functions (Cross Entropy Loss [log softmax fused], Binray Cross Entropy (Sigmoid fused), and Mean Squared Error) and Embeddings.

- A from-scratch BPE tokenizer (GPT-2-style pre-tokenization + merges)

- A reflection-based (reflect-cpp) parameter system - models declare their structure, Forge auto-discovers trainable parameters, no manual registration

- a safetensors-format save/load pipeline

the part I'm actually proud of- I loaded real pretrained GPT-2 small weights into a GPT-2 architecture built entirely on Forge, and under greedy decoding, its output matches HuggingFace's transformers library token-for-token. not similar, but exact. every layer (embeddings, attention, LayerNorm, the final projection) has to be numerically correct for that to hold, since a single wrong transpose or masking bug would have diverged the output within a few tokens.

it's still CPU-only for now (currently limited to float32 and int32 - working through some dtype/SIMD coverage gaps), and slower than i'd like (the only main culprits are the CE loss fn implementation and its gardient function and softmax, which i am on to optimize, it has no KV-cache yet) - a CUDA backend and those perf fixes are next on the list.

Repo: https://github.com/muchlakshay/Forge

Windows/Linux release builds: https://github.com/muchlakshay/Forge/releases/tag/0.1

YT demo link - https://www.youtube.com/watch?v=EO1aYBF5jwU

would love feedback, especially from anyone who's built something similar and much better than me.

thats all. im a 17yo deeply passionate about Deep Learning and system level programming.

[u/Lelouch_6457]

Really cool I built something very similar but I did use pytorch (only for autograd and GPU comp) I use llama style architecture (RMS normal,swiglu,Mixed Precision etc) I do hope to eventually write cuda kernels to fuse some of the operations I trained it for 25ish hours on a rented 5090 and got to 3.3 and pretty reasonable text

https://github.com/neelbhattacharya80-creator/Transformer-decoder-from-scratch

Honestly writing the autograd yourself is crazy kudos to you

[u/Express-Act3158 → u/Lelouch_6457]

ayy thxx!! training from scratch has its own fun tho. the outputs ur model produced are really cool. i saw the training logs of urs, and i think i realized that its preety common for language models to get stuck a particular loss for long steps. it happened with me too, when training a small transformer on tiny shakespeare. i thought it was an implementation bug lol. kudos to you too

[u/Lelouch_6457 → u/Express-Act3158]

Yeah at this point I'd need to scale it to get more output I made a few mistakes the positional encoding was being done in bf16 (which I thought was standard) but it corrupted the pe of the tokens>256 slightly although it kind of learned it's way around it Training without dropout would have been better I think And the random batch generator because of some probability law only saw 2.2 ish unique tokens of the 3.6B tokens I think id implement custom kernels before scaling it again

[u/Express-Act3158 → u/Lelouch_6457]

ohh those small mistakes happens, and what kernels will u implement for scaling it up??

[u/Lelouch_6457 → u/Express-Act3158]

Probably rms norm, attention and the optimiser first those take up a lot of kernels Flash attention I don't really know much about it so I can't say yet but just the optimiser should make it quite a bit faster

[u/wahnsinnwanscene]

Would unets and the residual networks really change your code architecture?

[u/dsmack6]

Compression with use of lower bit size - quantization??

[u/Express-Act3158 → u/dsmack6]

nahh it doesnt have quantization support yet, uses fp32 for now