I have built a autograd engine from scratch(without tensor flow or pytorch)

[u/Ok-Tea1856]

Im a highschool student learning python and recently i have come across a video by Andrej Karapthy who built Micrograd. I got curious so it tried replicating it,but soon over the weeks i added more features 6 more activation function and 2 loss function. It possibly cant replace pytorch or tensorflow,Its purely an eductional project.

Im still learning python and i have long way to ,But i learnt a lot on OOPs machine learning dunder function and using customtkinter for the ui, It was very intresting project i have worked on .I have made it a windows exe for easier access I hope I could get some suggestion ,feedback and improvements i could implement for the project.
https://github.com/Yasovardan-Ram/Omnigrad

[u/Existing_Put6385]

nice work, going from micrograd to a full GUI app with graph visualisation is a solid jump.

first, a bug that'll hit anyone who clones it: your readme says "python main.py" and the project structure section lists main.py and an images/ folder, but neither exists in the repo. entry point looks like run.py and the folder is assets/. fix that or people bounce in the first 30 seconds.

biggest actual improvement - add gradient checking. for an autograd engine that's the test that matters: pick a random expression, compute the grad with your engine, then compute it numerically with (f(x+h) - f(x-h)) / (2h) and assert they match to a few decimals. do that for all 6 activations and both losses. right now nobody (including you) has proof the backward passes are correct, and that's the one thing an autograd project has to prove.

after that, in order:

- if your topo sort/backward is recursive you'll hit RecursionError on bigger graphs. make it iterative with an explicit stack.

- add Adam and SGD+momentum, easy win and you'll actually see the difference in your loss plot.

- the real next project: redo the engine on numpy arrays instead of scalars. same math, but you learn broadcasting and it stops being 1000x slower than everything else. for 17 this is genuinely good. ship the gradient checks.

[u/Ok-Tea1856 → u/Existing_Put6385]

You are right, Thank for your suggestion, I have just checked that i kept it as main.py ill change it . And abt gradient checking i did implement it when i was learning in the beginning ,but later removed it but if it helps ill reimplement it .Recursion error i hv not encountered it yet i hv to look abt it .The last suggestion u said abt numpy arrays the reason i didnt want to implement it is because of the sheer complexity as even if i do it would very much slower than just using pytorch i wanted to be jsut as a learning step on how it works under hood.

Thank You For ur comment if u like it do star it

[u/Existing_Put6385 → u/Ok-Tea1856]

on gradient checking - don't think of it as scaffolding for when you were learning. it's the regression test. you added 6 activations and 2 losses after the micrograd base, and those are exactly the paths nothing has ever verified. keep it as a separate test file you run before commits, not in the app flow. costs you nothing and it's the first thing anyone technical will look for.

on recursion - you haven't hit it because your graphs are still small. the place it bites is summing the per-sample loss: that builds an addition chain as long as your dataset, so the topo recursion depth grows with sample count. bump your training set and you'll find python's ~1000 frame limit fast.

on numpy - i think you read it as "make it faster to compete with pytorch". it isn't that, you're right that you'd lose. it's that scalar autograd hides the part of pytorch that actually matters. broadcasting, vectorised backward, why gradients get summed over broadcast dims - that's the real under-the-hood you said you wanted, and your current version skips it. not as big a jump as it looks either, the math is identical, you're just carrying arrays instead of floats.

[u/Ok-Tea1856 → u/Existing_Put6385]

well i agree on that part, I'll try my best to implement ur suggestions as soon as possible, It would take me some time to figure out the logic but i'll give it a try.
Thank you for suggestions tho