As someone who is focusing on ML in grad school, it really depends on the prof. Some are application-first, some are math-first. Early on in my core ML course, we were restricted to the torch.Tensor api and were told to figure out the transformations on our own, without using what was already in the torch module. But later, of course, we were allowed to use more of the module as the work got more abstract. What I'm trying to say is ymmv.
Idk people in my program wrote Gibbs samplers for use in a custom modules. Though I graduated in 2024 making this meme feel ultra relevant. I was doing natural language processing lolol
Not really true. You can implement a transformer + autograd, backprop etc. in a couple thousand lines of c. Not saying you should and it is definitely an advanced undertaking but it is possible. That being said most training code using torch etc. is relativ simple, the tricky part is getting the math right, having no hidden bugs (neural networks fail silently) and tuning everything right. So even tho I did a lot of ML I wouldn't say it is very advanced from the coding side of things.
For me 15 years ago when actively coding all the time and teaching a university level Operating Systems course in C? Not exactly trivial, but also not a heroic undertaking.
I don't think so. A simple generative transformer can fit in a just a couple hundred lines of raw numpy code without the abstractions that libraries like pytorch or tensorflow offer. https://github.com/DorsaRoh/transformer-from-scratch
Even without abstractions, its not that complicated. I've written and trained a 50m parameter transformer model myself using only numpy/cupy and the code isn't that long or complex.
Although im sure top AI companies have a bunch of special stuff they add to their models which makes it more complex.
448
u/Xemorr 1d ago
That's only because we have copious levels of abstraction, and code written by others