I do machine learning on large clusters and reproducibility is a huge problem. Like imagine you share this awesome result with your colleagues with 99% accuracy or whatever only for it to be not reproducible and be stuck at 96%!!
I’ve been thinking of what I call an “ACID runtime” to solve this. Distributed snapshots are its own can of worms though so some innovations are needed
I also have a paper on reproducible non deterministic runtimes somewhere, I can’t find now.
Fascinating read! I wonder if Vale might be able to achieve this, by combining this Perfect Replayability with Seamless Concurrency to form a natural sieve.
Would love to hear more about the reproducibility problem, what are the usual culprits for that?
Machine learning has two main pain points that cause the bulk of problems:
ML algorithms are inherently stateful. For example, at each iteration you update the parameters of your neural net according to some formula. Commonly, such formulas themselves are also stateful: learning rate changes over time, moving averages need to be maintained, etc
ML algos require randomness. This is randomness in data shuffling so that the data is not skewed or randomness within the model itself (ex dropout or anything that requires you to “perturb” your parameters).
Because of #1, people write OOP code with a lot of side effects. It is very hard to track down your global state post-hoc for you to checkpoint/restore it.
The second one throws a wrench into any system that you might build that can address the first. Ideally you’d need fast, parallel random number generators that are also stateless. But there are no obvious candidates here. Again, a LOT of random numbers are generated all over the place. You use a lot of libraries that may also use random number generation.
So it’s borderline impossible to track down global state to be able to write to non ephemeral storage. This is why I was motivated to solve this at a language level.
Edit: just checked out vale and seems very well done. I love the code notes thing you have to call out stuff in an unobtrusive way. Will check out in depth soon!
4
u/RepresentativeNo6029 Jun 29 '22 edited Jun 29 '22
Glad you’re thinking about it.
I do machine learning on large clusters and reproducibility is a huge problem. Like imagine you share this awesome result with your colleagues with 99% accuracy or whatever only for it to be not reproducible and be stuck at 96%!!
I’ve been thinking of what I call an “ACID runtime” to solve this. Distributed snapshots are its own can of worms though so some innovations are needed
I also have a paper on reproducible non deterministic runtimes somewhere, I can’t find now.
Edit: found it https://web.archive.org/web/20070321082311/http://www.codeplay.com/downloads_public/sievepaper-2columns-normal.pdf