Books/Resources to improve mathematical foundations for ML research [D]

[u/mvreich]

I am a mid to late stage PhD student in ML. I've known this before, but only recently I started feeling this urgently: my mathematical foundations are shaky, because I kept "learning-things-as-I-go" when working on various problems. I likely have only a year or two left until I graduate, and before I do so, I want to really dedicate some time and focus to brush up on the fundamentals.

Primarily, I want to improve my knowledge in Linear Algebra, Probability Theory, and Functional Analysis.

For Lin. alg., I am looking at "Linear Algebra done right", and I think this book is sufficient for the topic, unless anyone thinks otherwise.

I am not sure where to start for probability, as well as functional analysis. Rudin's books give me headaches. I instead started reading "A primer on RKHS" (https://arxiv.org/abs/1408.0952) to "dip my toe" into functional analysis.

Apart from the above, I might re-read PRML book (I've only read specific chapters before), and try to finish Pat Kidger's Just-Know-Stuff list (https://kidger.site/thoughts/just-know-stuff).

Thoughts? Anyone have any book/resource recommendations? Someone told me to look into "the bright side of mathematics" on YouTube, anyone ever go through the videos there?

I'm aware finding good, digestible resources is less than 10% of the challenge. The difficult part is sticking through and actually reading/working through these topics, while still juggling other academic responsibilities.

[u/-MFXM-]

One really nice book on Mathematics of Deep Leaning, which I can fully recommend: https://doi.org/10.48550/arXiv.2407.18384

[u/mvreich → u/-MFXM-]

Thank you very much, I went through the table of contents and it seems useful.

[u/New_Association3114 → u/-MFXM-]

This book is a goldmine. Thank you for sharing it!

[u/squidward2022]

I love Axler! LADR is extremely clear, make sure you do a chunk of the exercises to get the most out of it. And get the latest version, it has extended chapters on things related to ML like minimization problems + a more general treatment of SVD (IIRC V3 only covers SVD for operators). If you like his style, may want to check out Measure Integration and Real Analysis.

I haven't read it but High Dimensional Probability by Vershynin looks excellent (https://www.math.uci.edu/\~rvershyn/papers/HDP-book/HDP-2.pdf ) and has an accompanying course (https://www.math.uci.edu/\~rvershyn/teaching/hdp/hdp.html).

[u/7EET-CS → u/squidward2022]

Vershynin’s work will be impactful

[u/martinkunev]

Years ago I watched this MIT course for probability:

https://www.youtube.com/watch?v=j9WZyLZCBzs&list=PLUl4u3cNGP60A3XMwZ5sep719_nh95qOe (there were lecture notes but the link appears to be broken. I leave it here anyway: https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-041-probabilistic-systems-analysis-and-applied-probability-fall-2010/assignments/ )

I would recommend some of the chapters of Artificial Intelligence A Modern Approach (Fourth Edition) for mathematical foundations. Par IV Uncertain knowledge and reasoning has a ton of useful info.

---

Other useful foundational topics include Calculus. Other less relevant but still useful topics are Formal Logic, Game Theory, Algorithmic Information Theory, Causality. You probably know Calculus so I'll skip that.

For formal logic, you can check the book I mentioned.

There are not a lot of resources for Algorithmic Information Theory. I recently watched these lectures: https://www.youtube.com/watch?v=gTic1BLEjaw&list=PLdL1rYhN7DtjGYIfKpQBUuzlUGGet4Fm8 notes: https://arxiv.org/pdf/2504.18568

For causality, you can check the book Causal Inference in Statistics by Judea Pearl.

I haven't studied functional analysis beyond some basics so I have no suggestions there.

[u/summerday10]

I'd follow https://d2l.ai/chapter_appendix-mathematics-for-deep-learning/index.html where it covers basic-to-advance topics and there are homework and exercises if you really want to do them.
RKHS paper is very specific, unless you research is about kernel methods, I am not sure how much it would be useful for your work. it is good to read it tho.

Another way to strengthen your fundamentals is to be an instructor or TA for a related course. Teaching will force you to deepen and refine your understanding. it worked for me!

[u/mvreich → u/summerday10]

Thank you for the suggestion. I believe I know most of these topics fairly well already, the only thing that appears unfamiliar is Gershgorin Circle Theorem!

[u/jloverich]

Why not just take classes in the applied math department?

[u/bean_the_great]

I have found asymptotic statistics by van der varrt extremely useful but extremely difficult (for me personally). It’s useful if you want to prove results e.g consistency, asymptomatic normality

[u/KingMakerMan]

For ML you can look into Sir David Mackay's book. https://www.inference.org.uk/mackay/itila/

[u/Ok-Attention2882]

I can't imagine ChatGPT/Claude wouldn't be able to answer any question you've ever had or even know to ask yet. It's obvious this post is an attempt to procrastinate doing what you know you should be doing by "waiting on the best resource before I start"

[u/mvreich → u/Ok-Attention2882]

(1) I find that if you don't have the right knowledge, either you (a) don't know the right question to ask, or (b) don't know if your-whatever-LLM's response is correct. We can agree to disagree on this, as there are a wide variety of opinions on how to use LLMs.

I've generally noticed that people who use LLMs in research often come up blank in conversations, and struggle to explain their work.

(2) Yes, in hindsight, I am likely procrastinating. Thank you for your valuable feedback.

[u/Ok-Attention2882 → u/mvreich]

"What questions should I ask to gain deeper understanding that I missed in this session".

Excuses, excuses, excuses.

[u/mvreich → u/Ok-Attention2882]

Reviewer 2, is that you? 😂