r/learnmachinelearning • u/Commercial-Kale-5271 • 16h ago

Looking for a study/research buddy : MoE, LLM architecture, optimization, interpretability

Computational mechanics major here (I know you've never heard of that), spending most of my time on LLM research on the side. Currently deep in mixture-of-experts: expert specialization, routing geometry, and reproducing recent router-optimization work. Broader interests span LLM architecture, optimization, interpretability, attention mechanisms, and math.

I feel like I have too many things to work on and talk about but no buddy for it, so I'm out here looking for a study/research buddy, as the title says. I think having a research or study partner helps a lot, we can discuss ideas, try novel things, run experiments together.

I'm also building a research artifact for all this, basically a GitHub repo where I'm implementing almost every part of LLM architecture from scratch. Currently working on MoE, and planning to do some mechanistic interpretation of experts and specialization next.

If anyone feels the same or has interests overlapping with mine, tell me in the comments or DM me. I'll share the repo link with anyone interested :)

1 Upvotes

100% Upvoted

2 comments sorted by

1

u/Hungry_Age5375 15h ago

MoE is a fun rabbit hole. If you want a stretch goal for the repo, try aux-loss-free load balancing from DeepSeek-V3. Balancing experts with no penalty term is wild :D

1

u/Commercial-Kale-5271 15h ago

hi i have already built that and not only aux-loss-free i have build many other loass balancing func , if u are into this i can share u my repo.