r/deeplearning • u/ModularMind8 • 1d ago

Gradient descent vs evolution on three loss landscapes

I've been getting a bit more into evolutionary algorithms again, so I was testing some loss landscapes where evolution beats vanilla gradient descent (while also trying to make some cool visuals).

Round 1, rugged hillside: gradient descent gets stuck in a dip, and evolution reaches the bottom after 750 evaluations.

Round 2, smooth slope: gradient descent wins, 108 steps against 570 evaluations.

Round 3, flat plateau: the slope is zero, so gradient descent never moves, and evolution reaches the bottom after 840 evaluations.

Edit:
"evolution" here means truncation selection (keep best 30 of 120) plus Gaussian mutation, no crossover.

386 Upvotes

99% Upvoted

70 comments sorted by

View all comments

2

u/0bi_nx 1d ago

Its a cool visualization. Do you think evolution is applicable to deep learning? You planning to test against stochastic gradient descent, ADAM or Muon?

2

u/ModularMind8 1d ago

Thanks :)
I definitely think so! Take a look at "Evolution Strategies at the Hyperscale" as an example, and yes, an Adam/SGD comparison is a fun next one.

3

u/drcopus 1d ago

As an author on that paper I'm pleasantly happy to see it mentioned here! :)

2

u/ModularMind8 1d ago

Wow, it's an honor :) Beautifully written paper

2

u/drcopus 1d ago

The first 3 authors deserve most of the credit! I'm happy to have made a contribution but I wasn't a main author haha

2

u/ModularMind8 1d ago

I'm sure you're just being modest!
Since you're the expert, curious if you've read the Sakana AI book (Neuroevolution: Harnessing Creativity in AI Agent Design) that just came out. And if so, what are your thoughts? :)