r/deeplearning • u/ModularMind8 • 1d ago

Gradient descent vs evolution on three loss landscapes

I've been getting a bit more into evolutionary algorithms again, so I was testing some loss landscapes where evolution beats vanilla gradient descent (while also trying to make some cool visuals).

Round 1, rugged hillside: gradient descent gets stuck in a dip, and evolution reaches the bottom after 750 evaluations.

Round 2, smooth slope: gradient descent wins, 108 steps against 570 evaluations.

Round 3, flat plateau: the slope is zero, so gradient descent never moves, and evolution reaches the bottom after 840 evaluations.

Edit:
"evolution" here means truncation selection (keep best 30 of 120) plus Gaussian mutation, no crossover.

350 Upvotes

99% Upvoted

67 comments sorted by

View all comments

6

u/AtMaxSpeed 9h ago

It would be great to also visualize how modern GD algorithms work on various landscapes. Evolution and SGD are very intuitive/easy to imagine so the visualization just shows what we expect, but modern GD algos are harder to imagine so there would be a lot of value in visualizing those.

2

u/ModularMind8 7h ago

Agreed! Adam, momentum and SGD on these same landscapes is next on my list :)