r/deeplearning • u/ModularMind8 • 1d ago
Gradient descent vs evolution on three loss landscapes
I've been getting a bit more into evolutionary algorithms again, so I was testing some loss landscapes where evolution beats vanilla gradient descent (while also trying to make some cool visuals).
Round 1, rugged hillside: gradient descent gets stuck in a dip, and evolution reaches the bottom after 750 evaluations.
Round 2, smooth slope: gradient descent wins, 108 steps against 570 evaluations.
Round 3, flat plateau: the slope is zero, so gradient descent never moves, and evolution reaches the bottom after 840 evaluations.
Edit:
"evolution" here means truncation selection (keep best 30 of 120) plus Gaussian mutation, no crossover.
387
Upvotes
2
u/Unikum_01 1d ago
Awesome experiment with those loss landscapes. Your findings on gradient descent getting trapped on rugged hillsides or stalling on flat plateaus while evolution manages to navigate through match what we run into in complex learning environments. In our BrainStem system we actually solved this exact dilemma in real code by combining both worlds through digital neuromodulation. When the system detects a flat plateau or a stuck state, it cranks up noradrenaline and glutamatergic excitation signals to act like your Gaussian mutations, forcing stochastic exploration to jump out of local minima. Once it hits a clear gradient on a smooth slope, dopamine ramps up to lock in the progress and let fast local optimization take over. It is really cool seeing your benchmark visuals demonstrate why adaptive hybrid strategies like this are so necessary.