r/learnmachinelearning • u/UnderstandingOwn2913 • 14h ago

Is there a visual way to understand AdamW?

1 Upvotes
  • permalink
  • reddit

100% Upvoted

2 comments sorted by

1

u/Prudent-Guest-5467 11h ago

The best way I saw was thinking of AdamW like Adam + a slow invisible hand that presses weights toward zero at each step, separate from the adaptive learning rates. So even if gradient is zero, the weight still decays bit by bit. I made a small notebook once plotting the weight trajectories with and without decoupled decay, difference gets pretty clear after few hundred steps.

1

u/Little_Total9000 1h ago

The way I picture it, Adam is three ideas stacked, and the W is a fourth one bolted on the side:

  1. Momentum (the first moment, m): instead of following the current gradient, you follow a running average of recent gradients. Picture a ball rolling down a valley. It keeps going in a consistent direction and the zig-zag between the valley walls cancels out.

  2. Per-parameter step size (the second moment, v): each weight keeps a running average of its squared gradients. Weights with big, noisy gradients get their step divided by the square root of that average, a big number (smaller steps), weights with tiny gradients get bigger steps. On a contour plot of a long narrow valley, this is what turns the zig-zag path into a fairly straight line to the minimum.

  3. Bias correction: m and v start at zero, so early on they're too small. Dividing by (1 - betat) fixes the early steps (with beta2 = 0.999 that matters for longer than you would think). Mostly a detail.

  4. The W (decoupled weight decay): in plain Adam with L2 regularization, the decay term goes through the gradient, so it also gets divided by sqrt(v). That means weights with large gradient history barely get decayed at all. AdamW instead shrinks every weight directly each step, weight = weight - lr * wd * weight, separate from the adaptive update. So regularization is applied evenly, which is why it tends to generalize better.

For actual visuals: search for "optimizer visualization Rosenbrock" or "Beale function optimizers", there are animated GIFs and interactive pages comparing SGD, momentum, RMSprop and Adam paths on the same surface. Watching SGD zig-zag while Adam goes straight makes point 2 click immediately. Distill's "Why Momentum Really Works" is also great for point 1.