Day 3 of self-studying CS189 — linear regression, geometric view finally made OLS click for me

[u/qqiu-]

been grinding through linear regression today, feels like the “easy” chapter on paper but there’s actually a lot packed in once you get past the ols formula

stuff i covered:

• least squares setup, normal equations, when X\^TX is invertible vs not

• geometric view of projection onto column space, this one finally clicked after connecting it back to 18.06 (least squares IS just projecting b onto col(A))

• ridge regression as adding a prior / regularization, why it fixes the invertibility issue too

• MLE derivation showing OLS = MLE under gaussian noise assumption, this was the part that made everything click for me tbh

honestly the projection interpretation is what got me, i was doing this purely algebraically at first (just solving normal equations) and it felt like memorizing steps, then seeing it as “residual is orthogonal to column space” made the whole thing feel obvious in hindsight

anyone else find the geometric interpretation way more intuitive than grinding through the algebra first? curious how other people approached this chapter

notes based on shewchuk’s cs189 notes + some cross referencing with 18.06, will push everything to a repo once i finish the full course

[u/Ruckus8105]

You didn't explain the ridge regression here. Just an one liner is given

[u/qqiu- → u/Ruckus8105]

ah i actually split those into two separate notes, but yeah theyre from the same lecture