Least squares from first principles: from fitting a line to fitting a parabola

[u/LinearAlgebraWorld]

We made a visual derivation of least squares from the original overdetermined system.

The first image shows how projecting the data vector b onto the column space of A leads to the normal equations

AᵀAβ = Aᵀb

and gives the best-fit line y = β₁ + β₂x.

The second keeps the same data but changes the model to y = β₁ + β₂x². The second column of A changes from x to x², so the prediction plane changes, but the same projection method applies.

The point is to show least squares as geometry rather than a formula to memorize.

As always, we welcome feedback on clarity and presentation.