Summary:

In this section, we examine this question: if a system of linear equations Ax = b does not have a solution, can we find an approximate solution? The answer, of course, is yes: we define a least squares solution, show it exists (because of the Best Approximation Theorem), and then show it can be easily computed by solving the associated normal system of equations A^TAx = A^Tb.

We then give an application by finding the line of best fit for a set of data. This is the so called linear regression of a set of data; it is obtained by finding a least squares solution of Mx = y, where y is the observed data, and M is a matrix with two columns: a column of all ones, and a column coming from the points at which the data was observed. Thus the line of best fit is y = a + bx, where the vector $LaTeX: \left[\begin{array}{c} a \\ b \end{array} \right]$ is a solution to LaTeX: M^TMx = M^T y

Outline:

The general problem: if a system does not have a solution, find the best approximation to a solution
The best approximation is the orthogonal projection onto the column space, which is hard to compute in general
So we develop the associated system of normal equations, which always has a solution and the solution is the best approximation that we desire, the so-called least squares approximation
We use this to solve the linear regression problem : find the line of best fit for given data

Video lecture:

Part 1

Part 2

Notes from the lecture: