Lecture 16 Projection Matrices and Least Square
- b in column space: project something already in the column space.
b is perpendicular to the column space: b is projected to be a point, a 0 vector.
N(A^T) is perpendicular to the column space.
Pb = A(A^T A)^-1 A^T b . A^T * N(A^T) = 0
b in column space: Ax = b
Pb = A(A^T A)^-1 A^T * Ax = Ax = b
2. e is another projection of b into the N(A^T)
3. Least Square, Linear Regression. The error is squared difference.
||Ax -b||²=||e||². There are two pictures. The vector picture. And Data picture.
4.Multiply A^T on both side. It is important because it is like taking a square of the A, which is the independent variables in data A. It will show how the data itself correlated with itself.
App: Statistic, Fitting(Learning), Estimation, Error, Noise.
5.
6. Calculus and Linear Algebra.
Calculus: Loss = (C+D-1)² + (C+2D-2)² + (C+3D-2)²
Take the partial derivative with respect to C and D.
Partial Derivative = 0 to get the extrema.
In the end, there will be two equations, should be same as the equations from linear algebra.
Linea Algebra: A^T * A * x = A^T * b
7. Error e: e perpendicular to p, e perpendicular to column space of A
8. A^T*A should be invertible. Prove:
9. Columns are certainly independent if they are unit perpendicular vectors(Orthonormal vectors).
Lecture 17. Orthogonal Matrices and Gram-Schmidt(Column space)
- Imagine a typical 3D coordinate frame. The frame itself is made by the orthonormal matrix. Each basis is perpendicular to each other, and they are independent.
Ortho means perpendicular. Normal means length is equal to 1.
For vectors, is they are same, give 0, they are different position, give 1.
2. A matrix that consists of orthonormal vectors is called Q. The product of Q is identity.
3. Orthogonal Matrx is Orthonormal, and square.
If Q is square, and Q^T * Q = I, means Q^-1 = Q^T
4. Normalize a matrix made of orthogonal basis to be orthonormal.
Adhemar Matrix:
5. Gram-Schmidt will need to divide the matrix by its’ length. It will involve a square root.
6. Sequence does matter when Q is not square. Q is square, P = I, no need for projection. If Q is square, and project b into Q, no need.
P is symmetric. P² = P
7. Normal Equation.
8. Gram-Schmidt. How to convert a vector into orthonormal.
9. Check whether the B is perpendicular to A by multiplied by A.
10. Find one more C that perpendicular to both A and B.
11. An example:
The original column spaces are not good enough, Gram-Schmidt make it better(to be orthonormal).
12. Find the Linear Algebra language for Q. A = QR.
What Gran-Schmidt does is that it find another matrix Q that all its’ columns are perpendicular to the columns in A. The connection is R, which is an upper triangular matrix.
