Perpframes, Aligners and Hangers
Linear Systems, Pseudo-Inverse
Matrix Norm, Rank One
Good Basis Theorem
Singular Value DecompositionThe singular value decomposition for a matrix A writes A as a product (hanger)(stretcher)(aligner).
It's an amazing and useful fact that every m x n matrix has a singular value decomposition.
The following theorem goes two-thirds of the way to proving this fact:
Two-thirds TheoremFor an matrix and any orthonormal basis of ,
Proof: Using first the row way and then the column way to multiply a matrix times a point, you see that the right hand side of the equation sends to .
Thus the two sides of the equation agree on the basis and so must be equal.
The two-thirds theorem gets you two-thirds of the way to the SVD.
It says that given any orthonormal basis of you can write
So you've got the stretcher and the aligner -- if were a hanger matrix then this would be a Singular Value Decomposition for A.
For to be a hanger matrix requires that the columns be pairwise perpendicular.
So one challenge to finding an SVD for A is to find an orthonormal basis of , so that for all , .
Theorem: If A is an m x n matrix, then there is an orthonormal basis of , so that for all , .
Proof 1: For matricesYou can get 2D perpframes using and specifying an angle .
You'd like to show that for any m x 2 matrix there's an angle t so that
How do you know there is always such a ?
Answer: Go with the example matrix and look at the following plot that shows the perpframe and before and after the hit with A.
By the time , the angle is less than .
That's enough to guarantee that somewhere between and there's an angle where and are perpendicular.
Most reasonable folks would accept the evidence given in the plots, but the doubters might want to look at the function:
(1) The definition of f.
(2) Linearity of matrix hits.
(3) The definition of f.
This equation says that and have opposite signs. This should convince even the doubters that there's a between and that makes zero.
The same argument holds for any m by 2 matrix.
Proof 2: General caseThe proof above works well enough for m x 2 matrices, but this proof shows that for all m x n matrices there's is a perpendicular frame of unit vectors so that for .
Here's one way to get such a perpframe.
Start with a m x n matrix A.
Step 1: Let be a unit vector maximizing .
Step 2: Let
be the space perpendicular to
Step 3: Let
be the space perpendicular to
This gets you a perpendicular frame of unit vectors ,
but how do you know that
and see that
is a unit vector in
for all ,
you know that
has a maximum at .
This tells you .
When you remember , , and are just numbers, you understand that its nothing more than tedious to compute:
gives you .
But you already know that , so after canceling the you get which is just what you wanted.
Proof 3: Based on the spectral theoremThis proof is slick IF YOU'VE ALREADY SEEN THE SPECTRAL THEOREM.
If you haven't seen the spectral theorem, then skip this proof.
Given and an orthonormal basis of ,
iff are all eigenvectors of .
Conclusion: The desired basis is guaranteed by spectral theorem since is symmetric.
Theorem: Every matrix has a singular value decomposition.The theorem above almost gives you the SVD for any matrix.
The only problem is that although the columns of the "hanger" matrix are pairwise perpendicular, they might not form a basis for .
For example, suppose for a 5x4 matrix the procedure outlined above gives you:
To complete the decomposition, let
be an orthonormal basis for the three dimensional subspace of
perpendicular to .
This, finally, is a singular value decomposition for A.
In either case, the dimensions of the stretcher matrix will always match
the dimensions of A.
Exercises1. Above, you saw that if A is a matrix then the SVD for A has the form
Give the form for the SVD of a 4x5 matrix .
(Hint: The matrix has five singular values, but
does not appear in the decomposition.)
2. Let . Following the steps of Proof 1 above, define and .
b. After applying trig identities it turns out that .
Use this fact to find a t so that
c. Using part (b) as a start, give an SVD for A.