The Rayleigh Quotient
Nuno Vasconcelos ECE Department, p , UCSD
Principal component analysis basic idea: • if the data lives in a subspace, it is going to look very flat when viewed from the full space, e.g. 1D subspace in 2D
2D subspace in 3D
• this means that if we fit a Gaussian to the data the equiprobability contours are going to be highly skewed ellipsoids
2
The role of the mean note that the mean of the entire data is a function of the coordinate system y • if X has mean µ then X – µ has mean 0
we can always make the data have zero mean by centering • if
and
|⎤ ⎡| X = ⎢⎢ x1 K xn ⎥⎥ ⎢⎣ | | ⎥⎦
X
T c
1 T ⎛ = ⎜ I − 11 n ⎝
⎞ T ⎟X ⎠
• then Xc has zero mean
can assume that X is zero mean without loss of generality 3
Principal component analysis If y is Gaussian with covariance Σ the equiprobability q p y contours
y T Σ −1 y = K
y2
φ2 λ2
φ1 λ1 y1
are the ellipses whose • principal components φi are the eigenvectors of Σ • principal lengths λi are the eigenvalues of Σ
by detecting small eigenvalues we can eliminate dimensions that have little variance thi iis PCA this 4
PCA by SVD computation of PCA by SVD given X with one example per column • 1) create the centered data-matrix
X
T c
1 T ⎛ = ⎜ I − 11 n ⎝
⎞ T ⎟X ⎠
• 2) compute t its it SVD
X cT = ΜΠΝT • 3) principal components are columns of N, eigenvalues are
λi = π i2 / n 5
Principal component analysis a natural measure is to pick the eigenvectors that explain p % of the data variability y • can be done by plotting the ratio rk as a function of k k
rk =
∑λ i =1
2 i n
2 λ ∑ i i =1
• e.g. we need 3 eigenvectors to cover 70% of the variability of this dataset
6
Limitations of PCA PCA is not optimal for classification • note that there is no mention of the class label in the definition of PCA • keeping the dimensions of largest energy (variance) is a good idea but not always enough idea, • certainly improves the density estimation, since space has smaller dimension • but could be unwise from a classification point of view • the discriminant dimensions could be thrown out
it is not hard to construct examples where PCA is the worst possible thing we could do
7
Fischer’s linear discriminant find the line z = wTx that best separates the two classes
bad projection
good projection
( E [Z | Y = 1] − E [Z | Y = 0]) w* = max
2
Z |Y
w
Z |Y
var[Z | Y = 1] + var[Z | Y = 0]
8
Linear discriminant analysis this can be written as between class scatter
wT S B w J ( w) = T w SW w optimal solution is
S B = (µ1 − µ 0 )(µ1 − µ 0 )
T
SW = (Σ1 + Σ 0 )
within class scatter
w* = SW−1 (µ1 − µ0 ) = (Σ1 + Σ 0 ) (µ1 − µ 0 ) −1
BDR after projection on z is equivalent to BDR on x if • the two classes are Gaussian and have equal covariance
otherwise, LDA leads to a sub-optimal classifier 9
The Rayleigh quotient it turns out that the maximization of the Rayleigh quotient
wT S B w J ( w) = T w SW w
S B , SW , symmetric i positive semidefinite
appears in many problems in engineering and pattern recognition we have already seen that this is equivalent to
max wT S B w subject j to w
wT SW w = K
and can be solved using Lagrange multipliers
10
The Rayleigh quotient define the Lagrangian
(
L = wT S B w - λ wT SW w − K
)
maximize with respect to w
∇ w L = 2(S B - λSW )w = 0 to obtain the solution
S B w = λSW w this is a generalized eigenvalue problem that you can solve using any eigenvalue routine which eigenvalue? 11
The Rayleigh quotient recall that we want
max wT S B w subject to w
wT SW w = K
and the optimal w satisfies
S B w = λSW w hence
(w *)T S B w* = λ (w *)T SW w* = λK
which is maximum for the largest eigenvalue in summary, we need the generalized eigenvector eigen al e S B w = λSW w of largest eigenvalue 12
The Rayleigh quotient case 1: Sw invertible • simplifies to a standard eigenvalue problem
SW−1S B w = λw • w is the largest eigenvalue of Sw-1SB
case 2: Sw not invertible • this is case is more problematic • in fact the cost can be unbounded • consider w = wr + wn, wr in the row space of Sw and wn in the null space
wT SW w = ( wr + wn )T SW ( wr + wn ) = ( wr + wn )T SW wr = wrT SW ( wr + wn ) = wrT SW wr 13
The Rayleigh quotient and
wT S B w = ( wr + wn )T S B ( wr + wn ) = wrT S B wr + 2wrT S B wn + wnT S B wn 1 424 3 ≥0
hence, if there is a (wr,wn) such that wrTSBwn ≥ 0, • we can make the cost arbitrarily large • by simply scaling up the null space component wn
this can also be seen geometrically
14
The Rayleigh quotient recall that
wT SW w = K with SW = ΦΛΦ T are the th ellipses lli whose h • principal components φi are the eigenvectors of Sw
w2
φ2 1/λ2
φ1 1/λ1 w1
• principal lengths are 1/λi
when the eigenvalues go to zero, the ellipses blow up consider the picture of the optimization problem
max wT S B w subject to w
wT SW w = K 15
The Rayleigh quotient max wT S B w subject to w
wT SW w = K w*
w2
wTSBw 1/λ2
1/λ1 w1
wTSww w=K K
the optimal solution is where the outer red ellipse (cost) touches the blue ellipse (constraint) • in this example, as λ1 goes to 0, ||w*|| and the cost go to infinity 16
The Rayleigh quotient how do we avoid this problem? • we introduce another constraint
max wT S B w subject to w
wTw=L
wT SW w = K ,
w =L
w*
w2
wTSBw 1/λ2
1/λ1 w1
wTSww=K • restricts the set of possible solutions to these points (surfaces in high dimensional case) 17
The Rayleigh quotient the Lagrangian is now
(
) (
L = wT S B w - λ wT SW w − K - β wT w − L
)
and the solution satisfies
∇ w L = 2(S B - λSW − βI )w = 0 or
(S B - λ [SW + γI ])w
= 0, γ = β / λ
but this is exactly the solution of the original problem with Sw+γI instead of Sw
max wT S B w subject bj t to t w
wT [SW + γI ]w = K 18
The Rayleigh quotient adding the constraint is equivalent to maximizing the regularized g Rayleigh y g q quotient
wT S B w J ( w) = T w [SW + γI ]w
S B , SW , symmetric positive iti semidefini id fi ite t
what does this accomplish? • note that
SW = ΦΛΦ T ⇒ SW + γI = ΦΛΦ T + γΦIΦ T = Φ[Λ + γI ]Φ T
• this makes all eigenvalues positive • the matrix is no longer non-invertible 19
The Rayleigh quotient in summary
wT S B w max T w w SW w
S B , SW , symmetric positive semidefinite
1) Sw invertible • w* is the eigenvector of largest eigenvalue of Sw-1SB • the max value is λK, where λ is the largest eigenvalue
2) Sw not invertible • regularize: l i Sw -> Sw + γI • w* is the eigenvector of largest eigenvalue of [Sw + γI]-1SB • the max value is λK, where λ is the largest eigenvalue
20
Regularized discriminant analysis back to LDA: • when the within scatter matrix is non-invertible non-invertible, instead of
between class scatter
wT S B w J ( w) = T w SW w • we use
wT S B w J ( w) = T w SW w
S B = (µ1 − µ 0 )(µ1 − µ 0 )
T
SW = (Σ1 + Σ 0 )
within class scatter
S B = (µ1 − µ 0 )(µ1 − µ 0 )
T
SW = (Σ1 + Σ 0 + γI )
regularized within class scatter 21
Regularized discriminant analysis this is called regularized discriminant analysis (RDA) noting that
SW = Σ1 + Σ 0 + γI = Σ1 + γ 1 I + Σ 0 + γ 0 I
γ1 + γ 0 = γ
this can also be seen as regularizing each covariance matrix individually the regularization parameters γi are determined by crossvalidation • more on this later • basically means that we try several possibilities and keep the best 22
Principal component analysis back to PCA: given X with one example per column • 1) create the centered data-matrix
X
T c
1 T⎞ T ⎛ = ⎜ I − 11 ⎟ X n ⎝ ⎠
this has one point per row
⎡ x1T − µ T ⎤ ⎢ ⎥ T Xc = ⎢ M ⎥ ⎢ xnT − µ T ⎥ ⎣ ⎦
• note that the p projection j of all p points on p principal p component p φ is
z = Xc φ T
(
⎡ z1 ⎤ ⎡ x1 − µ ⎢M⎥=⎢ M ⎢ ⎥ ⎢ ⎢⎣ zn ⎥⎦ ⎢⎢ xn − µ ⎣
(
) φ ⎤⎥ T
⎥ T ⎥ φ⎥ ⎦
)
23
Principal component analysis and, since
(
1 1 zi = ∑ xi − µ ∑ n i n i
)
T
T
⎛1 ⎞ φ = ⎜ ∑ xi − µ ⎟ φ = 0 ⎠ ⎝n i
the sample variance of Z is given by its norm
1 2 var( z ) = ∑ zi = z n i
2
recall that PCA looks for the largest variance component
max z = max X φ 2
φ
φ
T c
2
(
)
= max X φ X cT φ = max φ T X c X cT φ φ
T c
T
φ
24
Principal component analysis recall that the sample covariance is
( )
1 1 T Σ = ∑ ( xi − µ )( xi − µ ) = ∑ xic xic n i n i
T
where xic is the ith column of Xc this can be written as c ⎡ | | − x ⎡ ⎤ 1 1⎢ c c ⎥⎢ Σ = ⎢ x1 K xn ⎥ ⎢ M n ⎢⎣ | | ⎦⎥ ⎣⎢− xnc
−⎤ ⎥ 1 T = X X ⎥ n c c −⎥⎦ 25
Principal component analysis hence the PCA problem is
ma φ T X c X cT φ = max max ma φ T Σφ φ
φ
as in LDA, this can be made arbitrarily large by simply scaling li φ to normalize we constrain φ to have unit norm
max φ T Σφ φ
subject bj t to t
φ =1
which is equivalent to
φ T Σφ max T φ φ φ shows that PCA = maximization of a Rayleigh quotient 26
Principal component analysis in this case
wT S B w max T w w SW w
S B , SW , symmetric i positive semidefinite
with SB = Σ and Sw = I Sw is clearly invertible • no regularization problems • w* is the eigenvector of largest eigenvalue of Sw-1SB • this thi is i just j t the th largest l t eigenvector i t off the th covariance i Σ • the max value is λ, where λ is the largest eigenvalue
27
The Rayleigh quotient dual let’s assume, for a moment, that the solution is of the form
w = X cα
• i.e. a linear combination of the centered datapoints
hence the problem is equivalent to
α T X cT S B X c α max T T α α X S X α c W c thi does this d nott change h its it fform, the th solution l ti iis •
α* is the eigenvector of largest eigenvalue of (XcTSwXc)-1XcTSBXc
• the max value is λK, K where λ is the largest eigenvalue
28
The Rayleigh quotient dual for PCA • Sw = I and SB = Σ = 1/n XcXcT • the solution satisfies
S B w = λS W w ⇔ −1
1
n
T c
X c X w = λw ⇔ w = X c
1
X cT w
n4 λ 24 1 3 α
• and, therefore, we have • w* eigenvalue of SB = XcXcT • α** eigenvalue i l off (XcTSwXc)-11XcTSBXc = (XcTXc)-11XcTXcXcTXc =X XcTXc
• i.e. we have two alternative manners in which to compute PCA
29
Principal component analysis primal
dual
• assemble matrix
• assemble matrix
•
•
Σ = XcXcT
Κ = XcTXc
• compute eigenvectors φi
• compute eigenvectors αi
• these are the principal components
• the principal components are •
φi = Xcαi
in both cases we have an eigenvalue problem • primal i l on th the sum off outer t products d t
i
• dual on the matrix of inner products
( )
Σ = ∑ x ic x ic
K ij = (x
)x
c T i
T
c j
30
The Rayleigh quotient this is a property that holds for many Rayleigh quotient p problems • the primal solution is a linear combination of datapoints • the dual solution only depends on dot-products of the datapoints
whenever both of these hold • the th problem bl can b be kkernelized li d • this has various interesting properties • we will talk about them
many examples • kernel PCA, kernel LDA, manifold learning, etc. 31
32
