Bryan (2011) Presentación The Eigenvector of 25 Billions Dollars

Outline
Web Page Importance: Naive Approach

An Improvement: Eigenvectors
Another Improvement
Computing the Importance Vector
The $25,000,000,000 Eigenvector

The Linear Algebra Behind Google
Kurt Bryan
May 19, 2011
Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Outline
Another Improvement
Searching the Web
A search engine like Google has to:

Crawl the web and collect information from web pages.

Outline
Another Improvement
Searching the Web

Store this information in a suitable format.

Outline
Another Improvement
Searching the Web

When queried, retrieve the relevant links and present them in
some sensible order.

Outline
Another Improvement
Searching the Web

When queried, retrieve the relevant links and present them in
some sensible order.
An essential factor in this ordering is the importance of each web
page. How should “importance” be computed?

Outline
Another Improvement
A Simple Idea
Let xk = importance of page k.
Idea: xk = # of backlinks for

page k. Then

Outline
Another Improvement
A Simple Idea

page k. Then
x1 = 2

Outline
Another Improvement
A Simple Idea

page k. Then
x1 = 2
x2 = 1 (least important)

Outline
Another Improvement
A Simple Idea

page k. Then
x1 = 2
x3 = 3 (most important)

Outline
Another Improvement
A Simple Idea

page k. Then
x1 = 2
x3 = 3 (most important)
x4 = 2

Outline
Another Improvement
A Shortcoming
Backlink counting yields

x1 = 2, x2 = 1, x3 = 3, x4 = 2.
But shouldn’t x1 > x4 ?
Shouldn’t a link from Yahoo

count more than a link from
www.kurtbryan.com?

Outline
Another Improvement
A Shortcoming
Backlink counting yields

x1 = 2, x2 = 1, x3 = 3, x4 = 2.
But shouldn’t x1 > x4 ?
Shouldn’t a link from Yahoo

count more than a link from
www.kurtbryan.com?
Better Idea: Links from

important pages should
count more than links from
less important pages.

Outline
Another Improvement
An Improvement
If page j links to page k then

page j casts a vote for page k’s
importance, in amount xj /nj ,
where nj is the number of links
out of page j.
The importance score for any

page is the sum of its votes from
its backlinks.
xk = · · · + xj /nj + · · ·

Outline
Another Improvement
An Improvement

page is the sum of the votes from
its backlinks:

Outline
Another Improvement
An Improvement

its backlinks:
x1 = x3 /1 + x4 /2

Outline
Another Improvement
An Improvement

its backlinks:
x1 = x3 /1 + x4 /2
x2 = x1 /3

Outline
Another Improvement
An Improvement

its backlinks:
x1 = x3 /1 + x4 /2
x2 = x1 /3
x3 = x1 /3 + x2 /2 + x4 /2

Outline
Another Improvement
An Improvement

its backlinks:
x1 = x3 /1 + x4 /2
x2 = x1 /3
x3 = x1 /3 + x2 /2 + x4 /2
x4 = x1 /3 + x2 /2

Outline
Another Improvement
The Matrix Form
If x = [x1 x2 x3 x4 ]T then
x = Ax
where A is the link matrix

 
0 0 1 1/2
 1/3 0 0 0 
A=  1/3

1/2 0 1/2 
1/3 1/2 0 0
So x is an eigenvector for A with

eigenvalue 1.

Outline
Another Improvement
Eigenvectors
Recall that if A is an n × n matrix then a NONZERO vector x is

an eigenvector for A if
Ax = λx
for some scalar λ (the eigenvalue for x).
“In general”,

Outline
Another Improvement
Eigenvectors

Ax = λx
“In general”,
An n × n matrix has n distinct eigenvalues λ1 , . . . , λn , (some
may be complex), and

Outline
Another Improvement
Eigenvectors

Ax = λx
“In general”,
n corresponding linearly independent eigenvectors x1 , . . . , xn

Outline
Another Improvement
Eigenvectors

Ax = λx
“In general”,
n corresponding linearly independent eigenvectors x1 , . . . , xn
Any nonzero multiple of an eigenvector is again an
eigenvector.

Outline
Another Improvement
The Importance Eigenvector
Such an eigenvector exists here:
x ≈ [0.387, 0.129, 0.290, 0.194]T
(or any multiple) satisfies

x = Ax.
Such an eigenvector must

ALWAYS exist if A is
column-stochastic (columns of A
sum to one) and there are no
dangling nodes.

Outline
Another Improvement
Desirable Properties of the Eigenvector

Outline
Another Improvement
Should be non-negative.

Outline
Another Improvement
Should be unique up to scaling (one-dimensional eigenspace).

Outline
Another Improvement
Should be unique up to scaling (one-dimensional eigenspace).
Should be “easy” to compute for an eight billion by eight
billion matrix.

Outline
Another Improvement
Problems
The link matrix here is
 
0 1 0 0 0
 1 0 0 0 0 
 
A= 0 0

.
0 1 1/2 
 0 0 1 0 1/2 
0 0 0 0 0
Two-dimensional eigenspace,
basis x1 = [1/2, 1/2, 0, 0, 0]T ,
x2 = [0, 0, 1/2, 1/2, 0]T .
Any linear combination

c1 x1 + c2 x2 is an eigenvector too!

Outline
Another Improvement
Web With Multiple Components
Which eigenvector should we

use?
Notation: V1 (A) is the subspace

of eigenvectors for A with
eigenvalue 1.
Theorem: If a web has r

components (considered as a
graph) then V1 (A) has at least
dimension r .

Outline
Another Improvement
A Solution for Multiple Components
Give every page a “weak link” to every other page!

Outline
Another Improvement

Replace n by n matrix A with weighted combination
M = (1 − m)A + mS
where 0 ≤ m ≤ 1 and S is n by n,
 
1/n 1/n · · · 1/n
 .. .. .. .. 
S= . . . . 
1/n 1/n · · · 1/n

Outline
Another Improvement

Replace n by n matrix A with weighted combination
M = (1 − m)A + mS
where 0 ≤ m ≤ 1 and S is n by n,
 
1/n 1/n · · · 1/n
 .. .. .. .. 
S= . . . . 
1/n 1/n · · · 1/n
Then M is column-stochastic. Original problem is m = 0,

while m = 1 is egalitarian web—all pages equal importance.

Outline
Another Improvement
Desirable Properties of M
M is the matrix we want. If m > 0 one can show that

Outline
Another Improvement

The dominant eigenvalue for M is 1, and V1 (M) is
one-dimensional, so

Outline
Another Improvement

one-dimensional, so
P
There is a unique non-negative eigenvector x with i xi = 1.

Outline
Another Improvement

one-dimensional, so
P
We can use this x for unambiguous importance ratings.

Outline
Another Improvement

one-dimensional, so
P
Google (reputedly) uses m = 0.15.

Outline
Another Improvement

one-dimensional, so
P
Google (reputedly) uses m = 0.15.
All we need is a method for finding an eigenvector for an eight
billion by eight billion matrix!

Outline
Another Improvement
Multiple Component Example

With m = 0.15 the link matrix is
 
0.03 0.88 0.03 0.03 0.03
 0.88 0.03 0.03 0.03 0.03 
 
 0.03 0.03 0.03 0.88 0.46 
 
 0.03 0.03 0.88 0.03 0.46 
0.03 0.03 0.03 0.03 0.03
The importance eigenvector is

x=
[0.2, 0, 2, 0.285, 0.285, 0.03]T .
Interesting observation: As
m → 0+ ,
x → [0.2, 0, 2, 0.3, 0.3, 0.0]T .
Outline
Another Improvement
Computing the Eigenvector
The Power Method for find the dominant eigenvector of a matrix

M:

Outline
Another Improvement

M:
1 Take a non-negative guess x0 at the eigenvector, scaled so
P (j)
j x0 = 1. Set counter k = 0.

Outline
Another Improvement

M:
P (j)
2 Let xk+1 = Mxk .

Outline
Another Improvement

M:
P (j)
2 Let xk+1 = Mxk .
3 If xk+1 − xk is small, terminate with estimated dominant
eigenvector xk+1 . Otherwise, increment k and return to step
2.

Outline
Another Improvement

M:
P (j)
2 Let xk+1 = Mxk .
3 If xk+1 − xk is small, terminate with estimated dominant
eigenvector xk+1 . Otherwise, increment k and return to step
2.
In short, Mk x0 will converge to the eigenvector we want, for “any”
initial guess x0 .

Outline
Another Improvement
Power Method Example
With the M for the five page network above we find

     
0.1 0.196 0.199
 0.1   0.196   0.199 
20 40
     
x0 =  0.2  , M x0 =  0.289  , M x0 = 
   
 0.286 ,

 0.2   0.289   0.286 
0.4 0.03 0.03

Outline
Another Improvement
Convergence and Implementation

Outline
Another Improvement
In general the power method converges geometrically, as

k
|λ2 |
|λ1 | , where λ2 is the second largest eigenvalue.

Outline
Another Improvement

k
|λ2 |
For M we have λ1 = 1 and it can be shown that λ2 = 1 − m.
If m = 0.15 the method converges in proportion to (0.85)k .

Outline
Another Improvement

k
|λ2 |
For M we have λ1 = 1 and it can be shown that λ2 = 1 − m.
If m = 0.15 the method converges in proportion to (0.85)k .
But M is dense (mostly non-zero). Computing the
matrix-vector product Mv requires about 1.3 × 1020
operations for a web with eight billion pages!

Outline
Another Improvement
Mv can be computed efficiently: M = (1 − m)A + mS, where

S has all entries 1/n and A is sparse (mostly zeros), since
most web pages link to only a few other pages.

Outline
Another Improvement

Note Sv is a vector with all entries n1 j vj . In our case
P
Sv = [1/n, 1/n, . . . , 1/n]T .

Outline
Another Improvement

P
Sv = [1/n, 1/n, . . . , 1/n]T .

If each page has, on average, 10 outgoing links, A has only 10
non-zero entries per column. Av can be computed in about
1.6 × 1011 operations.

Outline
Another Improvement

P
Sv = [1/n, 1/n, . . . , 1/n]T .

If each page has, on average, 10 outgoing links, A has only 10
non-zero entries per column. Av can be computed in about
1.6 × 1011 operations.
Then
Mv = (1−m)Av +mSv = (1−m)Av +[m/n, m/n, . . . , m/n]T
can be computed in a reasonable time.

Outline
Another Improvement
Conclusions

Outline
Another Improvement
Conclusions
The “Google” approach to importance ranking has found

other applications, e.g., “food webs,” game theory, medicine.

Outline
Another Improvement
Conclusions

Sophisticated linear algebra is at the core of much scientific
computation, and pops up when you least expect it.

Outline
Another Improvement
Conclusions

Sophisticated linear algebra is at the core of much scientific
computation, and pops up when you least expect it.
If you pay attention to your math professors, you might
become a billionaire.

Outline
Another Improvement
Resources
1 The $25,000,000,000 Eigenvector: The Linear Algebra Behind

Google, SIAM Review (Education Section), Vol 48 (3),
August 2006.
2 www.rose-hulman.edu/~bryan/google.html

Bryan (2011) Presentación The Eigenvector of 25 Billions Dollars

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Bryan (2011) Presentación The Eigenvector of 25 Billions Dollars

Uploaded by

Copyright:

Available Formats

Outline

Web Page Importance: Naive Approach

The $25,000,000,000 Eigenvector

May 19, 2011

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Searching the Web

A search engine like Google has to:

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Searching the Web

A search engine like Google has to:

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Searching the Web

A search engine like Google has to:

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Searching the Web

A search engine like Google has to:

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Let xk = importance of page k.

Idea: xk = # of backlinks for

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Let xk = importance of page k.

Idea: xk = # of backlinks for

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Let xk = importance of page k.

Idea: xk = # of backlinks for

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Let xk = importance of page k.

Idea: xk = # of backlinks for

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Let xk = importance of page k.

Idea: xk = # of backlinks for

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Backlink counting yields

But shouldn’t x1 > x4 ?

Shouldn’t a link from Yahoo

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Backlink counting yields

But shouldn’t x1 > x4 ?

Shouldn’t a link from Yahoo

Better Idea: Links from

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

If page j links to page k then

The importance score for any

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

The importance score for any

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

The importance score for any

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

The importance score for any

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

The importance score for any

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

The importance score for any

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

The Matrix Form

where A is the link matrix

So x is an eigenvector for A with

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Recall that if A is an n × n matrix then a NONZERO vector x is

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo

Recall that if A is an n × n matrix then a NONZERO vector x is

Kurt Bryan The $25,000,000,000 EigenvectorThe Linear Algebra Behind Goo