Principle Components Analysis (PCA)

Rebecca C. Steorts, Duke University

STA 325, Chapter 10 ISL

1 / 26
Agenda

I Recap of information retrieval

I Relation to PCA
I Idea behind PCA
I Application to NYTimes data set
I Multidimensional scaling
I Scree plots

2 / 26
Information retrieval

I Recall that information retrieval systems often represent

documents as what are called bags of words.
I Such documents are represented as vectors
I Each component counts how many times each word in the
dictionary appears in the text.
I This throws away information about word order.
I Part of the representation of one document might look like:

a abandoned abc ability able about above abroad absorbed absorbing abstract
43 0 0 0 0 10 0 0 0 0 1

and so on through to zebra'',zoology’‘, “zygote”, etc. to the end

of the dictionary.

3 / 26
Information retrieval

These bag-of-word vectors have three outstanding properties:

1. Most words do not appear in most documents; the

bag-of-words vectors are very sparse (most entries are zero).
2. Small number of words appear many times in almost all
documents.
I (Examples: “the”, “is”, “of”, “for”, “at”, “a”, “and”, “here”,
“was”, etc.)
3. Many words’ counts are correlated with some but not all other
words.

4 / 26
Takeaway

I Don’t get much value from keeping around all the words.
I Better off projecting down to a smaller number of new
variables.
I Project partially since the words mean slightly different things.
I This is exactly what principal components analysis does and
it’s very useful as an exploratory data analysis exercise for
dimension reduction.

5 / 26
Principal components analysis (PCA)

I PCA is a tool for exploratory data analaysis and dimension

reduction.
I Take large set of correlated variables and replace with smaller
number that collectively explain most of the variability.
I The PC directions are directions in the feature space along
which the original data are high variable.
I PCA is an unsupervised learning tool – since it involves the
features X and no response Y

6 / 26
Why is it useful?

Suppose that we wish to visualize n observations with

measurements on a set of p features, X1 , . . . , Xp as part of an
exploratory data analysis.
We could do this by examining two-dimensional scatterplots of the
data, each of which contains the n observations’ measurements on
two of the features.
However, there are p = p(p − 1)/2 such scatterplots
For example, with p = 10 there are 45 plots!
PCA seeks to to find a low-dimensional representation of the data
that captures as much of the information as possible.

7 / 26
PCA

PCA seeks a small number of dimensions that are as interesting as

possible, where the concept of interesting is measured by the
amount that the observations vary along each dimension.
Each of the dimensions found by PCA is a linear combination of the
p features.

8 / 26
How many principal components are there?

There are a total number of min(n − 1, p) principal components.

For some reason R prints out the n principal component (and I’m
not sure why it does this)!

9 / 26
PCA

The first principal components of a set of features X1 , . . . , Xp is the

normalized linear combination of the features

p
X
zi1 = φ11 X1 + φ21 X2 + · · · + φp1 Xp = φi1 Xi
j=1

that has the largest variance.

Pp 2
By normalized, we mean that i=1 φi1 = 1.

I φ11 , . . . , φp1 are the loadings of the first principal component.

I Together the loadings make up the the principle components
loading vector φ1 = (φ11 , . . . , φp1 )T

10 / 26
Finding the first PC

Given a data set Xn×p , how do we compute the first principle

component?
Since we’re only interested in the variance, let us assume that each
of the variables in Xn×p has been centered to have mean zero.
We then look for the linear combination of the sample feature values
of the form

p
X
zi1 = φ11 xi1 + φ21 xi2 + . . . + φp1 xip = φi1 xi1 . (1)
j=1

that has the largest sample variance subject to the constraint

Pp 2
j=1 φj1 = 1.

11 / 26
Finding the first PC
Finding the first principle component loading vector solves the
optimatization problem:

 
1 X p
n X  p
X
max ( φj1 xij )2 subject to φ2j1 = 1 (2)
φ11 ,...,φp1  n 
i=1 j=1 j=1

Using equation 1, we can write the objective in equation 2 as

n
1X
z2
n i=1 i1

1 Pn
Note: Since n i=1 xij = 0, the average of z11 , . . . , zn1 is also zero.
Thus, the objective that we are maximizing in equation 2 is just the
sample variance of the n values of zi1
where z11 , . . . , zn1 are the scores of the first principal component.
12 / 26
PC

The optimization problem in equation 2 is beyond the scope of this

course, but it can be solved by what is called an
eigen-decomposition (it requires linear algebra).

13 / 26
Geometric Interpreation

There is a very nice geometric interpretation for the first principal

component.
The loading vector φ1 with elements φ11 , . . . , φp1 defines a direction
in the feature space along which the data vary the most.
If we project the n data points onto this direction, then the
projected values are the principle component scores z11 , . . . , zn1 .

14 / 26
New York Times Example

I Have news stories taken from the New York Times Annotated
Corpus
I Consists of about 1.8 million stories from the Times, from 1987
to 2007.
I Stories have been hand-annotated by humans with standardized
machine-readable information about their contents.
I From this corpus, have randomly selected 57 stories about art
and 45 stories about music.1

1
Turned them into a bag-of-words data frame, one row per story, one column
per word; plus an indicator in the first column of whether the story is one about
art or one about music.
15 / 26
PCA for NYTimes
load("pca-examples.Rdata")
# The workspace now contains:
# nyt.frame.raw: a data frame
#with counts of words (columns) in stories (rows)
# first column, "class.labels",
# is a factor indicating "art"
# or "music"
# nyt.frame: the same, with word
# counts suitably normalized and weighted
# art: vector where each row is itself a
# vector of words giving the
# actual stories about art, with
# punctuation removed, etc.
# music: ditto
# Some miscellaneous functions used to
# create the data sets (see end of
# this file for gory details)
16 / 26
PCA for NYTimes
# How big is it?
dim(nyt.frame)

## [1] 102 4432

# Remember: rows = stories, columns = words (except the first column, which
# is the type of story)
# What are some typical words?
colnames(nyt.frame)[sample(ncol(nyt.frame),30)]

## [1] "penchant" "brought" "structure" "willing" "yielding"

## [6] "bare" "school" "halls" "challenge" "step"
## [11] "largest" "lovers" "intense" "borders" "mall"
## [16] "classic" "conducted" "mirrors" "hole" "location"
## [21] "desperate" "published" "head" "paints" "another"
## [26] "starts" "familiar" "window" "thats" "broker"

# A little bit of the data

signif(nyt.frame[sample(nrow(nyt.frame),5),sample(ncol(nyt.frame),10)],3)

## jacket patch tapes want ford failed condemn intentional confined

## 24 0 0 0 0.0000 0.0000 0.0000 0 0 0
## 2 0 0 0 0.0275 0.0704 0.0000 0 0 0
## 85 0 0 0 0.0482 0.0000 0.0000 0 0 0
## 59 0 0 0 0.0000 0.0000 0.0000 0 0 0
## 76 0 0 0 0.0000 0.0000 0.0215 0 0 0
## destroyed
## 24 0
## 2 0
## 85 0
## 59 0
## 76 0 17 / 26
PCA for NYTimes

I Need to omit the first column in the first command because it

contains categorical variables, and PCA doesn’t apply to them.
I The second command just picks out the matrix of projections
of the variables on to the components.
I Called rotation because it can be thought of as rotating the
coordinate axes in feature-vector space.
# Do PCA
nyt.pca = prcomp(nyt.frame[,-1])
# Omit the first column of class labels
# Extract the actual component directions/weights for ease of reference
nyt.latent.sem = nyt.pca$rotation

18 / 26
Bi-plot −0.2 −0.1 0.0 0.1

art
47
0.2
46
1 museum 41
3833 55 14 49

0.1
6 images 20 28 50 37
53
artists 15
2710 49 3 44 3045
donations
painting
29 40 museums tax 22 26
0.1

paintings 17 16
sculpture
43
48 12sculptures
56
white gallery
painted 31 2 25 8
21
artist
nature
decorative
picasso color patterns
computer
paris
war
collections
contemporary
diamond
exhibition
works digital
panel
stone
space
media 54 51
service
feetstatue
computers
dealers
technology 96 88
32 11
13 workmachine
red
figure body
these century
X.s
soho
raphael
landscapes
organized chinese
floor
cultural
objects china
association
visitors
italy
building
garden
print personal
public soviet
viewing
chelsea hans
curators
views
hand s
7 gagosian human
paint
photographs heador
large
room
ground
small
sculptor
landscape
silverlaw
sometimes
curator
modern
surfaces
portrait
woman
photograph
conceptual
itself walls
portraits
gray
earth
form
collection
wall
italian
treasures
heroic
against
colors video
collectors
galleriesvienna
hidden
hong
recalling
include
past
naked
foot
carved
materials
whitney
america
separate
may
daumier
johnsongerman
frame
windows
palace grid
piece
light
experts wood
europeans queens
software
river
pills
city page feb
pictures immediate
painter
catalogue
impressionism
photography exhibitions
shop
almost
canvas
other problem
statements
models
life
propaganda
change plastic
orange
newman
congress
simultaneously arefront
directly
europe
prince
ends
ancient park
kong
must system
among
proposals
private
involved jack
entrancefine square
5 imagesubject
ruckus
paper
rothko
corner
paints
menil
makes
been
him
subjects
surface
many
formsmessages
prints
useflesh
collecting
black anonymous
french
suggests
some sale
revolution
appropriate
largest
canvases beyond
boughtclay
recovered
also
architectural
thereshades
colored
upon
man
government
skylight
impressionist
picture
figurative
provocative
that
school
depicts
commission
furniture
borders
transparent
women
made
dark
too usmake
shaped
possible
auction
fragmentation
remains
installed
murray
domestic
expected
tended
thoughtful
which
constructions
no
projections
hope
far
view
arranged
trees
74
european
culture
radical
screen money
land
object
parks
display
printing
seeminglymoore
johnsonart
surrounded
superimposed
worked
significant
abstractions gown
essential
theme
promising
few
percentage
animals
countrys
double
murals
used
spread odds
covers
island
percent
physical long
genre
consumer
heads
plaster
based
fashion
giving
embedded
cut
frescoescross
lures
collective
shapes sue
neo
pattern
hallucinatory
saul
viewer
stitched
photographicdeeper
spirit
countries
conscious
each
progress
several
possibilities
shape plain
hang
baroque
entirely
bottom
place
johns
blue
minimalist
photo
animal
unrelated
generations
provides
yellow high
nancy
blacks
allowing
buildings
developer
stylish
dealing
science
pictorial
narrative
renaissance dots
chris
glueckart
acquisition
size
layers
different
artwork
face
homeless
degree
female key
cotterart
intent
valuable
visualmike
silhouetted
ambiguous
holland
generated
golden
advantage
another
appears
arm
interested
combining
such easel mural
case
begun
exposureear
planet
men
pristine
online
williamsburg
notably
meaning
across
everyday
microscopickind
toward
depicting
floating
constructed
sculptural
community
generation
concerned
clothes world
indoors
million
original
smith
realm
suggested
montages
similarly
recenttable
reconstructs
france
leavesguards
engraved grace
X.this
professor
creatingrome
fabric
serious
placed
importance
millennium
improved
patrons
exception
trompe
ambiguously
virtuosity
grandiose
haveshot
mundane
dealer
mason labor
books
barrel
lithographs
appealing
unusual
merge
estimated
larger
council
produces
added
little
resembles resin
privately
property
infamous
transform
enjoymentrobes
himself
condemn
thomas squid
gun
sports
invasion
unknown
untitled
shouting
deadpan
intact links
kennedy
visitor
madison
officials
esthetic
references haunting
necessarycircles
compelling
every
macho
enhanced
germany
nowhere
doubt
intimacy
military
turning
maurice
surroundings
hair is
agree
gabriel
does
magazine
nearby
adamglobe
glory
snakes
samaras
making
taiwan
paces
killed
originality
shopping
simple
somewhat apple
loosely
realistic
centuries
transgressions
multimedia
brushed
composition
modernist
suggesting legs
mythological
manipulated
rendering
handsome
through
institutional
enthusiastic
efforts
mistress
sensual
japanese wax
rivers
content
painters
chemical
changes
eventually
headache
handspicked
philippines
continues hunt
identity
minimal
signsroman
fact
inflated
ray iron
exhibit
concrete
formal
projection
perception
political
model allow
history
displayed
requires
cause
myers
ceiling andy
pay
lies
hold
fighting
example
entriesbird
social
childs
plus
manhole
legal
shared
near
implies
department
nostalgia
minimum stands
sequence
vision
one
restoration upside
flipped
worlds
heavy
longer
architect
quantities
masks
areas
critical
printed
bronze
born
mood thick
neighborhood
plywood
renderings
panels
market
hundreds owner
appear
natural
psychedelic
pruning
creates
counter
beating
outright
numbers
sphere
bourgeois
gallerys
subversive drive
flames
installation
message
interpretation wry
ranging
window scale
conveyed
roberta
connected rely
design
per
tiny
value
artificial
remaining
casts
retreat
status
architecture
expansion
economic
revenue
statuary
uses
roughly
move
prompted
jasper ansel
explore
evoked
lucas
obvious
precision
removed
passionate well
partly
bodily
primarily
can exist
signed
ruschaside
boyunited
served
offices
union
africa
flow row
break.
facade
compositions
site
order glass
historic
built
givenoct
remain
industrial
lacking
map bleak
entire
traffic
measure
blast
italians
specialty
wooden symbol
apartment
more
collector
effect big
our
pack edge
urban
whether
travel
selected
mirror
silhouettes
former
respectively mid
inside
breathing
known
faces
inspired
overlapping provide
tops
henry
by
illegal
buy car
lobby
surrealist
covering
dancer
cubist
language
governmentschips
environmentdiverse
protection
here hungry
miniature
certainpower
others
civic
grids
erupted
spain
invisible
bust
ken
rocky
survey
mystery
gardens
reward planted
crude
encounter
mantegna
resting
closer
projects
scenery turn
younger
items
suburban
destroyed hugo
wooster
rocket
nude
austria
clarity w
states
visit
touch
granite
snout
exchange
feministstrip
text
finger
lackwithin
material
photographers
express
alien
supported
spending
dazzling
representation
overcrowded review
result
birds
grays
labeled
mask
lacks
illuminated
illustrational
obviously
detail
cinematic
abstraction
marked paolo
broker
eccentric
loeil
bamboo
strength
parisian
member
busy
from
northern
realism
mirrors
bathed
experimenting
recurring
preserve
rectangular
standing air
overlooking
believing
owned
understanding tends
explained
encountered
handling guide
bits
varying
volcanic
dreamlike
bodies
current
rectangle
womans install
offer
jewelry
architectsprice
ritual
restored
purpose circle
solomon
establish
houses
watchesbush
dynasty
presence
plants web
fire
else.
rifle
purity
loss
tradition
structure
pile
minority
graffiti
triangle
results
access
idealize
perhaps
worry
possessallude
main
beige
wide
engaging ma
ii42
lights
israeli
corcoran
warhol
die
u
designed
denis
purposes
lisa
single
pieces
tap wish
free
post
plan
styles
held
groups
match jan
rX.th
vistreet
avenue
internet
project
lent
serveeditor center
studio
process like way religion
people
family
cummings
only
look
abstract
themes oversight
improvement
captures
expressionistic
seeing
limited
took
packed
processes
acknowledge eye
strokes
published
psychologicalblurring
especially
manet
greatest
heeled
superintendent
rauschenberg
establishment
embody
impression
accounted
sharply
differently
although second
echoes
gerald
lined
recreate grow
arrive
banality
brick
wrong
sigmar
cars
registry
distant
fires
holdings
except
carlo
kinds
tears
matched
countless
trust
connections
were protected
arrived
academic
gives dehired
suspect
acre
evokes
resistance
foundation
depictions
captivated
monochromes
covered
destruction toy
shots
perched
undulating
shown
agreement
dressed
elementary concern
adviser
method
three
brother
conceptdiscuss
aesthetic
geometric
words
reverence
appreciation path
sunlight
drawings
referring pole
giulio
approval
literally
previously
happeningcrab
possibly
polke
affairs
thievesskyline
brandywine
taking
interiors
already
function.
reopen
library
escape
screens
julie
sponsored
mug
promote
instance
sits
doubling
study
green
venus
crowded
retrospective
certainly
emphasis
extensive
profoundly
peace
opposedenough
subsequent fingers
adjacentroots
anger
read
previous
development
represented
suspended
concoction
foreground
simpsons
confirms
identified
abandoned
come
forwardown shaping
scenes
feels
reverse
clients
legitimate
narrativesidea
products
foundations
inherited empire
speak
lying
perfectly
empty
newspaper
francesco
chanting
naturally
equipped
catholic
detached
response loans
county
hero
bronzes
shoulders
phenomenon brown
depicted
approve
campaign
estate
humiditydead
mixed
nightclub
flag
invitation
matisse
temporary
transforming sam
reality
pursued
artifacts
rushing
easily
darkened
infatuation
clinical
christianideas
take
w
household
surgeons rises slowly
bit
related
selling
canadian
sun
lifes
become
sell
preacher oval
noting
minister
heritage
hole
totem
start
vast
arrested
questions
youthful
cumming gate
fight
exterior
give
treasury
luxury
elements
increasing
contributes
installations
neighbor
tricky
ideologicalclimbve
energy
xavier
tail
salle
honor
income
gain
yielding
sex
inch
evoking
flatest
glimpse
outside
whole
permanent
shell
lawn
approved
signing
liked
disaster
lighter
walk
evidently
chicago
violence
parts lust
surprises
cease
electronic
skirt
activities
conventions yu
russian
reporters
section
wash
outstanding
patron trash
basic
vintage
increasingly
believes
committed
homage
mezzanine
carrying
manipulate realize
succeeded
stresses angry
protect
arte
two traces
besides
reputations
heroes
lot
sign
railings foam
club
risk
mind
share
adams
tree
pat
rapid
studies
illusions
hybrid
marblecy
identical
uncanny
charles
essentially
alain
lonely
pope
quiet
homes
mayor
mitchell
allusions
cerebral
woodcuts
reliefs
disappear
surrounding
breathed
an
compete
enterprise el
easy
enlarged
avoid
cloying
david
neil
itschairman
float
torso
islip
provespoints
balloon
evoke
living
sophisticated
scholarcommon
summers ays
howard
consists
inner
classical
wild
conceived
interior
english
illustrate
handed
michael
vivid
pleasure specific
replace
robert larry
tourists
found
radically
masterpiece
thousands fusion
jackson
sort
loan
fear
enablemale
forest
britain
effectively
roads
institute trip
midtown senate
directors
likely
tucked
buying
biographer
abroadlab
cited
gone
resulted
sitting
damaged
reason
fundamentally keen
died
bank
opposite
aug
similar
iris
depict
technically
learning
cradle
vaguely
ignores
audacity shiny
grass
search
patently
fuss
conjunction
category
juxtaposition
showing pulled
reduced
campus
output
wars
complex
emphasizes expert
accept
create
water
dominated
memory
andrea
deadly
skyscrapers
combinationcentral
forget levels
goods
drinking
twombly
ourselves
lawyers
remarkable mine
moreover
walker
legislators
momentum
age calvin
televisions
settings
designated
mount
assistants
intended
caused
using blunt
successor
completed
stimulating
engaged
commissions
characteristic
apprehension vincent
intense
highest
matter
pleasures steps
insists
interesting
rules
foreign
rejected
stones
visiting
ticketmaster
controversial
controlled rarely
stretch
helping
sketches
entrepreneur banal
further
mode
add
parents
partner
blackness
purchases
addition
over
studded
surprisingly
imaginejohnss
revival
historians
cracks
responsible
smearing
wethelp fewer
colleague
scholars
specifically
creamy
suggest
accompanied
hanging
decorated
background
excitement
meditation
laurence
minded
sawsawsdoesnt
associations
tomkinss divided
citys
retained
fled
narrow
liquid gold
drift
wave
citiesdi
doug
attitudes
regulations
historical
into
authoritiesabove
bunch
statuettes
mogul pan aim
mall
alternate
stained
february
decision
deal
travelers
onto canal
corporate
propslets
exits
block
titled
portion
degrees
upper
deeply
beautifully
cork
fields
nearly
questioningneatly
detailsboard
aging
falling
miller
severely
maybe
expressionist
marriage
definition
watercolor
concerns
creation
backgrounds
expressionism
acknowledged
assemblages silk
positioned
according
represent
jungle
manuel
giant
cement
functions
impress
demanded receive
plush
thing
applies
harvested
drafted
aspects
simpler
bernard
defines
viewed allen
moves
prime
exotic
competition
schools
locked
game
hallway
recover
exhibited
operating
rothenberg
groom
dedicated
backward crystal
stanley
francis
occurcozy
need
citizens
practical
drivenangels
razed
feature
fred
local
gayawe
total
outdoor
answer
issues
remind
additional
around paspaces
spatial
mix
rawviolins
fame
developers
com
vice
nonprofit
promiseacted
republican
strips
plum
breaking
a
arose
prices
gift end
steepedlink
enjoy
loose
mixing nov
beach
sets
senator
polished
thank
atlanticthus
tv
admission
barriers
medical
catch
ongoing
pioneer
prejudice
formedgraham
stable
healthy
in.
press
buried spend X.d 62 34 5887
0.0

screaming organization
sense refuse
synonymousanxiety
indeed
rejection
aware
aggressive rage
fantasy

0.0
he we
theydaughters
whose
taken
polaroid dress
photographer houston
figures
determined
modernism
arms
expressed
his
centerpiece
huge
awareness
deep
commissioner
so customers
their
brushes
advertisements
painterlyclosely
currently
squeezed
looking
traditional musings
bright
steinberg
esthetically
question
accompanying
while. provider
brand
condition
atmosphere
towards
realized
owners
rendered
businessman
possession
interviews
going
things
masterpieces
lovers
photographed
scenario
mostmedium
evidence
celebrates
pandemonium
number
see papier
experience
called
categorically
commitment depth
crowds
master
providing
chair
surge
intentionally
charm
meat
ones
artful
aligned
recently count
sky
ruins
managed
optimism
learned
discussed
nelson
influential
blended
removing
structures
addresses
romance require
horse
metropolis
blow
snake
including
distribution
measuring
witnesses
meeting
cataloghas
rather level
type
X.nd
potential
financing
glue
sightings
idealistic
crane
associatestare
arrangements
charcoal
behavior
intimate
drawn
mouth
pair
captured
wonderful sides
palette
churchso
turner
courting
mismatched
brought
california
watch
contribution
universal equate
meadows
style
reduction
renovation
immediately
associated
muted threat
colleagues
sizes
fellow
adamss
limousine
wading
stepped
support
advance
preservation
sending
treat
monet
length
smithtown
hicksville
consumers
demonstrate
couple
anything
father
quarter
attracted
william
mounted
otherwise
partnerships
blank
thoughstuff
considered
pick
dumpy
issue
acrylics
ebullience
influenced
revisions
unfinished folds
mache
generally
knee
pages
speculation
produced
drawing
mountains
independent
still
assembled
weight
representative
interview
envisioned
manages
conservative
absorbed
neverthelessany low
transformed
varied
creations
advanced
say
adaptation
punishment
would
missing puts
possibility
substance
meanwhile throw
sharp
staircase
commissioned
foundedlend
fashionable
self away
where
contrast neal
sardonic
inhabited
network
avant
grows
specializes winslow
joy
conventional
transformations cliff
pablo
olson
effects
rise
sixth buzz
delicate
exceptional
monumental interest
boundaries
monolithic axis
difficult
either
softer
actually
strange
compact
craft
implications
questioned soap
false
police
adopted
robinson
traded
flowers bay
periods
albeit
expression
shooting
reflect enter
rice
north
flat
fill
chadds
bare
acquired
sons
distinct
yard
albert
moscow
proportions
pressing
entrances
dad
dreamily
coincide
between
combined
traveling vista
attract
extent
format
sure
babies
henceamid
fresh
full
artworks
moment
seems
noticed
depressionfamilys
chief
smoothstand
identify
intellectually
register
corridor
smoke
remarks donald
famous
pass
saved
jail
womens
caught
certificate
comic
nondescript
represents
memoirs wiped
ryder
enjoyed
to
advising
trade
tragedy
concluded repeated
percussionists
driving
partners
percussion
guggenheim
expect shut
feelings
everywhere
methods goesicons
richard
often
deborah
elizabeth
imagination
real with royal
lots
bizarre
jeremy
emily
administration
puzzling
attempt
momentarily
opportunity
ability
treatment
saxophone rat
paths
passion
anti
rude v
sleek
timed
sufficient
followers
signature
down
somebody
then unique
colorful
visibly
flight
tend
spacey
remembers
presents
necessity
celebrated
cotton
brancusi eighth
sought
streets
wolf
informal
comment
normally glenn
assurance
passingfloors
truly
bushy
grand
shortly
oils
approaches
sinking
cool
sudden
component
someone
kings
force
societieships
woogie
boogie
motionbigger
pointed
planning
existence
surrounds
construction
coloring ships
christopher
routinely
ago
poster
abundance
burned
severe
step
under
mostly
quest
delayed
couples
engineering
seem
numerous
primary oil
prize
alfred
clubs
titles
hoping
produce johnny
rich
killing
favored
recognition
deciding suit
quietly e
tech
ford
lrented
incorrectly
greater
cite
texts
politics
kiosk
stop
reveals
goghflair
sept
bull
antonio
nicole
costs
sharon
serene
designs
visited
whats
threatening
unlikely viennese
poet
lasting
sink
earlier
distinction
address
derived
edges
appreciate cover
spinal
rob
rochelle
leonard
movedsuffolk
fits
lived
pulls
accessible
set
prospect
elsewhere
gradually port
vogue
ralph
western
latter
bowl
thrown
ripple
massive
washington
helped
opportunities
incorporated traveled
magic
italys
filled
literary
futuristic
sexual
fragments decide
probably
humans sargent
tacticswore
symmetry
jetsons
profits
stick
district
victims
curiously
gesture
holding
philosophical
lower
institutions
respectable
collage
perspective
fashioned spare
experiences
heading
dream
amount
coordinatedsemi
attended quite
defining
develop
stream
dry
allowed
sleep
piles
dynamic
fanciful
barely
continue
nothing visible
exploits
coupled
guidelines
engineerscuts
accordingly
mens
territory
mondrians
suggestive guilty
worse
advocates
glamorous
indiansampling
poolthin
clearly
fringes
active
running
revolutionary
compare cry
motifs
celebratory
august
nonetheless
camera
surprising 63
advised
waived gap
baker
mines
pam
mexico
towers
pentagon
skin
negotiations pulse
twice
alert
leftover
elegant
relatively
eight
shares
catherine
catastrophe
exactly
changing
writes recall
hopes
rush
ask
printmaking
handled os
hat
spring
burton
cold
ignore
collaborated na
needed
estimate
japan
mans
threw
filter evenings
wing
snow
research coca
leon
reached
accents
wealth
widely
cathedral nor
interact
lost
gertrude
jacket
analysis ticket
balance
anne
copyright
salary
irving
stages
scramblepain
scary
settled
panic
uncertainty
alternative danger
lasted
reputation
preparing
ideals
keeping
admits
representatives
shadow older
stepping
knowing
handle
themselves
working slide
defined
sponsors
wound
gothic
mention
future
trace
endless
expressing
snappy
flower
proposed
watching
allows broad
strings
border
bacon
topped
servicesand
ronald
hospital
virtually
harder
lavish
boost
undoubtedly
spoke
existing
enthusiasm
even
discover
janis
harbor
warmth
discovered
complete
exhibiting
satisfaction
penchant
done return
considerable todays
luminous
smart
thinksdads
lone
surprise
bring
tools
charge
call
growth
philadelphia
source rough brian
control
warren
centers
pavilion
firm
primal
capital
recapture
cartoons equally
options
archetypal
carries
selection c entry
stark
aside
implied joke
jokes
resources
orchard
stopped
crafty
latino
introducing re
distinctive
phillips
pants
gained
crucial
preparatory
cowboy
renovating
absorbing
raised romantic
attorney
instant
capture
december
shirts
convinced
dr debt
ambitious
reopening
precisely
blanket
encouragedbadly
individual
fellowship
conversationleft
contemplative
thinking range
might
robustly
statement
magazines
advertising south joint
pub
expand
saint
authority
cannot
terms
followed
turns
conclusion
substantial
explains
wondercontracts
masters
enormousterm
fishing
skeptical uniform
president
volume
stripped
damage
meant
name
exploring
qualities
cutting elder
together
editors
presenting
state
listening
towns the
returns line
build
melody
lee
metropolitan
connection
concludes
dimension
minneapolismilesstraight
members
follows
clark
reinforce
industry
depart berlin
letting
information
crowd
rating
stuck
theres
holds
restaurants
gently
of
ride
burleigh
replied
action
coast
eagerly
dollars
retired
indicated
laughed frozen witch
des
continuing
casual
below
descending
very
choosing
motif
personality
refused basis
mess
steinway
fascinating
knocked
positions
heavenau yorks
affected
roomsdot
affair
presented
speaking
remained
choice
invention
arent
jones
phase
adhering
approached
standard
necessarily
viewers poor
cunningham
discussing
experienced copied surely
tower
movement
repetition
slow
influence
on
swinging
unnoticed
infrared
whenever
edwards
tennessee
popular
focused eric
let
tender
loved
subtle
decidedly
cheaperroot
leaving
anyway
creativity
playfulgreat
finances h
quintet
negotiated
willing
gather
fulton
du
bound
approach
dolphin
apparent
bergman
suburb
quality
talent
listened
profession proud
acquisitions
forbidding
arrangement
premise
direction
lighting mixes steel
present
alive
player
pre
giantsmtv
rita
global
harsh
push
forces
merce
chef
optimistic
frederick
strongly
notnotable quick
starting
students
required
operation nine
mass
soft
hot
interests
leagues
ominous
brings
simplicity
costume
tapes
road
max
masses
jeffbus
directing
anchored
essence
walked
carried
chest
forced
happens
propelled top
bounds
desire
fueled
ambitions
enveloping
special
financed
difference shine
ball
winter
taaffedisk
venue
late
compagnie
nicely
proportion
starred
this
confident
championships
round flashes
revisit
seating
faster
adds
proscenium
explain error
wadsworth
atheneum
unable
restless
strollscott
attending
contained
nicholas
reasons
consuming tie
closes
clich
invited
focus
greek
doors
professionals
seductive vernon
oldest
relative
involvement
character
alter rising
movie
elemental
bed
wichita www
direct
loud
pit
shops african
blocks
fifth
gary
leading
davis tall
ties
pure
reading
international norm
today
tin
gathering
reflects
clips
comes
explosive
anthony wei
john
cola
able
etchings
venice
parties hows report
jazz
samuel
committee
op
homer
times
peter
gelder
following
composednote
billion
captain
loyal
clotted
rapper
tour tully
van
piano
pierre
effort halls minor
brant
west
bar
practice
emerging
largely
taylor
urge
initially liberty
brahms
patch briefing 73
emulsion
what
out
photos
daughter
show texture
because
if.home tfeeling
popularity
splash
preparation
guarded
richardson
described
marvelous
purple
accommodate
inspiration
techniques
years timeless
house
convincing
became
youngdeath
boxes
husband
know
impressive
yearas marc
akes
confined
parsons
repeatedly
security
ringing
think
knowledge
during
moving
talking
middle
elected
them
collaboration
everything
dancing
gettingdecades
complexity
being
broadcasting
wasnt
tendency
backing
eagerness
havinghaydn
uplooks
changed rebecca
mansion
wears
nassau
innocence
thematic
went
sentiment
assumedbut
screenings
food
mary
americans
mysterious
shimmeringkiss
growing
course
friendship
quilts
cartoon
commercials
off budding
cheerful
promoting
eyes
back
chosen
problems
without
funny nervous
returned
desperate
sold now
neck
wouldnt
did
reopened
lives
heart
continued
alwayspuns
shakespeare
startling
usually
want
adventurous
drama
involve
word
banks
raise
gods starts
established
longtimepoint
newly
busiest
backdrop
contributed
mainstream
theyre
billboards
traditionally
survive
loft
dayjuly
alienation
examination
prepared
functionalbe
daily
than
portrays
grasp
amsterdam
agreed
humor
faith eyed
brooding
vibrant
particularly
audiences
grammy
backers
believed keep
education
super
sitcoms
newest
africas
impending
station
insisted
weapon
garbage
everyone subtlety
spokesmanpretty
dreaming
contemporaries
strongest
permissionneither
originally
veterans
believe
barefoot
developingwalks
elaborate
married
admired closed
should
obliterate
author
shout
misspelled
secret beat liam
presenters
reflecting
artistic
telephone
amusement
accused stood
labels
stretched
ready
creatively
putting
finding
revealed
sheer
somber
hintship
junior
theatrical
shocked owns
roy
musician
posters
overall
complained
gestures
son
mission
critic
weather
lynne
rocks
occasionally
could
wanted earthly
seattle
posturing
soldier
bohemia
insurance
mcdonald
climate
freedom
standards
shakespearessend
rolling
settle
attentionreplaced
bravado
sentimental
emphasized
ultimately
citizen
whom yet
those macys
ending
stores
spur
noted
mentions
dreams
novemberplease
conn
biggest
brice
passed
valley
filedsent
partnership
affection
adding
notice
boulder
grants
injuries
animated knew
companies
dissolved open
vaudeville
gets
forever
reported
finished
means toured
resolved
months
familiar
inventions sea
trouble
calm
training
chance
evolution
carol
blessed
september
wear
command
put
supposed
vulnerable flew
marketing
extends
dont ignatius
care
criticized
radio
since
again sight
fail
experimental
inherent
chapin notion
resident
wants
multiple
losing
witness
garage
scholarship
complicated pink
ended deft
hartford
once
court san
sunny
contributions
jane
renovated
failure
wilson
offensive
bounced
outsider
scene
swirling seatsed
staff
diana
true
leaps
strictly
restaurant
worldwide
morning
worthissued
promised
millions
periodcrew
producing
headed
demands am
cable
creatures
premieres
obtained
gilded metal
days
linda
careers
smell
reflected
reports
wife
hearing
demand
spanish ky
tells
bridge
until
comfort
anyone
weekly
trick
achieved
cap era
ambition
joseph
seven
clinton
productsarah
dawn
comparison
much
album
lookedsuccess
compared von
track
youth
half
swing
bridges
lines
afternoon
bruce
meet
ross
cole
provided
workshops
adventure
contours
swiss
workshop
clarinetist
brilliant
strengths
cousin
preview
creative
powerful
landed flemish
eager
touches
crawl
breakthrough follow
spectacular
matching
modest
dog
best
impossible
phalluswire
becomes vain group
chic
balanced
reminder
warhols
try
timpani
chose
utter
curtain
part
succeed
louis
hearings
remember
choreographed
cautious b
tone
mock
hoffman at
setting
bills
successful
october
poetry
occasion
ben
studied
cream
publications
resemble
stories various
account
suffered
obstacles
stefano
midnight
episodes
programs
shoes values
british
intellectual
happened seeking
checks
menacing itaffect
manner
butch
cycle
investment
wheels
cans hill
yorkers
dallas
reference
eastern
awkward
intentional
grounds apart
likes
sat
meldshappen
conduct
afterward
consider
exciting
kiley sees
nails
reach
leslie
duet joe
opened
context
talks
exceptionally alliance
laura
named
news
started
find
oclock
hinted
along
channel
spent
check
before
porter
slash
londonnew
carousel
personalities
ceremony
douglas
chooses
gag
auditorium
programming
dresses harrisons
hard lists
nostalgic
recordings
gaining
librettist
frustration
letterman
label
earned
wondering jerry
ann
voices
costumes
moon
fabulous
elegantly
simply
suddenly
curious
feldmans
bubble
perfect
god
texas
relations
receives
kenneth
sounded
beautiful
kept rateabc
nbc
pacific
smaller
honoring
tear
owed
deadline yogi
pirated
games
russia
minute
offered
cramer
appearance
beneath
designers
trying
autobiographical
however
decided mark
matters
job
fully
begins
january
lloyd
mountain
praised
planned
steady
offering
missed
victor
philip
plagued
mutual
loves
raising
organizationsballad
five
ahead d
fair
nights
door
hide
names
fit
martha
blake
jimmy
weekend
touching
rca
worsttimbre
division
rhythm
duesonata
steven
least
involves
tenure
why
highly tex
vegas
dive
draw
earnest
picking
bonus
office
sized
throughout
melodramatic
tom
for.
calling
schwartz
legend
briefly
welcome
rembrandt
devised sonic
haven hop
noise
prot
nick
none
choices
american
jennifer
includes
dozens
leno
carry tries
adults
nice
lynn
six
four
plenty
actor
tune
brave
warner
rarity
reservations
correction
helen
created
later
hits
academy
amy
cost dec
offers
toccata
unlike
brief
waltz
novel
goal
unclear
sales tonality
suits
sites
tbeethovens
weill
jr
featuring
anniversary
leigh
casting
weeks
afford
workers
kane fate
signaled
irony
profile
thrift
oscar
backstage
delightful treated
alliances
principal
both
proof
behest
hunter
legendaryhes
league
right
attempts
thoughtstiming
tractor
agent
official
webb
circa finds
instead
authentic brook
plans
seemed
canada
patti
stills
felt
annual
choral
ex
dancers
closing lelincoln
business
montreal
stint rare rose
bass
dick
brooklyn
conference
referred
retitled
minutes
york
critics
invoke
somehow task
leave
cyaleapacity
giuseppe
celebrate
wildly
clue
ultimate chamber
friday
telling
particular
afraid
general
birthtomorrow
increase
ads
watercolors
third
francisco
scattered
football
columbia
version
calls lawrence
luck usual
violin
party
twist
strong
series
violist
stein
hustle
list
hasnt
lounge
lifetime
lead walter
cafecalif
lichtenstein
relevant
florence
favorite
extra
begin
heavily
position
booth
waiting run
available
marie pairing
features
miracle
improvisations
improvisation
williams tried
instrumental
unfortunate
fascination
never tuesdays
extraordinary
description
phone
paying
chapter g
del illorgan
beaux
arts
larkX.
quartet
musicians
alice
jsound
instruments
shrill
columnjoined
recitals
opulent
tchaikovsky
salute cast
lyricism
appearing
occasional
cambridge
schedule
advice
elvis
olestatic
james nedva
recital
silvery
debuts
harvey
said met
person
gentle
religious
all
childrens
whodoing
angeles relate
appeared
england
supporting
playwriting
friendmonth
watson
showed
obsessed
really laugh
beverly
recalled
child
fish
unwelcome
surreal
mermaid same
hollywood
folk
ive
explaining
bored guy
exultation
involving
pastiche
challenge staged
dirty
relationship
discussion
laughing
developed senior
harlem
hills
your
broadcast
higher
sondheim
worried
forgotten
wings
summer
documentary
informed
merely
commerce
yang
dinner
struggle
gave
playwright
taught
old broke
soon
pursuing
nominations
mentioned
student
thought
wearing
saying
completely record
beauty
actors
nashville
variety
scandal
professional
searching tony
good
lover
toughroll
puccini
ever
saga
blend
choreography
classclasses failed
rain
credit
cheered
important
assertion
places
confidence
commercial
nobody feel
battle
libretto
impresario
teens
devoted
introduction dozen
truck
justice
dates
winning
stylistically
dissonances
gilman
goodman
university
reprise
autobiography learn
sweetly
emotionally
lengthy sketch
letters
consecutive
moments
paid
versions
guys
records
celebration
happy
got contract
team
something
constantly wit
warm
seat
boston fell
patrick
danced
paul
fare
frederic
harold
boulevard
verses
behind
financial billy
manhattan
useful
verdis
bear
returningseen
tentatively
less
budget
seriousness
lean
greenwich
sitcom
tonights shirt
alone
store
characters
encore drifted
rent
acclaimed
despite
dollar
revealing brass
finally
expletive fall
victoria
recorded
purely
cynical
wont
finish
drew
apparently oh
extended
rid
sings
sweet fun
randall
runs
monday
opens
kevin
guest
scheduled
flavored
arthur
toned
papp
yes
major
lord
chords
alan
families
choreographer
susan
stephen
better stay
airy lion
regular
string
chorus
rest
lyric
choruses
led
concerts
scores
church
soloists
sturdy
ran
host
go
renee fade
juan
musicals
bloom
songwriters
town
drummer
studiosearly
daytime
how
ensemble
youre
nurtures
touring article
jon
king kviolinist
elderly
routine
sections
heard
witty
genres
spotlight fourth
victory
majestic
harmonic
decade
blues
performers
leader
proved
bernstein
bread
butter
hockey
live
slightly
thursday
listen clear
frank
solo
footlights
starring
written
revivals
meetings
imitation inflate
release
alternates
pop turned
falls
appearances
executive
fund cd 52
sounds
misstated
societypy
players
june
pianist
conductor
george
dutch
bill event
cellist
actions
teams trio 5997
story
began
comedy
asked
just nominated
coming
los
about
do
feldman
herself
when
had aftertold
television
film
grew
onstage
applause
precocious
country crazy
bad
stars
classics
tales sister
stentorian
writers
gospel
shows
wind
girl
producer
came
didnt
southern acting
attitude
divaqueen
recording
solid
tunes
hotel
fathers
achievement
hurt
girls
walking
plucky
classic
pianissimoletter
robust
act
movies
andrew
kids films
emmy
austin
included
steve
mr
wisely
exquisite
strike notes
conducted
richness lovely
title
staging
repertory
perform
residence
first
aria
jaunty
soul
hit
short
harvard co
harmonies
boom seasons
wednesday
cabaret
national
canceled
striking
debut
festivals
cat cant
wrote tale
acts
children
sung blood
tours
regional
duke hours
dances nballet
played
purchase
isnt
wolfgang
bands
operatic
carnegie
saturday
announced
mezzo
defiant companys
saints
don
close st
hall east
sunday
program
wagners
symphony
dance
102 et 39
coat time
playwrights
introduced
released
couldnt
opening get
sing
received lane
sunset
friends
producers
beginning
garde
knows
star
tell last
final
next.
rehearsal book
date
orchestral
performed
writer score
talk
orchestras
mozarts
miss hearpreviews
soprano won
events
award
guitarist
engagement
bob will concert
guitar
evening
company
location
college
rock
performing
tickets
roles operas
yesterday theaters 81
99 68
me you love
shes
thats mean
networks
tribute 90
fan
plays
sir
dramatic tuesday
premiere
audience
tonight
tenor
baritone village
brothers
productions
philharmonic playing
morris
director
directed
role night
box jersey
march
composer 80
10119
baby career
rabbit
jonathan actress
singers
writing song lyrics
stage
cbs
PC2

18 cooper
mother
my mrswas idvoice
broadway 57 season
singer performances
week april m 9875 65
79
−0.1

23 imsays 35play
awards performance composers
band78orchestra
matinee 61
singing songs
vocal la musical 66 70
sang 36 production festival music
i 6792 hour 76 89 82 94

−0.1
91 85 24 6069 72
93
−0.2

ms opera
83
71 64 theater
95
−0.3

100 77

−0.2
84 86
sheher
−0.4

−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2

PC1

I This shows the two first PC for the NYtimes data.

I Blue scores represent the scores for the first two PC’s.
I The red arrows represent the first two PC loading vectors.
I Here, a bi-plot is not useful at all since the data is too high dimensional. (See Figure 10.1 for where a 19 / 26
Look at the leading components
# What are the components?
# Show the 30 words with the biggest positive loading on PC1
signif(sort(nyt.latent.sem[,1],decreasing=TRUE)[1:30],2)

## music trio theater orchestra composers opera

## 0.110 0.084 0.083 0.067 0.059 0.058
## theaters m festival east program y
## 0.055 0.054 0.051 0.049 0.048 0.048
## jersey players committee sunday june concert
## 0.047 0.047 0.046 0.045 0.045 0.045
## symphony organ matinee misstated instruments p
## 0.044 0.044 0.043 0.042 0.041 0.041
## X.d april samuel jazz pianist society
## 0.041 0.040 0.040 0.039 0.038 0.038

# biggest negative loading on PC1, the other end of that scale

signif(sort(nyt.latent.sem[,1],decreasing=FALSE)[1:30],2)

## she her ms i said mother cooper

## -0.260 -0.240 -0.200 -0.150 -0.130 -0.110 -0.100
## my painting process paintings im he mrs
## -0.094 -0.088 -0.071 -0.070 -0.068 -0.065 -0.065
## me gagosian was picasso image sculpture baby
## -0.063 -0.062 -0.058 -0.057 -0.056 -0.056 -0.055
## artists work photos you nature studio out
## -0.055 -0.054 -0.051 -0.051 -0.050 -0.050 -0.050
## says like
## -0.050 -0.049

# Ditto for PC 2
signif(sort(nyt.latent.sem[,2],decreasing=TRUE)[1:30],2)

20 / 26
Look at the leading components

I These are the thirty words with the largest positive and
negative projections on to the first component2 .
I Words with positive projections mostly associated with music
I Those with negative components with visual arts.
I Why do we see women and mothers here?

2
Which direction is positive and which is negative depend on internal choices
in the PCA algorithm
21 / 26
PCA for NYTimes
# Plot the projection of the stories on to the first 2 components
# Establish the plot window
plot(nyt.pca$x[,1:2],type="n")
# Arts stories with red As
points(nyt.pca$x[nyt.frame[,"class.labels"]=="art",1:2],pch="A",col="red")
# Music stories with blue Ms
points(nyt.pca$x[nyt.frame[,"class.labels"]=="music",1:2],pch="M",col="blue")

A
0.2

A
A AA A
A A AA
A A A
A A
AA A
A A AA A A
A AA
A
0.1

A A A AA A
A A A A M
A AA M
A AA
A
A M
A M M
A
0.0

M MM
A M
A M
M MM
PC2

M
M MM
A M A
A M M M
−0.1

A A M M
M M
A M M
M M MM
M A MM
M M
M
−0.2

M
M M
M
−0.3

M M

M M

−0.4 −0.3 −0.2 −0.1 0.0 0.1 0.2

22 / 26
How well is PCA doing?
Even though we have gone from 4431 dimensions to 2, and thrown
away a lot of information, we could draw a line across this plot and
have most of the art stories on one side of it and all the music
stories on the other.
If we let ourselves use the first four or five principal components,
we’d still have a thousand-fold savings in dimensions, but we’d be
able to get almost-perfect separation between the two classes.
This is a sign that PCA is really doing a good job at summarizing
the information in the word-count vectors, and in turn that the bags
of words give us a lot of information about the meaning of the
stories.
The figure also illustrates the idea of multidimensional scaling —
finding low-dimensional points to represent high-dimensional data by
preserving the distances between the points.

23 / 26
Scree Plot

I We can figure out the number of principal components by

fitting what’s called a scree plot.
I Choose the smallest number of principal components that are
required such that an adequate amount of variability is
explained.
I We look for the point at which the proportion of variance
explained by each subsequent principal drops off.
I This is called the elbow of the scree plot.
I These plots are application specific and ad-hoc.

24 / 26
Scree Plot
plot(nyt.pca,type="l", main="")
0.020
0.018
Variances

0.016
0.014
0.012

1 2 3 4 5 6 7 8 9 10

25 / 26
Summary

I Visual analysis of PCA is adhoc (such as the scree plot).

I In fact, the question of how many PC to choose is very
ill-defined in practice and depends on a specific application.
I In practice, we tend to look at the first few PC in to find
interesting patterns in the data.
I If not interesting patterns are found, then further PC are
unlikely to be of interest.
I We could use PC in the context of regression (see Section 6.3.1
for further details and this is more principled).

26 / 26