Professional Documents
Culture Documents
0 2
• What is the first name of the man credited for Markov Chains?
2 0.6 0.4 0 0
4 0.3 0 0 0.7
Overview of Markov Chains: Simple Example
• Let’s use a Markov Chain to make a simple prediction of the weather:
http://www.americanscientist.org/issues/pub/first-links-in-the-markov-chain
Markov Chain Beginnings: Probability + Poetry
• ”If the current letter I’m reading is a vowel, what is
the probability that the next letter I’m reading is a
vowel?
• A consonant?
• What about three letters later?
• Ten letters later?
http://www.americanscientist.org/issues/pub/first-links-in-the-markov-chain
Markov Chain Beginnings: Probability + Poetry
"My uncle's shown his good intentions
By falling desperately ill;
His worth is proved; of all inventions
Where will you find one better still?
He's an example, I'm averring;
But, God, what boredom -- there, unstirring,
By day, by night, thus to be bid
To sit beside an invalid!
Low cunning must assist devotion
To one who is but half-alive:
You puff his pillow and contrive
Amusement while you mix his potion;
You sign, and think with furrowed brow --
'Why can't the devil take you now?' "
http://www.americanscientist.org/issues/pub/first-links-in-the-markov-chain
Markov Chain Beginnings: Probability + Poetry
• Andrey Markov created a new branch of probability
with his findings, now known as Markov chains
http://www.americanscientist.org/issues/pub/first-links-in-the-markov-chain
Markov Chain Beginnings: Probability + Poetry
Markov Chains: Two Algorithms/Applications
• PageRank
• Application – we use it everyday!
• Algorithm
• Implementation
• Experiments
• Directed graph!
PageRank: Introduction
• Purpose
• Rank all webpages in terms of “importance”.
• How to define “importance”
• Analogy: citation, papers with more citations
are more important.
• Option: count how many backlinks a webpage has.
• Caveat: if a page has a backlink from an “important” page, it also should be
somewhat important.
• Weighted directed graph!
PageRank: Introduction
• Original Paper
• Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation
ranking: Bringing order to the web. Stanford Info Lab.
*https://web.archive.org/web/20111104131332/https://www.google.com/competition/howgooglesearchworks.html
PageRank: Definition
• Intuitively
• A webpage has high rank if the sum of the ranks of its backlinks is large.
• Simple definition
• u: a webpage
• R(u): rank of u
• F(u): the set of webpages that u points to.
• B(u): the set of webpages that point to u.
• N(u): the number of links from u (forward links).
• c: a factor for normalization, i.e., a constant.
𝑅(𝑣)
𝑅 𝑢 =𝑐
𝑁𝑣
𝑣∈𝐵𝑢
PageRank: Definition
𝑅(𝑣)
𝑅 𝑢 =𝑐
𝑁𝑣
𝑣∈𝐵𝑢
• Construct a n by n matrix A:
1/𝑁𝑣 , if there is an edge from v to u
• 𝐴𝑢,𝑣 =
0 , otherwise
𝑅0 0 0 0 1 𝑅0
𝑅1 1/2 0 0 0 𝑅1
= ×
𝑅2 1/2 1/2 0 0 𝑅2
𝑅3 0 1/2 1 0 𝑅3
PageRank: Rank sink issue
• Loop (u, v) accumulates rank, but never distributes rank (no forward
links).
u v
• Modified PageRank
• E(u) is some vector that is considered as a source of rank
𝑅′ (𝑣)
𝑅′ 𝑢 = 𝑐 + 𝑐𝐸(𝑢)
𝑁𝑉
𝑣∈𝐵𝑢
PageRank: Random Surfer Model
• A “random surfer” keeps clicking on successive links randomly.
• If entering a rank sink, the surfer “gets bored” and jumps to a random
page.
𝑅′ (𝑣) 1 − 𝑐
𝑅′ 𝑢 = 𝑐 + , c = 0.85
𝑁𝑉 𝑁
𝑣∈𝐵𝑢
u v x
w
PageRank: Markov chain
𝑅0 0 0 0 1 𝑅0
𝑅1 1/2 0 0 0 𝑅1
= ×
𝑅2 1/2 1/2 0 0 𝑅2
𝑅3 0 1/2 1 0 𝑅3
Observation:
1. Results of all graphs converge in less
than 20 iterations.
2. No obvious pattern between
convergence speed and graph size
Wikipedia administer vote graph Citation graph from the e-print arXiv
V = 7,115, E = 103,689 V = 27,770, E = 352,807
iterations = 17 iterations = 17
http://snap.stanford.edu/data/wiki-Vote.html http://snap.stanford.edu/data/cit-HepTh.html
Next Algorithm/Application! Markov Chain Text
Markov Chain Text (Intro)
• Process input text, analyzing letter/word (token) probabilities
• Create graph (transition matrix) to represent those probabilities
• Generate “similar” text using graph
• Order: how many prior tokens to use when generating next token
1 .33
• Example:
• Input: Foo bar foo bar bar Foo Bar \n
.33
.33
Markov Chain Text (Algorithm)
• Simple (bare bones!) pseudocode for order 1:
• PROCESS:
Create dictionary
For line in input:
For token_1, token_2 in line:
Add to dictionary -> {token_1, token_2}
• GENERATE:
Choose first word from dictionary -> token_1
For each desired word in output:
Lookup token_2 in dictionary using token_1
token_2 -> token_1
Markov Chain Text (Implementation)
• Store probabilities for each word based on order; use transition
matrix rather than dictionary.
Alternative caption: ”Although the Markov-chain text model is still rudimentary, it recently gave
me “Massachusetts Institute of America”. Although I have to admit it sounds prestigious.
Applications – Autocomplete on Phones
• Most text messaging applications use a Markov Chain model to
predict what word you wish to type next, based on the last word.
Applications – Subreddit generation
• https://www.reddit.com/r/
SubredditSimulator/comme
nts/3g9ioz/what_is_rsubre
dditsimulator/
• Fully-automated subreddit
that generates random
submissions and comments
using Markov chains
Applications – Subreddit generation
• Explanation of the Markov Chain on Reddit:
• “Basically, you feed in a bunch of sentences, and even though it has no
understanding of the meaning of the text, it picks up on patterns like "word A
is often followed by word B". “
• “Then when you want to generate a new sentence, it "walks" its way through
from the start of a sentence to the end of one, picking sequences of words
that it knows are valid based on that initial analysis. “
• “So generally short sequences of words in the generated sentences will make
sense, but often not the whole thing.”
Markov Chain Text (Experiment)
• Writing a sonnet using Markov chain generated using Billy Shakespeare’s extant sonnets:
5 1
4 0.8
3 0.6
2 0.4
1 0.2
0 0
0 10000 20000 30000 40000 50000 0 10000 20000 30000 40000 50000
C++ Python C++ Python
• Applications
• Ecology
• Telecommunications
• Queueing theory
• Chemical kinetics
0 1 2 3 4 5 …
𝜇1 𝜇2 𝜇3 𝜇4 𝜇5
0 1 2 3 4 5 …
• Poisson process
𝜆 𝜆 𝜆 𝜆 𝜆
0 1 2 3 4 5 …
Population dynamics: Birth-death process
𝑃𝑛 𝑡 + ℎ = 𝑃𝑛 𝑡 1 − 𝜆𝑛 ℎ − 𝜇𝑛 ℎ + 𝑃𝑛−1 𝑡 𝜆𝑛−1 ℎ + 𝑃𝑛+1 𝑡 𝜇𝑛+1 ℎ + 𝑜(ℎ)
𝑃𝑛 𝑡 + ℎ − 𝑃𝑛 𝑡
= − 𝜆𝑛 + 𝜇𝑛 𝑃𝑛 𝑡 + 𝜆𝑛−1 𝑃𝑛−1 𝑡 + 𝜇𝑛+1 𝑃𝑛+1 𝑡
ℎ
Population
Transition rate matrix:
−𝜆0 𝜆0
𝜇1 −(𝜆1 + 𝜇1 ) 𝜆1 n+1
𝜇2 −(𝜆2 + 𝜇2 ) 𝜆3 ⋯ n
𝜇3 −(𝜆3 + 𝜇3 ) 𝜆4 n-1
𝜇4 −(𝜆4 + 𝜇5 )
time
t t+h
Population dynamics: Birth-death process
• Applications
• Predict extinction time
• Chemistry:
• radioactive transformations: birth process
• New atom decay: death process
• Queueing theory
• M/M/1 model
• M/M/k model
Applications: Adaptive Cruise Control
• Some cruise-control on vehicles use Markov chains to calculate speed
• Operating environment is stochastic
• Road grade on route can also be seen as stochastic
Applications: Adaptive Cruise Control
• Vehicle speed and following distance can be optimized for best-on-
average fuel economy and optimized for travel time
• More in Optimization and Optimal Control in Automotive Systems
Open Problem
Open Problem: Cutoffs
• Stationary distribution: vector that represents the proportion of time
a given Markov chain occupies each state, given enough time.
• Mixing time: number of steps required for a chain for the distance
from stationary to be small
Open Problem: Cutoffs
• Build a Markov chain via a random walk over the symmetric group on
n symbols
http://pages.uoregon.edu/dlevin/MARKOV/markovmixing.pdf
References
Page, L., Brin, S., Motwani, R., & Winograd, T. (1999). The PageRank citation ranking: Bringing order to the web. Stanford Info
Lab.
snap.stanford.edu
https://en.wikipedia.org/wiki/PageRank
https://math.dartmouth.edu/~pw/math100w13/li.pdf
https://en.wikipedia.org/wiki/Population_process
https://en.wikipedia.org/wiki/Birth%E2%80%93death_process
https://en.wikipedia.org/wiki/Queueing_theory
https://en.wikipedia.org/wiki/Andrey_Markov
http://setosa.io/ev/markov-chains/
http://www.suaybarslan.com/birthdeathprocess4datamodelling.pdf
Hayes, B. (2013). First Links in the Markov Chain. American Scientist, 101(2), 92.
Harald Waschl, Ilya Kolmanovsky, Maarten Steinbuch, Luigi del Re. Optimization and Optimal Control in Automative Systems.
https://books.google.com/books?id=ebS5BQAAQBAJ&lpg=PA148&dq=how%20does%20cruise%20control%20work%20mark
ov%20chains&pg=PA142#v=onepage&q=how%20does%20cruise%20control%20work%20markov%20chains&f=false
Discussion
Questions Revisited
• What is the matrix A given the following graph when calculating
PageRank according to the simple definition of PageRank?
1 3
0 2
• What is the first name of the man credited for Markov Chains?