Master equation, non markovian and markovian process

Sophia Vent
June 2023
Seminar on theoretical statistical Physics, SS23 University of Hidelberg
Overview

1. historical background

2. markovian process
Chapman Kolmogorv equation

3. Master equation

4. non markovian process

Nikajama zwanzig equation

5. Monte Carlo Markov Chain

1
historical background
Historical background

1906 1931 1940 1950 1970 2000

Andrey Markov Champan
pm Kálmán, Wiener research in
Markov property Kolmogorov stochastic - markovian dynamics
equation differential - embedding
equations

2
markovian process
Markovian process

Definition
A Markovian process is a sequence of
events or states where the future
depends only on the present state, not
the past. The state is memoryless

Figure 1: Random walk in 1D with 30 timesteps

3
Markov property

Definition
A sequence Xt of random variables has the Markov property if for all t, the
probability distribution for Xt+1 depends only on Xt , but not on Xt−1 , ..., X0 .

P[Xt+1 |X0 , X1 , ..., Xt ] = P[Xt+1 |Xt ].

4
Markov chain

Figure 2: three state system

5
Markov chain

Definition
 
PAA PAB PAC
 
Pij = Aij =  PBA PBB PBC 
PCA PCB PCC

Figure 3: three state system with transition

probabilities

6
Markov chain

Transitionprobability matrix
 
PAA PAB PAC
 
Aij =  PBA PBB PBC  |ij
PCA PCB PCC

PAA + PAB + PAC = 1

Figure 4: three state system with possible
transitions

7
Markov chain

Definition
 
PAA PAB PAC
(1)  
Pij =  PBA PBB PBC 
PCA PCB PCC
two step Probability:
(2)
PBA = PBA PAA + PBB PBA + PBC PCA
 
' ( PAA
(2)  
PBA = PBA PBB PBC ·  PBA 
Figure 5: two step probability PCA

8
Markov chain

Definition
 
PAA PAB PAC
(1)  
Pij =  PBA PBB PBC 
PCA PCB PCC
two step Probability:
(2)
PBA = PBA PAA + PBB PBA + PBC PCA
 
' ( PAA
(2)  
PBA = PBA PBB PBC ·  PBA 
Figure 5: two step probability PCA

→ what is the n step probability?

8
n-step probability matrix

Definition
   
PAA PAB PAC PAA PAB PAC
   
= A2ij = [ PBA PBB PBC  ·  PBA PBB PBC ]ij
(2)
Pij
PCA PCB PCC ) PCA PCB PCC

n step probability:
Pij = Anij
(n)

Chapman-Kolmogorov theorem
(n+m)
) (n) (m)
Pij = Pik · Pkj
k

9
Proof Chapman-Kolmogorov theorem

[?]
[?] [?]

10
Master equation
Master equation

System can be in one of mutually exclusive discrete states !x = (x1 , x2 , , , , xn )

Probability distribution function of finding the system in state or configuration !x at time
t is defined as
p(!xi , t) ≥ 0

recall the joint probability function

p(!xn , tn ;!xn−1 , tn−1 ; ...;!x2 , t2 ;!x1 , t1 )

11
Master eqaution

Consider the process from going from !x1 at t1 to !x3 at t3 in two steps
Recall Chapman- Kolmogorov theorem
(n+m)
) (n) (m)
Pij = Pik · Pkj
k

with
i = (!x1 , t1 ) j = (!x3 , t3 )

the conditional probability is given by:

)
p(x!3 , t3 |x!1 , t1 ) = p(x!3 , t3 |x!2 , t2 )p(x!2 , t2 |x!1 , t1 )
!x2

12
Derivation of the Master equation

Lets consider times t1 = t and t2 = t + τ , where τ is an infinitesimal short time interval.

The short time evolution equation is given by:
)
P(!xj , t + τ ) = p(!xj , t + τ |!x1 , t)P(!xi , t)
!xi

13
Derivation of the master equation

)
P(!xj , t + τ ) = p(!xj , t + τ |!x1 , t)P(!xi , t)
!xi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi
hi 14
master equation

Definition
xj , t) )
dP(! )
= P(!xi , t)wt (!xj ,!xi ) − P(!xj , t)wt (!xi ,!xj )
dt
!xi !xi

continuous state system

* ∞ * ∞
dP(!x" , t) "
= p(!x, t)wt (!x ,!x)dx − p(!x" , t)(wt (!x,!x" )dx
dt −∞ −∞

15
Gain and loss terms in the master equation

xj , t) )
dP(! )
= P(!xi , t)(wt (!xj ,!xi ) − P(!xj , t)wt (!xi ,!xj )
dt i i
+ ,- . + ,- .
Gain Loss

P(!xi , t)wt (!xj ,!xi ) is the probability to go from !xi to !xj per unit of time. → Probability flux

16
Gorini-Kossakowski-Sudarshan-Lindblad Master eqaution

Quantum Master equation : describing the time evolution of an open quantum system in
contact with its environment1
dρ )/ 1
0
= −i[H, ρ] + Lk ρL†k − {L†k Lk , ρ}
dt 2
k

1
G. Lindblad, ”On the generators of quantum dynamical semigroups,” Communications in Mathematical
Physics, vol. 48, no. 2, pp. 119–130, 1976.
17
Gorini-Kossakowski-Sudarshan-Lindblad Master eqaution

Quantum Master equation : describing the time evolution of an open quantum system in
contact with its environment2
 
dρ )  † 1 † 
= −i[H, ρ] + Lk ρLk − {Lk Lk , ρ}
dt + ,- . 2 + ,- .
k
Gain Loss

2
G. Lindblad, ”On the generators of quantum dynamical semigroups,” Communications in Mathematical
Physics, vol. 48, no. 2, pp. 119–130, 1976.
18
Gorini-Kossakowski-Sudarshan-Lindblad Master eqaution

Quantum Master equation : describing the time evolution of an open quantum system in
contact with its environment2
 
dρ )  † 1 † 
= −i[H, ρ] + Lk ρLk − {Lk Lk , ρ}
dt + ,- . 2 + ,- .
k
Gain Loss

• Gain: system gains coherence or population in certain states due to the interaction
with the environment tem
rm
• Loss: dissipative processes or decay channels
2
G. Lindblad, ”On the generators of quantum dynamical semigroups,” Communications in Mathematical
Physics, vol. 48, no. 2, pp. 119–130, 1976.
18
non markovian process
non markovian process

The Probability of being in a state Xt depends on the path or the history

P(Xt |Xt−1 , ...X1 ) $= P(Xt |Xt−1 )

19
non markovian process

The Probability of being in a state Xt depends on the path or the history

P(Xt |Xt−1 , ...X1 ) $= P(Xt |Xt−1 )

Embedding Transformation
Introduce augumented space Ht which captures the history. Define the augumented state
as (Xt , Ht ) = Yt
Key idea: choose a higher dimensional space Yt such that the markovian property holds

P(Yt |Yt−1 , Yt−2 , ...Y0 ) = P(Yt |Yt−1 )

19
Embeddding

20
System-bath problem

Quantum system coupled to a bath of harmonic oscillators

H = Hs ⊗ Ib + Is ⊗ Hb + Hsb

Quantum Liouville- von Neumann equation for the density operator ρ(t)

dρ(t)
= Lρ(t)
dt

21
Projection operator

Introduce a projection operator P and (1 − P) results in

ρ = Pρ + (1 − P)ρ

Pρ(t) = ρs (t)ρb (0) = trb (ρ(t))ρb (0)

22
Projection operator

Introduce a projection operator P and (1 − P) results in

ρ = Pρ + (1 − P)ρ

Pρ(t) = ρs (t)ρb (0) = trb (ρ(t))ρb (0) → projects onto state where the bath does not move

22
Nikajama Zwanzig equation

apply to Liouville von Neumann equation

d(Pρ(t))
= PLPρ + PL(1 − P)ρ
dt
d((1 − P)ρ(t))
= (1 − P)L(1 − P)ρ + (1 − P)LPρ
dt

solving the second equation results in

* t
(1 − P)ρ(t) = dτ e(1−P)Lτ LPρ(t − τ ) + e(1−P)Lt (1 − P)ρ(0)
0

23
Nikajama Zwanzig equation

substituting into the first equation

* t
d(Pρ(t))
= PLPρ(t) + PLe(1−P)Lt (1 − P)ρ(0) + dτ PLe(1−P)LP ρ(t − τ )
dt 0

24
Nikajama Zwanzig equation

substituting into the first equation

* t
d(Pρ(t))
= PLPρ(t) + PLe(1−P)Lt (1 − P)ρ(0) + dτ PLe(1−P)LP ρ(t − τ )
dt 0

resulting in
Nikajama-Zwanzig equation
* t
d(Pρ(t))
= PLPρ(t) + dτ K(τ )Pρ(t − τ )
dt 0

where K(τ ) = PLe(1−P) L is the memory kernel

24
modeling the memory kernel

problem: memory kernel quite complex

3
L. Banchi, E. Grant, A. Rocchetto, and S. Severini”Modelling non–markovian quantum processes with
recurrent neural networks”New Journal of Physics 20, 123030 (2018)
25
modeling the memory kernel

problem: memory kernel quite complex

solution: use recurrent neural networks (RNN) to model the memory kernel 3
RNN: use feedback connections to capture temporal dependencies and make predictions
based on the context of the entire sequence

3
L. Banchi, E. Grant, A. Rocchetto, and S. Severini”Modelling non–markovian quantum processes with
recurrent neural networks”New Journal of Physics 20, 123030 (2018)
25
RNN’s to model memory kernel

Consider two level system

∂ρ(t) 1
= −i[ωσz , ρ(t)] + γ(t)σ− ρ(t)σ+ − σ+ σ− ρ(t)
∂t 2

1. simulate quantum trajectories

2. train the RNN
3. reproduce the data

26
Results

Figure 6: Learning the Lindblad operator

describing the Markovian evolution of a two
level system

4
L. Banchi, E. Grant, A. Rocchetto, and S. Severini”Modelling non–markovian quantum processes with
recurrent neural networks”New Journal of Physics 20, 123030 (2018) 27
Results

Figure 6: Learning the Lindblad operator

Figure 7: Learning a non-Markovian master
describing the Markovian evolution of a two
equation
level system

4
4
L. Banchi, E. Grant, A. Rocchetto, and S. Severini”Modelling non–markovian quantum processes with
recurrent neural networks”New Journal of Physics 20, 123030 (2018) 27
Monte Carlo Markov Chain
Monte Carlo Markov Chain

Goal: Model a 2D lattice

1. Choose a lattice size L, a coupling constant J and a temperature T
2. initialize the lattice randomly with spin up ↑ or down ↓
3. flip the spin based on the Metropolis Hastings acceptance criterion
4. perform it numerous times

28
Metropolis Hastings acceptance criterion

• Calculate the energy of the lattice E1

• choose a lattice point (i,j), flip the spin and calculate E2 and ∆E = E1 − E2

acceptence criterion
∆E
pA = min(1, exp(− ))
kb T

29
2D lattice simulation

Choosing a temperature below Tc we expect the spins to align

(a) initital lattice (b) after 100 iterations (c) after 10000 iterations

Figure 8: Simulation of a 2D lattice using the MCMC and Metropolis Hastings algorith,

30
magnetization

simulate a lattice for different temperatures

Figure 9: Magnetization dependant on the temperature

31
summary

• markov property P[Xt+1 |X0 , X1 , ..., Xt ] = P[Xt+1 |Xt ]

dP(!
x ,t) 7 7
• master equation dtj = i p(!xi , t)(wt (!xj ,!xi ) − i p(!xj , t)wt (!xi ,!xj )
• non markovian process → embedding
• Nikajama- Zwanzig equation generalized the Master equation
• Introduced MCMC for simulations

32
References 2

X. Li.
Markovian embedding procedures for non-markovian stochastic schrödinger
equations.
Physics Letters A, 387:127036, 2021.
G. Lindblad.
On the generators of quantum dynamical semigroups.
Communications in Mathematical Physics, 48(2):119–130, 1976.
N. Van Kampen.
Stochastic Processes in Physics and Chemistry.
Elsevier, Amsterdam, third edition edition, 1992.

31
33
References 1

G. Haag.
Modeling with the Master Equation.
Springer, Berlin, Heidelberg, 2008.
A. Klenke.
Probability Theory: A Comprehensive Course.
Springer, London, 2013.
D. Layden, G. Mazzola, R. V. Mishmash, M. Motta, P. Wocjan, J.-S. Kim, and
S. Sheldon.
Quantum-enhanced Markov chain Monte Carlo.
3 2022.

33
34
Projection operator

)
P(ρ) = pk (Is ⊗ |k(|)ρ(Is ⊗ |q(|)
k,q

where |q( is an arbitrary basis of the environment, whereas |k( is the eigenbasis or
ρb = rhob (0)

35
References 2

G. Haag.
Modeling with the Master Equation.
Springer, Berlin, Heidelberg, 2008.
A. Klenke.
Probability Theory: A Comprehensive Course.
Springer, London, 2013.
D. Layden, G. Mazzola, R. V. Mishmash, M. Motta, P. Wocjan, J.-S. Kim, and
S. Sheldon.
Quantum-enhanced Markov chain Monte Carlo.
3 2022.

34