March, 2022 EE 6150: Stochastic Processes and Theory of Queues Instructor: TG Venkatesh

Renewal Reward Process Scribe: Rohan

Renewal Reward Process

A large number of probability models are special cases of the following model. Consider a renewal process {N (t), t ≥
0} having inter arrival times {Xn , n ≥ 1}, and suppose that each time a renewal occurs we receive a reward. We denote
by Rn the reward earned at the time of nth renewal. We shall assume that Rn , n ≥ 1, are independent and identically
distributed. However Rn may depend on Xn , the length of the nth renewal arrival.
N (t)
�
R(t) = Rn
n=1

then R(t) represents the total reward earned by the time t. Let

E[R] = E[Rn ], E[X] = E[Xn ]

Proposition: If E[R] < ∞ and E[X] < ∞, then

R(t) E[R]
(a) with probability 1, limt→∞ t = E[X]

E[R(t)] E[R]
(b) limt→∞ t = E[X]

Proof. We give the proof for (a). To prove this,

�N (t) �� �� �
N (t)
R(t) n=1 Rn n=1 Rn N (t)
= =
t t N (t) t

By strong law of large numbers we obtain

�N (t)
n=1 Rn
→ E[R] as t → ∞
N (t)
and by proposition
N (t) 1
→ as t → ∞
t E[X]
If we say that a cycle is completed every time a renewal occurs, then the above proposition states that the long run
average reward per unit time is equal to the expected reward earned during a cycle divided by the expected length of a
cycle.
Although we have supposed that the reward is earned at the time of a renewal, the result remains valid when the reward
is earned gradually throughout the renewal cycle.

Example: Throughput of CSMA/CD

Carrier sense multiple access with collision detection ( CSMA/CD ) is a popular medium access protocol for wireless
networks. Consider a network with large number of nodes contending for the channel. Time is virtually slotted into
mini slots. To avoid collision nodes keep sensing the channel and will transmit only at the end of an idle mini slot.
Each node has a capability to sense channel as well as detect the collision of its packet. Each node will sense the mini
slot and if the channel is found to be idle, it will transmit at the beginning of the next mini slot with probability ”p”.
The duration for transmission of a packet is integral multiple of the length of mini slots. Let σ, denote the length of
the mini slot. The transmission duration of a packet is a RV denoted as L in units of σ.
Following things can happen when nodes attempt transmission .
Stochastic Modelling T G Venkatesh
• If no node transmit in that slot, the slot becomes idle, and all nodes sense an idle slot and make an attempt at the
beginning of the next mini slot with probability ”p”.
• If two or more nodes transmit, a collision occurs. Nodes involved in the collision can detect collision and stop
transmission within that mini slot itself. This slot will be followed by one Idle slot for nodes to sense idleness
before attempting transmission again.
• If exactly one node transmit, then after an idle slot following the end of the transmission another attempt can
begin.
We can model CSMA/CD as a renewal reward process to calculate the throughput. Throughput is defined as the
expected number of packets successfully transmitted per unit time.
We can say that at the end of every idle mini slot, nodes can attempt transmission and these are the renewal points.
The duration between successive renewals is
• σ if no node transmits ( Idle case )
• 2σ if more than one node transmit ( collision case )

• E[L] + σ if exactly one node transmits ( success case ).

Let Ps and Pc be the probability of a successful transmission and collision to take place respectively.
Then expected duration of a renewal cycle is

σ(1 − Ps + Pc ) + 2σPc + (E[L] + σ)Ps (80)

The reward is the number of packets transmitted successfully in a renewal cycle. Expected reward is
1 * Ps + 0 * ( 1- Ps ) = Ps . Therefore throughput S is
Ps
S= (81)
σ(1 − Ps + Pc ) + 2σPc + (E[L] + σ)Ps

Example: A Car Buying Model

Suppose the lifetime of a car is a uniform random variable U (5, 15). The owner has a policy that he buys a new car
as soon as his old one breaks down or reaches the age of 10 years. Suppose that a new car costs 5000 units ( under an
exchange program ). It means the car should be in working condition when exchanged. Assume that it costs of 800
units to repair whenever the car breaks down. What is the long run average cost?
We say that a cycle is completed whenever the owner purchases a new car. Then the long run average cost is
E[Cost incurred during a cycle]
E[length of a cycle]
Let X be the lifetime of the car during some arbitrary cycle, then the cost incurred in that cycle is
�
5800, if X ≤ 10
cost =
5000, if X > 10

The expected cost incurred over a cycle is

� � � �
1 1
5000P {X > 10} + (5800)P {X ≤ 10} = 5000 + 5800
2 2
= 2500 + 2900 = 5400

97
Stochastic Modelling T G Venkatesh
The length of the cycle is a Random Variable defined as
�
X, if X ≤ 10
length =
10, if X > 10

The expected length of a cycle is

� 10 � ∞ � 10 � � � 15 � �
1 1
xfX (x)dx + 10fX (x)dx = x dx + 10 dx
0 10 5 10 10 10
2 �10
x �
= � +5
20 5
= 3.75 + 5 = 8.75

Average long run cost per unit time is 5400/8.75 = 617.14 units per year.

Example: Dispatching a train

Suppose that customers arrive at a train depot in accordance with a renewal process having a mean interarrival time
of µ. Whenever there are N customers waiting in the depot, a train leaves. If the depot incurs a cost at the rate of nc
dollars per unit time whenever there are n customers waiting, what is the average cost incurred by the depot?
If we say that a cycle is completed whenever a train leaves, then the preceding is a renewal reward process. The
expected length of a cycle is the expected time required for N customers to arrive, and since mean inter arrival time is
µ, this equals
E[length of a cycle] = N µ
If we let Tn denote the time between the nth and (n+1)st arrival in a cycle, then the expected cost of a cycle may be
expressed as
E[cost of a cycle] = E[cT1 + 2cT2 + ... + (N − 1)cTN −1 ]
Since E[Tn ] = µ, we get

cµN (N − 1)
E[cost of a cycle] = cµ[1 + 2 + .. + N − 1] =
2
Average cost incurred is
E[cost of a cycle] cµN (N − 1) c(N − 1)
= =
E[length of a cycle] 2N µ 2

Regenerative Process
Consider a stochastic process {X(t), t ≥ 0} with state space 0,1,2,..., having the property that there exist time points
at which the process (probabilistic-ally) restarts itself. That is, suppose that with probability 1, there exists at time T1
, such that continuation of the process beyond T1 is a probabilistic replica of the whole process starting at 0. Note that
this property implies the existence of further times T2 , T3 , .. having the same property as T1 . Such a stochastic process
is known as a regenerative process.
From the preceding, it follows that T1 , T2 , .. constitute the arrival times of a renewal process and we shall say that a
cycle is completed every time a renewal occurs.

Examples
• A renewal process is regenerative, and T1 represents the time of first renewal.
• A recurrent markov chain is regenerative, and T1 represents the time of the first transition into the initial state.

98
Stochastic Modelling T G Venkatesh
The length of the cycle is a Random Variable defined as
�
X, if X ≤ 10
length =
10, if X > 10

The expected length of a cycle is

� 10 � ∞ � 10 � � � 15 � �
1 1
xfX (x)dx + 10fX (x)dx = x dx + 10 dx
0 10 5 10 10 10
2 �10
x �
= � +5
20 5
= 3.75 + 5 = 8.75

Average long run cost per unit time is 5400/8.75 = 617.14 units per year.

Example: Dispatching a train

Suppose that customers arrive at a train depot in accordance with a renewal process having a mean interarrival time
of µ. Whenever there are N customers waiting in the depot, a train leaves. If the depot incurs a cost at the rate of nc
dollars per unit time whenever there are n customers waiting, what is the average cost incurred by the depot?
If we say that a cycle is completed whenever a train leaves, then the preceding is a renewal reward process. The
expected length of a cycle is the expected time required for N customers to arrive, and since mean inter arrival time is
µ, this equals
E[length of a cycle] = N µ
If we let Tn denote the time between the nth and (n+1)st arrival in a cycle, then the expected cost of a cycle may be
expressed as
E[cost of a cycle] = E[cT1 + 2cT2 + ... + (N − 1)cTN −1 ]
Since E[Tn ] = µ, we get

cµN (N − 1)
E[cost of a cycle] = cµ[1 + 2 + .. + N − 1] =
2
Average cost incurred is
E[cost of a cycle] cµN (N − 1) c(N − 1)
= =
E[length of a cycle] 2N µ 2

Regenerative Process
Consider a stochastic process {X(t), t ≥ 0} with state space 0,1,2,..., having the property that there exist time points
at which the process (probabilistic-ally) restarts itself. That is, suppose that with probability 1, there exists at time T1
, such that continuation of the process beyond T1 is a probabilistic replica of the whole process starting at 0. Note that
this property implies the existence of further times T2 , T3 , .. having the same property as T1 . Such a stochastic process
is known as a regenerative process.
From the preceding, it follows that T1 , T2 , .. constitute the arrival times of a renewal process and we shall say that a
cycle is completed every time a renewal occurs.

Examples
• A renewal process is regenerative, and T1 represents the time of first renewal.
• A recurrent markov chain is regenerative, and T1 represents the time of the first transition into the initial state.

98
Stochastic Modelling T G Venkatesh
We are interested in determining the long-run proportion of time that a regenerative process spends in the state j.
To obtain this quantity, let us imagine that we earn a reward at a rate 1per unit time when the process is in state j and
0 otherwise. That is I(s) represents the rate at which we earn at time s, then
�
1, if X(s) = j
I(s) =
0, if X(s) �= j
Total reward earned by time t is � t
I(s)ds
0
E[reward by time T1 ]
The average reward per unit time = E[T1 ]

Alternating Renewal Process

Another example of a regenerative process is the alternating renewal process, which contains a system that can be in
one of two states, on or off. Initially it is on, and it remains in on for a time Z1 ; it then goes off and remains off for a
time Y1 . It then goes on for Z2 and off for Y2 and so on.
We suppose that the random vectors (Yn , Zn ), n ≥ 1 are independent and identically distributed. That is both the
sequences {Zn } and {Yn } are independent but we allow Zn and Yn to be dependent.
Each time the process goes on, everything starts over again, but when it goes off, we allow the length of the off time
to depend on the previous on time.
Let E[Z] = E[Zn ] and E[Y ] = E[Yn ] be the mean lengths of on and off period.
We are concerned with Pon , the long term proportion of time that the system is on. Let
X n = Yn + Z n , n ≥ 1
At time X1 the process starts over again. That is, process renews after one complete cycle consisting of an on and off
interval. A renewal occurs whenever the cycle is completed.

E[Z]
Pon =
E[Y ] + E[Z]
E[on]
=
E[on] + E[of f ]
Pof f = 1 − Pon
E[of f ]
=
E[on] + E[of f ]

Example: A Production Process

One example of an alternating renewal process is a production process(a machine) that works for a time Z1 , then
breaks down and has to be repaired for a time Y1 and the process repeats. If we suppose that the machine is as good as
new after repair, then its an alternating renewal process.

Example: Insurance Company

The rate a certain insurance company charges its policy holders alternates between r1 and r0 . A new policy holder is
initially charged a rate of r1 per unit time. When a policy holder paying rate r1 has made no claims for the most recent
s time units, then the rate charged becomes r0 per unit time. The new rate remains at r0 until a claim is made, at which
time it reverts to r1
Suppose that a policy holder lives forever and makes claims at times chosen according to Poisson process with rate λ,
find

99
Stochastic Modelling T G Venkatesh
1. Pi , the proportion of time the policy holder pays at rate ri , i = 0, 1.
2. The long-run average amount paid per unit time.
Solution: If we say that the system is ’on’ when the policy holder pays at rate r1 and off when she pays at rate r0 ,
then this on-off system is an alternating renewal process with a new cycle starting each time a claim is made.
Let X be the time interval between two claims, then the on time in the cycle is smaller of s and X.

E[on time in cycle] = E[min(X, s)]

� s
= xλe−λx dx + se−λs
0
1
= (1 − e−λs )
λ

since E[X] = λ1 , we see that

E[On time]
P1 = = 1 − e−λs
E[X]
P0 = 1 − P1 = e−λs
The long run average amount paid per unit time is

r0 P0 + r1 P1 = r1 − (r1 − r0 )e−λs

100