Distributed Scheduling in Smart Grid Communications with

Dynamic Power Demands and Intermittent Renewable Energy

Resources

Shengrong Bu , F. Richard Yu , Peter X. Liu , and Peng Zhang

Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, Canada
Department of Electrical and Computer Engineering, University of Connecticut, Storrs, CT, USA
Email: shengrbu@sce.carleton.ca; richard yu@carleton.ca; xpliu@sce.carleton.ca; peng@engr.uconn.edu

Abstract - Concerns about climate change, rising fossil fuel

prices and energy security have spurred interest in renewable energy generation and smart grid. Due to the dynamic
power demands and intermittent renewable energy resources,
optimal scheduling of power generation systems is important
to minimize cost and green house gas emissions, and to
avoid blackouts in smart grid. In this paper, we propose a
distributed stochastic scheduling scheme in smart grid communications with dynamic power demands and intermittent
renewable energy resources. Due to meteorological instability
and complex system dynamics, hidden Markov models are used
in modeling renewable energy resources. We formulate the
stochastic scheduling problem as a partially observable Markov
decision process multi-armed bandit problem. A value iteration
algorithm is used to solve the above problem. Simulation results
are presented to show the effectiveness of the proposed scheme.
Keywords - Scheduling, renewable energy, smart grid, Stochastic systems
I. I NTRODUCTION
As concerns about climate change, rising fossil fuel prices
and energy security increase, there is growing interest around
the world in obtaining energy from natural resources, such as
wind, solar and hydro, which can be naturally replenished or
renewed [1].
The traditional electricity grid was designed to transmit and
distribute electricity generated by large conventional power
plants. The electricity flow mainly takes place in one direction
from the centralized plants to consumers. In contrast to large
power plants, renewable energy plants have less capacity, and
are installed in a more distributed manner at different locations
(i.e., where generation conditions are most favorable). The
integration of distributed renewable energy generators has a
great impact on the operation of the grid and calls for new grid
infrastructure. Indeed, it is a main driver to develop the smart
grid for infrastructure modernization [2].
A smart grid is an intelligent electricity network, which
can reduce peaks in power usage, encourage manufacturers to
produce smart appliances to reduce energy use, and sense
and prevent power blackouts by isolating disturbances in the
grid [3]. A major challenge in the smart grid is how to use
information and communication technologies, such as advanced

metering, and bi-directional communication to save energy,

reduce cost and increase reliability.
Since most renewable energy sources are intermittent in nature, it is a challenging task to integrate a significant portion of
renewable energy resources into the power grid infrastructure.
Scheduling of power generation systems is an important issue
in smart grid communications to effectively coordinate energy
demand and generation in order to minimize cost and green
house gas emissions, and to avoid blackouts in the event that the
nature phenomena necessary to generate energy do not occur
[4].
There is some work related to scheduling in smart grid.
The authors of [5] use augmented Lagrangian relaxation (LR)
and dynamic programming (DP) methods for the short-term
resource scheduling of integrated thermal and photovoltaicbattery generation. A genetic algorithm solution is presented
in [6] to solve the hydrothermal coordination problem. Risklimiting dispatch is proposed in [7], where the scheduling with
known demand and uncertain supply is treated as a multi-stage
decision problem with recourse. The authors of [8] study model
predictive dispatch of wind generation, where the future wind
generation is described by an ARMA model.
Although some work has been done about scheduling in
smart grid, most previous work uses the current observed power
output from renewable energy resources to make scheduling decisions for subsequent energy generation. Moreover, managing
the power supply has been the focus in previous work, and
energy demand is considered to be known but uncontrollable.
However, these assumptions may not be realistic given the
time-varying nature of renewable energy resources [9] and
uncertainties on the demand side [10].
Finite-state Markov models have been widely accepted in the
literature as an effective approach to characterize the correlation
structure of the renewable energy outputs [9, 11]. Considering
Markov models may enable substantial performance improvement over the schemes with memoryless models. Moreover, it
is generally assumed that the renewable energy system state
is known perfectly in previous work. However, meteorological
instability and complex system dynamics (e.g., wind turbine
malfunction) make it difficult to fully observe the system state.
Therefore, hidden Markov models (HMMs) are more efficient
in modeling renewable energy systems [12].
In this paper, we propose a distributed stochastic scheduling

978-1-61284-955-3/11/$26.00 2011 IEEE

scheme in smart grid communications with dynamic power

demand loads and intermittent renewable energy resources.
Some distinct features of the proposed scheme are as follows:
HMMs are used for the renewable energy resources. To the best
of our knowledge, this is the first paper to consider distributed
scheduling in smart grid communications with HMM models.
Costs, reliability and pollutant emissions are considered in the
scheduling scheme. We model the operational cost of the utility
grid as a convex function of instantaneous power consumption,
so as to reflect the fact that each additional unit of power needed
to serve demand become more expensive as the total power
demand increases. We formulate the scheduling problem as a
partially observable Markov decision process (POMDP) multiarmed bandit problem [13, 14]. The proposed scheme is fully
distributed and scalable.
The rest of the paper is organized as follows. Section II
introduces the system models. Section III describes the solution
to the scheduling problem. Section IV presents the simulation
results. Finally, we conclude this study in Section V.
II. S YSTEM M ODELS
We consider a smart grid system including solar panels, wind
turbines and traditional fossil fuel power generators. As solar
radiation is always absent at night time, wind turbines are
necessary in order to improve the performance of the system.
Fossil fuel power generators are also used in order to mitigate
or even cancel out the fluctuations.
Assume that there are Gw wind turbines, Gs solar panels and
Gf fossil fuel power generators in the system, for G units total.
These units are integrated together using smart grid communications in order to serve the dynamic loads. In this system, it is
critical to optimally schedule these units, considering demand
loads, costs, reliability and pollutant emissions.
A. Renewable Energy Models
The models for renewable energy produced from wind turbine systems and solar panels are presented in this subsection.
The energy produced by a wind turbine per unit time when it
is available is as follows [15]:

0, v < vcutin 

3
F v 3 , v
Pmax /F
cutin v
Pw =
3

P
,
P
/F
<
v
< vcutout
max

max
0, vcutout v.
In the above equation, vcutin and vcutout are wind speeds at
which the turbine starts to generate power, and the turbine
must be turned away from the wind to protect the blades,
respectively. Pmax represents the maximal power the wind
turbine can produce. F = 1/2R2 , where is the efficiency
of the turbine and R is the radius of the turbine. represents
the air density.
For each photovoltaic solar cell, the amount of generated
current Ic depends on the intensity of solar radiation . Given
by a diode model, the relationship between its voltage Vc and
current Ic can be described as follows [16]:
Ic () = (D0 + D1 Tc ) I0 (.)

(Vc Ic Rs )
,
Rsh

(1)

where D0 and D1 are constant coefficients, Tc is the temperature of the solar cell, I0 (.) is a function of diode saturated
current, and Rs and Rsh are series and shunt resistances,
respectively.
B. System States
(n)

Let the state of a generator n, n {1, 2, . . . , N }, be sk

at time slot k. Each state represents the power supply level of
generator n. The power supply of each generator can be divided
into L discrete levels. The power supply state space S includes
all the power supply states {s1 , . . . , sL }. The state of renewable
energy generator n, evolves according to a L-state Markov
(n)
chain with state transition probability
matrix J , which is

(n)
(n)
(n)
described as follows: J
= sij
, where sij =
i,jS


(n)
(n)
P sk+1 = j|sk = i .
The observation of selected generator ns power level state is
(n)
denoted as ysk at time slot k, which belongs to a finite set Mn
indexed by m(n) = 1, . . . , |Mn |, and |Mn | denotes the number
of possible observations of the generator ns power generation
level state. If generator n is picked at time k, and the system
(n)
state sk equals to i, the probability of observation m will be
obtained from generator n is denoted as:




(n)
(n)
(n)
bi ak = n, ysk = m = P ysk = m|sk = i, ak = n ,
(2)
where i S, and m Mn . Define the observation matrix as:


(n)
Bs(n) ysk = m = diag[b1 (ak , m), . . . , bL (ak , m)]. (3)
A state transition matrix U (n) and an observation probability
(n)
matrix Bd can be defined for the dynamic demand load. The
observation probability matrix of the hidden Markov chain can
be calculated based on the conditional probability function of
(n)
(n)
the actual system state dk and observation states ydk .
In the proposed scheme, the number of scheduled units, U , at
each time slot is determined by the renewable energy generation
situation and the total power demand loads. For simplicity
of the presentation, we assume that one generator (either a
renewable energy generator or fuel generator) will be chosen
at each time slot. Note that it is straightforward to generalize
the model to picking U generators (where 1 U N ) at each
time slot.
III. S OLUTION TO S TOCHASTIC U NIT S CHEDULING
P ROBLEM
It is critical to optimally schedule the generation units in
smart grid communications with renewable energy resources.
We formulate the above problem as a partially observed Markov
decision process (POMDP) multi-armed bandit problem [14],
which is a powerful stochastic control modeling framework and
has been successfully used in solving distributed scheduling
problems [13].
A. Information State Formulation
The decision on which generation unit is chosen should not
totally depend on the current observation values, since the
renewable energy generation units states are only partially

observable. According to [17], the above POMDP multi-armed

bandit problem can be re-expressed as a fully observable multiarmed bandit problem in terms of the information states, which
means optimal units can be chosen based on the information
states.
The information state of a unit refers to a probability distribution over the units states. The entire probability space (the
set of all possible probability distributions) is referred to as the
information space. For an arbitrary unit n, the information state
(n)
at time k, a new
at time k is denoted as k . If unit n is chosen

(n)
(n)
(n)
observation yk+1 (i. e. ys(k+1) , yd(k+1) ) is obtained at time
(n)

k + 1. Its information state at that time k+1 can be recursively

updated by the hidden Markov model state filter known as the
forward algorithm with the new observation [13].
For our proposed scheme, the optimal policy has an
indexable

rule [14], meaning the units Gittins indices
(n)
(n)
k
(n {1, . . . , N }) are used to choose the appro
priate unit at time k. Furthermore, the optimal policy at time
k is that the unit with the largest reward Gittins index at that
time should be selected.
B. Stochastic Unit Scheduling Process
To reduce the computational complexity of the proposed
scheme, the scheduling process can be divided into off-line
and on-line parts.
a). Off-line Computation of Gittins Index
As with any dynamic programming formulation, the computation of the Gittins index for each unit can be done off(n)
line. For an arbitrary unit n, a finite set of vectors k
at each iteration k are computed in advance based on the
following parameters: state transition probability matrix G(n) ,
observation state probability matrix B (n) , cost vector C (n) ,
(n)
initial information state 0 , horizon length H, and discount
factor . The cost vectors and discount factor are defined in
Section III-C.
b). Real-time Unit Selection over Horizon H
At time k, each unit stores the units current Gittins indices
into an N -dimensional vector. The real-time unit selection
includes the following steps:
1) Select the unit with the highest Gittins index at time k.
(n)
2) Get the new unit observation yk+1 at time k + 1.
3) Update the information state of the chosen unit using the
corresponding hidden Markov filter.


(n)
(n)
4) Compute the Gittins index H k+1 for the chosen
unit only.
5) Broadcast the new Gittins index to the other units.
6) On receiving the messages, the other units get the updated
Gittins index.
C. Cost Definition
In order to minimize costs and green house gas emissions,
and to avoid blackouts, we consider utility costs (UCs), customer interruption costs (CICs) and pollutant emission costs
(PECs) in our scheme. At time slot k, let unit n be selected
(a )
(i.e., ak+1 = n), based on the history information Yk k . The



(n)
instantaneous cost C xk , n incurred at time k is as follows
[1, 18]:






(n)
(n)
(n)
(n)
(n)
xk
xk
C xk , n
= U Ck
+ CICk



utility
(n)
+ P ECk

(n)
xk

interruption

(4)

pollution
(n)

(n)

(n)

where xk = (sk , dk ), which includes the state of unit n

and the demand state at time k.
(n)
Utility costs U Ck include the average costs associated
with the required investments and operations of the chosen
generator n during the time slot k, when a renewable energy
generator is chosen. If a fuel generator is chosen, the utility
cost for purchasing
 power at time k can be calculated as:
(n)
(n) (n)
sk , dk
= pk (pk ) where pk is the power
U Ck
purchased at time k, and (pk ) is the power price. If only
(n)
one fuel generator is chosen at time slot k, pk = dk . (.)
is an increasing, differentiable convex function, which reflects
that each unit of additional power needed to satisfy increasing
demand becomes more expensive to obtain and make available
to the consumers.
(n)
Customer Interruption Cost CICk is directly related to the
type of customers and the duration of interruptions (i.e. the
length of time slot k), which is defined as follows:

F (), > 0
(n)
CICk =
0, 0,
where equals to the power demand subtracting to the
power
Z generated from generator n at time slot k. F () =
z=1 fz (z ), where = 1 + + z . Function fz (.) is
the CIC in customer sector z, and Z is the number of customer
sectors.
With the increasing concerns about environmental protection,
stricter regulations on pollutant emissions have been introduced,
often including financial penalties associated with emissions
(e.g., carbon credits), as well as non-financial costs associated
with environmental damage, which is represented as financial
penalties as well in our model. There are no pollutant emissions
for renewable energy generation systems. The amount power
generators must pay depends on fuel consumptions pk , which
can be calculated as follows [1]:
(n)

P ECk

= + pk + p2k ,

(5)

where , , and are the coefficients approximating the

generator emission characteristics.
The total expected discounted cost over an infinite-time
horizon is given by:





(n)
k
C xk , n ,
(6)
J = E
k=0

where is the discounted factor, which models the fact future

cost is worth less than immediate cost, because the future is less
certain. The objective is to find the optimal stationary policy
= arg max J to minimize the cost in (6).

D. Value Iteration Algorithm for Computing Gittins Index

IV. S IMULATION R ESULTS AND D ISCUSSIONS

In this section, we use computer simulations to evaluate
the performance of the proposed scheme. We consider the
following simulation scenario. There is a wind turbine generator
(WTG), a photovoltaic (PV) system, and a conventional fossilfuel generation system (FFGS) in the smart grid. Each generator
includes two power supply level states {high, low}, and there
are two demand load states {peak, off-peak}, which means that
there are 4 states for each generator, i.e., {(high, peak), (high,
off-peak), (low, peak), (low, off-peak)}. In general, the utility
cost of wind turbines is lower than that of solar panels. The
utility cost of fossil fuel generators is the lowest.
Following [11, 19], the power supply level state transition
probability matrices of the WTG, PV and FFGS in our simulations aredefined as follows:



0.756 0.244
0.85 0.15
, J (2) =
,
J (1) =
0.15 0.85
 0.279 0.721
1.0 0.0
.
J (3) =
0.0 1.0
Power demand loads are determined by two Poisson processes,
one for peak hours and another one for off-peak hours, where
the inter-arrival times between two requests are exponentially
distributed with a mean of 1 hour and 12 hours, respectively.
Assume the demand in each request is 100 kWh/h. The power
demand load
matrix is defined as:
 state transition

0.70 0.30
(n)
=
.
U
0.40 0.60
Their observation matrics
 are defined
 as:

0.90 0.10
0.90 0.10
B (n) = Bs Bd =

,
0.10 0.90
0.10 0.90
where n = 1, 2, 3.
The price of electricity differs for peak rate 9.3 cents/kWh
and off-peak rate 4.4 cents/kWh [20] and that of solar power is 80.2 cents/kWh [21] in Ottawa, Canada.
The price of wind power is set to 20.5 cents/kWh.
The cost matrices are as follows in $/h: C (1) =
(20.5, 25.5, 890.5, 1.7), C (2) = (80.2, 85.2, 950.2, 6.7) and
C (3) = (309.3, 304.4, 1074.4, 25.4).
We use pomdp-solve, a program in C++ from [22], to compute the set of vectors H , and implemented the computation
of the Gittins indices in Matlab. All simulations are run on a
computer equipped with Window 7, Intel Core 2 Duo P8400
CPU (2.26Ghz), 4GB memory.
Simulations are performed to compare the total costs of the
proposed optimal scheme with and without considering the
dynamic power demand loads and an existing scheme that does
not consider (hidden) Markov models for renewable energy
resources. We also study how the pollutant emission costs affect
the total costs and CO2 emissions in the system. Each value is
the average result of 1000 simulations in the following figures.

10000
Existing scheme
Proposed scheme w/o considering the dynamic power demand loads
Proposed scheme considering the dynamic power demand loads

8000

Total Costs ($)

A value iteration algorithm can be used to solve the above

scheduling problem. An elementary coordinate transformation
is introduced to transform the value function with a parameterized retirement reward to the value function of a standard
(n)
POMDP. The near-optimal Gittins index H ( (n) ) for unit n
is given by the finite-dimensional representation in [13].

6000

4000

2000

Fig. 1.

10
Hours (h)

15

20

Total costs in different schemes.

Fig. 1 illustrates the total costs in dollars for the first

20 simulated hours. The initial state is (low, off-peak). The
results show that the proposed scheme has a lower cost than
the existing scheme. In the proposed scheme, considering the
dynamic power demand load leads to a lower cost than not
considering it. Since the system state of a renewable energy
resource is not fully observable due to meteorological instability
and complex system dynamics, a scheduler can make wrong
decisions when hidden Markov models are not used. In our
POMDP-based scheme, wrong decisions can be minimized by
optimally scheduling these units, thus decreasing the total cost
of the system. The figure also shows that these three schemes
perform similarly in the early stages, since the system is in a
(low, off-peak) state, where the scheduling does not make a
significant difference.
Different values of pollutant emission costs are also used
in the simulations to analyze the effects of pollutant emission
costs, since reducing pollutant emissions is the main motivation
for employing renewable energy generation. The results are illustrated in Fig. 2, which shows that the total costs increase with
the pollutant emission cost. When the pollutant emission cost
is $0/kWh, only conventional generation systems are scheduled
in the system. We can observe that the higher the pollutant
emission cost, the greater the chance that renewable energy
generators are scheduled. The levels of emissions resulting from
the system are also compared in our simulation. In our paper,
the CO2 emission is set with 0.6999 kg/kWh [23]. The results
are shown in Fig. 3. The figure shows that CO2 emissions
increase with the reduction of the pollutant emission costs.
V. C ONCLUSIONS AND F UTURE W ORK
In this paper, we have presented a distributed stochastic
scheduling scheme in smart grid communications. In the proposed scheme, the most suitable generation unit is dynamically
scheduled based on the intermittent renewable energy power
generation situations, dynamic power demand loads, utility
cost, reliability and pollution emissions of the generation units.
Hidden Markov models was used in modeling renewable energy

6000
Pollutant emission cost: $10/kWh
Pollutant emission cost: $1/kWh
Pollutant emission cost: $0/kWh

5000

Total Costs ($)

4000

3000

2000

1000

10
Hours (h)

Fig. 2.

15

20

Total costs in hours.

1400
Pollutant emission cost: $0/kWh
Pollutant emission cost: $1/kWh
Pollutant emission cost: $10/kWh

1200

CO2(kg)

1000

800

600

400

200

10
Hours (h)

Fig. 3.

15

20

Total costs in hours.

resources. We formulated the scheduling problem as a POMDP

multi-armed bandit problem, and its optimal policy can be
chosen using Gittins indices. Simulation results were presented
to show that the proposed scheme can decrease the cost and
green house gas emissions. Future work is in progress to
consider system reliability in the proposed framework.
R EFERENCES
[1] L. Wang and C. Singh, Multicriteria design of hybrid power generation
systems based on a modified particle swarm optimization algorithm,
IEEE Trans. Energy Conversion, vol. 24, pp. 163172, Mar. 2009.

[2] National Institute of Standards and Technology, US Dept. of Commerce,

NIST Framework and Roadmap for Smart Grid Interoperability Standards. Release 1.0 (draft), 2009.
[3] C. Feisst, D. Schlesinger, and W. Frye, Smart grid, the role of electricity
infrastructure in reducing greenhouse gas emissions, tech. rep., Cisco
Internet Business Solution Group, Oct. 2008.
[4] H. Siahkali and M. Vakilian, Stochastic unit commitment of wind farms
integrated in power system, Electric Power Systems Research, vol. 80,
no. 9, pp. 10061017, 2010.
[5] M. Marwali, H. Ma, S. Shahidehpour, and K. H. Abdul-Rahman, Short
term generation scheduling in photovoltaic-utility grid with battery storage, IEEE Trans. Power Syst., vol. 13, pp. 10571062, Aug. 1998.
[6] C. E. Zoumas, A. G. Bakirtzis, J. B. Theocharis, and V. Petridis, A
genetic algorithm solution approach to the hydrothermal coordination
problem, IEEE Trans. Power Syst., vol. 19, pp. 13561364, May 2004.
[7] J. Bialek, P. Varaiya, and F. Wu, Risk limiting dispatching smart grid,
in Talk at CNLS Smart Grid Seminar Series, 2009.
[8] L. Xie and M. Ilic, Model predictive dispatch in electric energy systems
with intermittent resources, in Proc. IEEE Intl Conf. on Systems, Man
and Cybernetics, pp. 4247, 2008.
[9] F. Y. Ettoumi, H. Sauvageot, and A. Adane, Statistical bivariate modelling of wind using first-order Markov chain and weibull distribution,
Renewable Energy, vol. 28, pp. 17871802, Sept. 2003.
[10] M. He, S. Murugesan, and J. Zhang, Multiple timescale dispatch and
scheduling for stochastic reliability in smart grids with wind generation
integration, submitted to IEEE INFOCOM2011.
[11] D. Niyato, E. Hossain, and A. Fallahi, Sleep and wakeup strategies in
solar-powered wireless sensor/mesh networks: Performance analysis and
optimization, IEEE Trans. on Mobile Comp., vol. 6, pp. 221 236, Feb.
2007.
[12] J. Mur-Amada and A. A. Bayod-Rujula, Wind power variability model
part II - probabilistic power flow, in Proc. 9th International Conference
Electrical Power Quality and Utilisation, Oct. 2007.
[13] V. Krishnamurthy and B. Wahlberg, Partially observed Markov decision
process multiarmed bandits - structural results, Math. of Oper. Res.,
vol. 34, pp. 287302, May 2009.
[14] P. Whittle, Multi-armed bandits and the Gittins index, J. R. Statist. Soc.
B, vol. 42, no. 2, pp. 143149, 1980.
[15] K. Brokish and J. Kirtley, Pitfalls of modeling wind power using Markov
chains, in Proc. IEEE/PES Power Systems Conf. and Exposition, (Seattle,
WA), Mar. 2009.
[16] M. Marwali, S. M. Shahidehpour, and M. Daneshdoost, Probabilistic
production costing for photovoltaics-utility systems with battery storage,
IEEE Trans. Energy Conversion, vol. 12, pp. 175180, June 1997.
[17] J.C.Gittins, Multiarmed Bandit Allocation Indices. Wiley, 1989.
[18] B. Bagen, Reliability and Cost/Worth Evaluation of Generating Systems
Utilizing Wind and Solar Energy. PhD thesis, University of Saskatchewan,
2005.
[19] A. D. Sahin and Z. Sen, First-order Markov chain approach to wind
speed modelling, Journal of Wind Engineering and Industrial Aerodynamics, vol. 89, pp. 263269, 2001.
[20] Hydro Ottawa. http://www.hydroottawa.com. [Online; accessed 05August-2010].
[21] Ontario Power Authority, Feed-In-Tariff program. http://fit.
powerauthority.on.ca. [Online; accessed 05-August-2010].
[22] A. R. Cassandra. http://www.cassandra.org/pomdp/code/index.shtml/.
[Online; accessed 05-August-2010].
[23] P. Arun, R. Banerjee, and S. Bandyopadhyay, Optimum sizing of batteryintegrated diesel generator for remote electrification through design-space
approach, Energy, vol. 33, pp. 11551168, 2008.