IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 33, NO.
5, SEPTEMBER 2018
Reinforcement Learning Approach for Optimal
Distributed Energy Management in a Microgrid
Elham Foruzan , Graduate Student Member, IEEE, Leen-Kiat Soh , Member, IEEE,
and Sohrab Asgarpoor, Senior Member, IEEE
Abstract—In this paper, a multiagent-based model is used to
study distributed energy management in a microgrid (MG). The
suppliers and consumers of electricity are modeled as autonomous
agents, capable of making local decisions in order to maximize their
own profit in a multiagent environment. For every supplier, a lack
of information about customers and other suppliers creates chal-
lenges to optimal decision making in order to maximize its return.
Similarly, customers face difficulty in scheduling their energy con-
sumption without any information about suppliers and electricity
prices. Additionally, there are several uncertainties involved in the
nature of MGs due to variability in renewable generation output
power and continuous fluctuation of customers’ consumption. In
order to prevail over these challenges, a reinforcement learning al-
gorithm was developed to allow generation resources, distributed
storages, and customers to develop optimal strategies for energy
management and load scheduling without prior information about
each other and the MG system. Case studies are provided to show
how the overall performance of all entities converges as an emer-
gent behavior to the Nash equilibrium, benefiting all agents.
Index Terms—Microgrid, reinforcement learning, distributed
control, renewable generation.
I. INTRODUCTION
IGH penetration of distributed energy resources (DERs),
H small-scale, renewable and non-renewable generators,
and distributed energy storage into power systems, has revo-
lutionized the systems from traditionally passive to active sys-
tems in which small-scale generation resources are located in
the vicinity of the customers load [1]. In this regard, effective co-
ordinated operation of these small scale DERs and loads inside
a power system is achievable through microgrid (MG) systems
[2]. A MG is a low-voltage, autonomous small-scale power
grid that interconnects loads and DERs. It has defined electrical
boundaries and acts as a single controllable entity with respect
Manuscript received October 13, 2017; revised February 6, 2018; accepted
March 28, 2018. Date of publication April 5, 2018; date of current version
August 22, 2018. Paper no. TPWRS-01548-2017. (Corresponding author:
Elham Foruzan.)
E. Foruzan is with the Department of Electrical and Computer Engineer-
ing, and Computer Science and Engineering, University of Nebraska—Lincoln,
Lincoln, NE 68588 USA (e-mail:,elham.foruzan@huskers.unl.edu).
L.-K. Soh is with the Department of Computer Science and Engineering,
University of Nebraska—Lincoln, Lincoln, NE 68588 USA (e-mail:, lksoh
@cse.unl.edu).
S. Asgarpoor is with the Department of Electrical and Computer Engi-
neering, University of Nebraska—Lincoln, Lincoln, NE 68588 USA (e-mail:,
sasgarpoor1@unl.edu).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TPWRS.2018.2823641
0885-8950 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
5749
to the rest of a power system (or a maingrid). Microgrids also
offer a range of technical benefits, including improving system
reliability and local energy delivery [3]. Also, MGs have eco-
nomic benefits as they can provide additional sources of capital
investment for small-scale DERs, which in turn leads to rapid
extension of MGs. Consequently, it reduces the customer de-
pendency on maingrid electricity, which has high production
and transmission investment.
Notwithstanding the benefits of MGs, the operation of multi-
ple small-scale DERs introduces considerable complexity into
a system, which may not be addressed using the existing tradi-
tional controller mechanisms. Specifically, DERs and customers
may have different owners, and therefore they tend to change
their operations to maximize their profits without considering
other DERs operations and overall system performance. Thus,
the actions or decisions of these individual competing entities
may violate the robustness in an MG, for example, in terms of
maintaining load-generation balance or cause inefficient use of
DERs resources, which may lead to transfer excessive power
from maingrid [3]. Additionally, renewable DERs have variable
output powers. Thus, serving customers’ loads with these re-
sources without a controller may cause system instability [4].
Thus, systematic energy and load management is critical to pro-
vide a market model that enables competitive participation of
DERs and customers within a MG to optimize entities utility
while reducing MG dependency on maingrid, and also miti-
gating the variability of the system so as to match the often-
unpredictable energy supply provided by renewable resources.
Recently, a promising approach based on distributed energy
and load management has emerged that can be achieved using
local monitoring and information exchange in MGs [5], [6]. In-
deed, such solutions have gained popularity, especially in the
form of multiagent systems (MAS), for energy management
and load scheduling in MGs [7]–[11]. MAS-based solutions, as
a distributed control approach, provide an effective solution for
MGs with different DER owners and customers, since all enti-
ties can be modeled as autonomous agents with a certain degree
of intelligence and the capability to improve their performance.
MAS-based solutions have been used to simulate and ana-
lyze emergent behavior of systems as a result of autonomous
decision makers called agents. Several studies have presented
a comprehensive market model, based on MAS to control en-
ergy dispatch in MGs [12]–[15]. In [12], [13], customers and
DER agents exchange energy and maximize their utility in a
market via many-to-many negotiations on electricity price and
 5750
amount between m buyers and n sellers at every time frame,
requiring a long computational time of O(mn) for a real-time
market. On the other hand, for many-to-many negotiation, auc-
tions mechanisms seem more intuitively appropriate [16], as all
agents submit their bids and electricity amount to the market
and auctioneer clear the market based on auction rules. Indeed,
auction-based market has proven to be a suitable mechanism
for MGs to coordinate renewable and conventional generation
resources and match supply and demand [14], [15]. Likewise, in
competitive auctions market, finding optimal strategy for agents
to improve their utility is essential.
Reinforcement learning (RL) has been shown to achieve ex-
cellent performance in optimizing agents’ utilities in MAS-
based modeling of power systems. Indeed, some studies have
used RL to improve agents’ decision making and increase their
expected utility in an energy market [17]–[19]. For example,
in [19] Q-learning, an RL algorithm, was used to learn a cost-
effective day-ahead consumption plan, for charging a fleet of
electric vehicles. Authors in [17] used Q-learning to optimize
the power suppliers’ profit function in the auction based frame-
work. Other researchers used RL to improve agent decision
making in an MG system [20]–[22]. For example, strategic bid-
ding for load-shedding in an MG using RL is discussed in [20].
In [21], RL was used to minimize customer agents’ energy costs
by using strategic battery scheduling. In [22], the authors used
an RL for both customers and a service provider to minimize
their expected cost in an MG. In later work, however, the au-
thors did not consider direct energy management for DERs,
especially those having stochastic energy output, as they as-
sumed that MG buys power through the power service provider.
In most of the reported advances in MAS-based MGs, there is
a lack of modeling of both DERs and customers as individual
agents in an MG. Such modeling is critical to understand the
interactive relationships between the supply and demand sides
with their competing needs and perhaps, conflicting goals, in
order to adapt to others in the environment and decrease the
MG dependency on maingrid electricity in addition to focusing
on single-entity optimal strategies. Additionally, most of these
researches fail to address stochastic behavior of both renewable
DERs and customers’ load in an MG, while it is more realistic
to consider all agents’ variability in models that aim to maintain
load-generation balance.
Therefore, we present a MAS-based distributed energy and
load management approach for DERs, including battery storage
resources, and customers by adopting reinforcement learning in
an MG energy market. In this approach, each agent-on either the
supply or the demand side- is designed to optimize its utility in
an hourly MG auction-based market via reinforcement learning,
allowing it to adapt its behavior to other agents present in a com-
petitive and stochastic MG energy market. The reinforcement
learning capability, based on the model-free Q-learning algo-
rithm [23], allows agents to find the optimal policy to maximize
their utility without direct communication with other entities in
the system. To address the challenge of including agents’ vari-
ability, we used random models for these agents to consider the
variability of renewable DERs and customers’ consumption in
our model.
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 33, NO. 5, SEPTEMBER 2018
In terms of contributions, we developed an RL-based en-
ergy and load management model which can be conducted in a
distributed manner at each DER and customer. Our proposed
model provides a framework for MG energy and load manage-
ment taking into account the variability of stochastic entities in
the MG. In this framework, both customers and suppliers are
rational autonomous agents that can adapt to each other’s behav-
iors. We further propose a set of performance metrics to evaluate
the effectiveness of the distributed RL method on DERs, cus-
tomers, and the overall MG system, including MG dependency
on maingrid for supplying its local customers. We investigate
several configurations to investigate the operation of the pro-
posed model and validate its effectiveness for all participating
agents in the system.
II. ENVIRONMENT AND MARKET DESIGN
In this section, first a multiagent environment is described,
where it includes a set of physical entities: DERs, customers,
and the maingrid. We couple each DER as well as each cus-
tomer with the notion of agent-hood, such that both DERs and
customers have full control over their own decisions of power
generation/consumption and profits/costs. Then, auction design,
to match supply and demand, is presented.
A. Environment Design
The MG contains NG agents comprising energy buyers, sell-
ers, auctioneer, and maingrid. Each customer is an energy buyer
and decides on the electricity amount it needs to purchase. The
renewable resources (PVs and wind generators) and diesel gen-
erators are energy sellers that provide their bids and amount of
electricity to the energy market. A storage system can be either a
buyer or a seller; at each time slot, each storage system can only
assume one role. Furthermore, we also assume an auction-based
market design for our environment where there is an auctioneer
agent. The auctioneer agent is an independent entity without
any physical property. Its duty is to control the auction process
between the buyers and sellers according to the auction rules
and clear market price for each time period in order to define
the dispatch points of resources and market clearing price (PC).
At the same time, because of variable energy outputs of the re-
newable resources and variable load demands, MGs often relies
on the maingrid agent to absorb any excess power and subsidize
any deficits, resulting in (1):
Pcu stom er = Pw in d + Pdiesel + Pstor ag e + PP V + Pg r id
(1)
where Pcu stom er is all customers’ energy consumption and
Pw in d , Pdiesel , Pstor ag e , PP V , and Pg r id are wind, diesel,
storage, PV and maingrid energy generation, respectively. The
maingrid buys and sells electricity at the different rates of
Pmt ain ,b and Pmt ain ,s , respectively; therefore, we represent a
tuple Pmt ain = (Pmt ain ,s , Pmt ain ,b ) as a maingrid price at time
t. Also, maingrid’s price varies during the day. There are three
different price rates depending on the time of use (TOU) for four
time periods. The maingrid buys at 5–15 C| /kWh and sells from
three uniform distributions 13–16, 23–26, and 40–42 C | /kWh
 FORUZAN et al.: REINFORCEMENT LEARNING APPROACH FOR OPTIMAL DISTRIBUTED ENERGY MANAGEMENT IN A MICROGRID 5751

for time intervals 11 p.m. to 8 a.m.; 8 a.m. to 12 and 6 p.m. TABLE I

PROPOSED ENERGY MANAGEMENT
to 11 p.m.; and 12 to 6 p.m., respectively. [24]. The desired
emergent behavior of the MAS system is for its energy resources
to supply electricity to meet the demands of the customers with
as little dependency on the maingrid energy as possible while
optimizing the expected rewards of all agents.

B. Market Design
Our MG market is auction-based, and it consists of an auction-
eer agent, customers (or buyer agents), and energy resources (or
seller agents). Every seller agent submits its amount of electric-
ity and corresponding bid at each time interval t. Buyer agents
submit their energy demand to the auctioneer agent as well.
The auctioneer clears the market by finding the intersection of
demand curve from loads and the ascending supply curve from
generation resources in a uniform price auction. If a seller’ sub-
mitted energy does not accepted in the local MG auction, the
seller can sell its energy directly to the maingrid with its buying
rate.
Each agent i aims to maximize its profit by selling energy A. Markov Game
to the MG, Eࢩ . Based on the market rules, agent i will sell its
surplus energy E to the maingrid to salvage cost. For each agent The Markov game has been shown to be effective in mod-
i in the system, the value of agent’s profit for selling electricity eling stochastic multiagent systems with non-cooperative na-
is calculated based on (2). ture [23], [28]. Here, a Markov game is used to model the
stochastic and adaptive behavior of all agents in a sequence of
finite iterations. At each iteration, an agent senses the current
P rof iti = P rof itM
i
G
(E ∗ ) + P rof itm
i
ain g r id
(E) state and decides on an action. After that, it receives an imme-
− Cost (E ∗ + E) (2) diate reward that depends on the current state and that chosen
action, and the resulting next state. The transition to the next
state is stochastic. Mathematically, a Markov game is defined as
where P rof itMi
G
(E ∗ ) is the profit of selling Eࢩ amount of en- a tuple (n, S, A, r, P ) where n is the set of agents; S is a finite
ergy in the MG’s market and P rof itm i
ain g r id
(E) is the profit of state space; A = ×Ai=1,..,n is a joint action space of n play-
selling E amount of energy to the maingrid. The function Cost(.) ers; r is the immediate reward; and P : S × A × S → [0, 1] is a
calculates the operational cost of DER agents. The DRE’s cost transition function over the set S [26]. In a stochastic game, the
function is considered to be quadratic functions of their opera- objective of each agent is to maximize the discounted sum of
tional point with constant coefficients. rewards with a discount factor, β ∈ [0, 1). Let πi be the strategy
of player i and it tries to maximize its value at each state s ∈ S,
which is defined by Vi (s)
III. MODEL DESIGN

t=∞
In the MG, the agent’s utility depends not only on the actions Vi (s, π1 , π2 , . . . , πn ) = β t ri (π1 , π2 , . . . , πn ) (3)
it takes, but also on the actions of other agents as well. Like- t=0
wise, agents do not have stationary and deterministic policy due
Considering that n agents are present in the game that can
to the stochastic environment and their ability to change their
adapt to each other, the Nash equilibrium is often considered
policy over time to improve their expected profit. Therefore, the
as a joint policy such that no self-interested agent can improve
Markov game is used to represent non-stationary interactions of
its expected discounted reward by deviating to a different strat-
agents inside the MG [25]. Then, the RL, a model free algorithm,
egy in stochastic games [25], [28]. The Nash equilibrium in
is discussed to solve distributed optimization for each agent so
a stochastic game with n player is defined as a joint policy
that each agent adapts to others [26], [27]. Next, random model
π = {πi=1,..,n } in which for every agent i ∈ n policy πi is the
is discussed to represent the stochastic behavior of agents in the
best response to the others’. In other words, if all agents except
MG. Thus, agents would consider their variability in their au-
agent i plays with joint policy π−i agents i gets the maximum
tonomous decision and would converge to a Nash equilibrium
possible reward by playing policy πi [29].
using RL, where each agent’s best response is to every other
agent’s best response, resulting in a stable solution. Then, pro-
B. Agent Learning
posed system evaluation indices for analyzing the distributed
energy management are presented. Summary of the proposed Because of the stochastic environment and actors adapting to
algorithm is given in Table I. each other, Q-learning, a RL algorithm, is used to power each

Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
 5752
TABLE II
AGENTS STATES, ACTIONS, AND IMMEDIATE REWARD, THE P C t E ∗ t E t , ARE MARKET CLEARING PRICE, SOLD ENERGY TO THE MG MARKET, AND SOLD
ENERGY TO THE MAINGRID AT TIME SLOT t, RESPECTIVELY
agent’s reasoning to discover its optimal policy with respect
to other agents’ actions. Q-learning is essentially a Markov
decision process, and follows directly from the properties of a
Markov game [30].
Considering that S is a discrete set of environment states,
and A is a discrete set of actions, an agent experiences every
state s ∈ S and possible sequences of actions a ∈ A through
the finite learning horizon. After performing an action, a at state
s, the agent makes a transition to the new state s ∈ S. The agent
observes the immediate reward rt for performing an action a
in state s and also observes the new state s , then updates the
value of the state s. Given the information of < s, a, s , rt >,
the updating rule of Q-learning is:
 
Qt+1 (s, a) = (1 − α) Qt (s, a) + α rt + β V ∗ (s , a) (4)
where α ∈ [0, 1) is the learning rate, β ∈ [0, 1) is a discounted
factor, and V ∗ (s ) is the optimal value of state s . The optimal
value of a state at each iteration is obtained by computing the
maximum value that an agent could obtain by carrying out an
action in that state [23].
Considering the Markov game tuple, (n, S, A, r, P ), how
the model is mapped into the MG problem is illustrated.
In the system, there are five types of agents, namely: wind
resources, photovoltaic (PV) or solar resources, diesel genera-
tors, battery storage, and customers. There are total of five self-
interested agents capable of learning, i.e., updating the value
of each state based on its actions and immediate rewards re-
ceived from the environment. The discrete and finite set of ac-
tions A = {a1 , a2 , . . . , aA } for the distributed energy resource
(DER) agents (wind, PV, diesel, and battery) includes their bid
and the amount of power they want to produce; while customers’
actions are to decide about their consumption given each state.
The immediate reward function r is defined as a function of
profit/expenses received from the market. Also, for each agent
type, except battery storage, the state s ∈ S at time slot t is a
tuple of the current time slot t and the maingrid price, while
different values for the maingrid price were selected randomly
from the maingrid’s price discrete distribution set during a learn-
ing process. The state of battery storage has an extra element,
which represents the battery storage’s state of charge (SOC). Ad-
ditionally, the well-known Q-learning approach did not require
prior information about the transition function P. The DERs do
not know the information of other DERs. However, during the
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 33, NO. 5, SEPTEMBER 2018
learning procedure, each DER updates its strategy based on the
performance of different actions in various states without ex-
plicit modeling of the environment. Interactions of agents and
the environment during the learning processor lead the system
to asymptotically converge to the Nash equilibrium. A detailed
summary of all agents’ actions, states, and rewards are shown
in Table II.
C. Random Models (RMs)
To represent the stochastic characteristics of customers’ con-
sumption and the renewable generators’ output power, random
models are used. In this paper, random values for customer con-
sumption are generated based on the actual data corresponding
to the average values of the feature under study. In other words,
the random sampling of customers load consumption during
a day are modelled using an exponentially distributed random
variable with a mean value of daily profile given in [31]. Addi-
tionally, the output power of wind and PV resources are selected
randomly from Weibull and Beta probability distribution func-
tions, respectively, based on [32]. For each renewable genera-
tion resource, each simulation time frame was divided into four
subintervals, and several samples from respective probability
distribution are taken for each sub-interval. The average value
of these samples has been used to realize the value of generated
power for each subinterval of that time frame. Each agent offers
the lowest value of four subinterval to the market to make sure
that it can track its offer.
D. Agent Design
Our proposed energy and load management consists of five
types of agents, namely: wind resources, photovoltaic or solar
resources, diesel generators, battery storages, and customers.
Each agent type behaves as a self-interested learning agent that
updates the value of possible actions at each state based on some
rewards received from the environment. Details of components
in the MG are described below.
1) Customer Agent: Each customer i has a set of aggregated
loads, ADi , which is defined as the total amount of electricity
that the customer i should use for its appliances. Also, in our
environment, a customer can practice load management, which
means curtailing some amount of consumption when electricity
price is high in order to reduce cost [24]. The customer agent i
 FORUZAN et al.: REINFORCEMENT LEARNING APPROACH FOR OPTIMAL DISTRIBUTED ENERGY MANAGEMENT IN A MICROGRID 5753

tries to minimize its cost function during the day by choosing TABLE III
INPUT PARAMETERS OF CASE STUDIES
energy function Fit ∈ Fi , where Fi is the set of energy functions
for customer i. By choosing Fit , customer i satisfies Fit (ADit ) =
Elit amount of its total aggregated demand at time slot t. De-
spite the fact that the customer i obtains profit by curtailing flex-
ible portions of its aggregated load, ADit − Elit , it also loses
some utilities from not using this amount of its aggregated load.
The function gi (.) models the utility dissatisfaction for agent B. Fairness Factor (FF)
i. This function was set to gi (ADit − Elit ) = K|ADit − Elit |, This metric is calculated by dividing the average value of
where K is a dissatisfaction coefficient. Different customers AEP for all DERs by the average value of electricity cost of
may have different K value; for example, customers who value customers (AEC), for all customers in the system. This metric
electricity more should choose a higher value for coefficient discovers how the designed system optimizes both customers
factor K. and DERs expected reward.
The total expense of the customer agent i at time slot t is N
defined as: AEPi /N
F F = Ni=1 (8)
  L
i=1 AECi /NL
Expensei = P C t . Elit + gi ADit − Elit . (5)
where N and NL are the number of DERs and costumers. The
2) Generation: Two renewable energy resources i.e., wind value of AECi for customer i is calculated by dividing the daily
and PV energy resources are modeled in the environment [29]. energy expenses of customer over its daily power consumption.
Also, diesel generators are considered as a non-renewable gen- 24 t
t=1 Expensei
eration with deterministic output power. AECi =  24 t
3) Battery Storage Systems: A battery storage system can be t=1 ADi
charged and discharged by buying and selling electricity during FF metric increases with either increasing average profit of
operation and changes its energy state at each time slot. We DERs or decreasing the average cost of customers or both.
denote the battery storage state of charge (SOC) at the time Therefore, the system that optimizes both customers and DERs
t with SOC t . The operation constraints of the storage agent expected reward has the highest fairness factor and it is fair to
require that SOC remains bounded with some limit. all agents.
SOCmin < SOC < SOCmax
C. Electricity Purchased from Maingrid (EPM)
where SOCmin and SOCmax are maximum and minimum bat- This metric calculates daily energy that a MG purchases from
tery SOC and are set to 0.8 and 0.2 in our model. Let xti denote the maingrid, normalized by the daily average MG’s load.
a charging or discharging energy during time slot t for the ith 24 t
storage unit, its status at time t + 1 is: t=1 (P rof it − Cost)M ain g r id
EP M =  (9)
N L 24 t
SOC t+1 = αSOC t + xt (6) i=1 t=1 ADi /NL

A positive value of xt indicates battery charging, and its neg-
ative value means battery discharging. For each battery storage,
the net profit is calculated as a difference of its profit for selling
electricity and its cost for buying electricity.
IV. EVALUATION METRICS
Three metrics are designed to evaluate the desired emergent
behavior of the proposed multiagent model. Among these three
metrics, the first two metrices, are proposed to evaluate the per-
formance of each agent and estimate how the proposed method
assists agents to optimize their expected reward. The last metric
evaluates the performance of the MG and how agents’ learning
contribute to reduce MG dependency on maingrid.
A. Average Energy Generation Profit of a DER (AEP)
This metric is calculated by dividing the daily energy profit
of a DER i, P rof iti , over its daily power generated.
24
P rof itti
AEPi = 24t=1 ∗t , (7)
t=1 (Ei + Ei )
t
where the (P rof it − Cost)tM ain g r id is the difference between
maingrid profit from selling electricity to the MG and its cost
from buying electricity from MG. The lower value for EPM
indicates that the maingrid has a smaller profit due to smaller
amount of electricity transfer to the MG.
V. RESULTS
In this section, first, sensitivity analysis is performed to fur-
ther analyze the role of learning parameter i.e., learning rate (α)
and discounted rate (β). Then, the performance of the proposed
learning-based solution approach is investigated for four differ-
ent configurations of agents. The model was implemented in the
Repast Simphony based on the Java programming languages. In
the set of experiments, one day consists of 24 time-slots each
of which lasts for one hour. Here, the value of dissatisfaction
coefficient, K, was set to 10 for all customers. Tables III shows
the parameters for our experiments.
A. Sensitivity Analysis with Respect to Learning Parameters
In the process of learning, the learning parameters, i.e., learn-
ing rate and discounted rate are internal variables for every agent

Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
 5754
Fig. 1. Storage average profit as a function of learning parameters.
TABLE IV
INPUT PARAMETERS OF CASE STUDIES
that impact on their optimal strategy. An agent uses its expe-
rience to improve its estimate, blending new information into
its prior experience according to a learning rate. So, agents that
are greedy about new experience and prefer to choose the maxi-
mum immediate reward right away have high learning rate [17].
On the other hand, discount factor controls how much effect
future rewards have on the optimal decisions, thus small values
of discount factor emphasize near-term gain and larger values
give significant weight to later rewards. Thus, agents choose
high discounted rate if the expected future reward is valuable
for them. To further study the variation in the performance of
DERs learning with different learning parameters, sensitivity
analysis is performed for all DERs.
Fig. 1 shows the value of storage average net profit in three-
dimension coordination system defined by learning rate, dis-
counted rate, and storage average net profit. Such representation
of storage average net profit for different values of α and β gives
a comprehensive illustration of the preference for storage strat-
egy. In this experiment, the learning parameters for other agents
set to 0.5 and 0.5. As can be seen in Fig. 1, the high value of dis-
counted rate and low value of learning rate have the maximum
profit for a storage agent. This is because storage agents gain
from trading electricity, as opposed to producing electricity, so
they should be able to buy electricity from microgrid when the
price is low and sell it back when the electricity price is high.
Therefore, it is very important for the storage agents to have a
future prospective vision. High value of β and low α suits this
requirement appropriately by allowing them to explore more
state-actions to find the optimal policy.
Same study was performed for diesel and renewable genera-
tion resources. The learning parameters that leads to maximum
average profit were presented in Table IV. Same as battery stor-
age, diesel agent has its maximum profit with the high value
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 33, NO. 5, SEPTEMBER 2018
Fig. 2. Wind agent daily profit (upper), and generated energy (lower) in the
four configurations.
of α and low β. In contrast to the storage and diesel, renew-
able resources prefer lower value of β and higher value of α.
This result shows that renewable generation resources follow
a policy to maximize their profit by choosing the maximum
immediate reward. This is because renewable generations have
unpredictable energy supply, with a negligible operational cost.
Thus, they prefer to trade all of their available energy and max-
imize their average profit by following a greedy policy. The
learning parameters of agents have been set based on Table IV.
B. Simulation Results and Analysis
In this section, four configurations are presented to show how
RL affects the performance of agents’ behavior and their ex-
pected profits/ expenses in the MG environment. These four con-
figurations are: (1) system without learning capability (Configu-
ration NoL), (2) system with learning capability for DERs only,
and fixed consumptions for customers (Configuration GenL),
(3) system with learning capability for customers only (Con-
figuration LoadL), and (4) system with learning capability for
all agents (Configuration AllL). Figs. 2 to 7 show the results
of these four configurations after they have learned their best
response.
1) Configuration NoL: System without Learning: In this
configuration, agents do not have learning capability. Figs. 2
to 7, configuration NoL, illustrate the agents’ actions in the MG
without using their learning feature. As can be seen, the ex-
pected cost of agents is quite varied and system did not reach a
convergence.
Configuration GenL: System with Learning Capability for
DERs and Fixed Consumptions: In the set of simulations, agents
choose a value to bid in the market from a discrete set of 9
prices, distributed uniformly between 7 to 45 C| /kWh. Also,
the discrete amount of electricity that diesel agents can choose
has 4 values, including their full capacity (100 %), medium
capacity (a number between 70% to 80% of their capacity),
low capacity (a number between 30% to 40% of their capacity),
and zero capacity. Therefore, a diesel agent’s action set has
9 × 4 = 36 different actions. Battery storages can buy and sell
 FORUZAN et al.: REINFORCEMENT LEARNING APPROACH FOR OPTIMAL DISTRIBUTED ENERGY MANAGEMENT IN A MICROGRID
Fig. 3. Solar agent daily profit (upper), and generated energy (lower) in the
four configurations.
Fig. 4. Diesel agent daily net profit (upper), and generated energy (lower) in
the four configurations.
electricity; they submit one of the four values to sell energy, or
negative values to buy energy. Thus, a battery agent’s action set
has 9 × 7 = 63 values. Renewable resources sell their available
capacity; thus, each of their action sets has 9 values.
A total of four optimality equations as (4) were formu-
lated, which correspond to the aforementioned four DER agents.
Based on the microgrid system agents’ design, each DER agent
updates its own Q-table through interactions—explorations—
with the environment in the form of rewards—i.e., selling and
buying prices of electricity resulting in profits—and thus learns
a best response strategy allowing the environment to reach a
steady state or the Nash equilibrium. The decision made by ev-
ery DER agent in the environment impacts the other agents’
gains in the system. Yet, DER agents eventually converge to a
Nash equilibrium point in less than 9600 time-slot (hours) or
400 “days” of learning. The total computational time for a PC
with 16 Gb RAM was 20 sec. After an agent has learned the
optimal policy, it chooses its actions based on its optimal policy
derived from their Q-tables as can be seen in Figs. 1 to 6, GenL.
The renewable resources, Figs. 2 to 3, need to decide about
their offered hourly bids. If they offer higher bids than the hourly
market clearing price, they lose out on selling energy in the MG
market. Besides, submitting a lower bid in the market, if it was
accepted, leads to lower profit. Therefore, they choose a policy to
increase the expected profit in the long run. Fig. 5 shows the net
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
5755
Fig. 5. Storage agent daily net profit (upper), and generated energy (lower) in
the four configurations.
Fig. 6. Customer agent daily expenses (upper), and consumed electricity
(lower) in the four configurations.
profit and daily net sold electricity of the storage agent. As can
be seen, the storage agent gains a higher expected net profit with
a lower amount of electricity trading with the system. The daily
expenses for a customer agent and its daily consumption are
shown in Fig. 6. Customer agents did not make any decisions in
this configuration since they satisfy all of their loads. However,
as a result of strategic actions of other agents in the multiagent
environment, their expected expenses were reduced. Fig. 7 also
verifies the performance of the agent design system. The ratio-
nal optimal decisions of agents with the presence of learning
capability led to a Nash equilibrium in this configuration. Con-
sequently, electricity interactions between the maingrid and MG
decreases 14%.
2) Configuration LoadL: System with Learning Capability
for Customers: In this configuration, only customers have the
capability of learning an effective load scheduling policy to
adapt to what it observes and receives as rewards from the
environment over time. Fig. 7, Configuration LoadL, shows that
the maingrid, on average, buys electricity (as indicated by the
negative amount in transmitted power) as a result of excess
energy produced in the MG, but the maingrid still has a positive
profit. This is due to the maingrid making more profit by selling
electricity to the MG rather than buying excess power from the
 5756
Fig. 7. Maingrid daily profit (upper), and transmitted electricity from maingrid
to the MG (lower) in the four configurations.
Fig. 8. The patterns of customers’ load curtailment during 24 hours with
changing dissatisfaction coefficient, K.
MG, on average. This advantage for the maingrid appears to
be due to the DER agents not following any particular policy
to maximize their profits, leading to random bids and losses in
their transactions with the MG.
To better illustrate the effectiveness of learning for customer
agents, the results of daily curtailment coefficient for 6 differ-
ent dissatisfaction coefficient values, K, were plotted in Fig. 8.
The customers select their load curtailment percentage from the
range {0, 0.05, ..., 0.65, 0.70}. We can observe that as K in-
creases, a customer agent decreases its curtailment, as K can
be seen as a penalty factor. This thus creates a trade-off for
customers between (1) saving money because of forgoing some
portion of their loads and (2) their dissatisfaction from having
curtailed electricity. As the sequence of figures in Fig. 8 shows,
customer agents learn to maintain the high curtailment during
on-peak periods and reduce their curtailment during off-peak
periods. These observations confirm that the RL mechanism
enables customers to adapt and manage their individual con-
sumption amounts when the utility from using electricity is less
than saving money from curtailing loads.
3) Configuration AllL: System with Learning Capability for
All Agents: All of the agents in the system have the reinforce-
ment capability in this configuration. Figs. 2 to 7, Configuration
AllL, show the daily generated/consumed electricity and elec-
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 33, NO. 5, SEPTEMBER 2018
Fig. 9. Comparison of average energy generation profit (AEP) for all DER
agents in the four configurations.
tricity profit/expenses for agents inside the MG. The RL, as in
the Configuration GenL, enables DERs to learn best response
strategies and increase their expected profit. Additionally, the
same as Configurations LoadL, customers can schedule their
consumption and curtail some portion of their flexible loads, if
it is preferable. Fig. 7 shows that the load reduction significantly
reduces the maingrid profit to 874 cents.
C. Comparison of NoL, GenL, LoadL, and AllL
In this section, the four configurations NoL through AllL are
evaluated using the performance metrics introduced earlier.
Fig. 9 shows values of Average Energy Generation Profit
(AEP) for four different types of DERs in four configurations
NoL, GenL, LoadL, and AllL. The AEP values are the highest
overall in Configuration GenL for all DERs. The DERs also
roughly maintain their high values in Configuration AllL, though
lower than those in GenL. This is due to the impact of load-
curtailing customers in AllL, where the customer agents are able
to learn and adapt to decisions by the DERs to maximize their
own profits. In Configurations NoL and LoadL, DER agents do
not learn to maximize their profits and that leads to relatively
lower daily profits. The AEP values for DER agents have their
lowest possible value in configuration LoadL because DERs,
without learning capabilities are disadvantaged against customer
agents that have learned their best response policies, resulting
in reduced profits for DERs.
We see from Fig. 9 that the AEP value for a battery storage
is negative in configurations NoL and LoadL. A negative value
of AEP for storage indicates that this agent is buying more
expensive electricity than what it is selling back to the grid,
in these configurations. This is due to the storage agent not
taking into account future prices for buying and selling. On the
other hand, in Configurations GenL and AllL, the storage agent
learns to improve its profits from buying electricity when the
electricity price is low, and selling it back to the market when
the electricity price is high, leading to much higher daily profits
than those from the NoL and LoadL configurations. Therefore,
the impact -in terms of average energy profit- that storage agents
can gain from RL is much more substantial than other types of
DER agents in the environment.
The values of FF for the four configurations are provided in
Fig. 10. Configuration AllL has the highest FF value (∼1.81).
 FORUZAN et al.: REINFORCEMENT LEARNING APPROACH FOR OPTIMAL DISTRIBUTED ENERGY MANAGEMENT IN A MICROGRID
Fig. 10. Comparison of the fairness factor (FF) and electricity purchased from
maingrid (EPM) in the four configurations.
In Configuration AllL, DER agents learn to effectively improve
their profit while customers simultaneously learn to effectively
reduce their average expenses, which leads to maximum value
for FF among all configurations.
Now, let’s look at EPM. The EPM index has its lowest value in
Configuration AllL (∼11.78). Two main reasons collaborate to
lower the maingrid profit. First, DERs can effectively offer their
bids to the market; and, therefore, instead of selling to the main-
grid, DERs sell electricity to the customers. Second, customer
demand is lowered such that the MG has a smaller electricity
shortfall and does not need to import as much electricity from
the maingrid. As a result, the MG ends up having fewer interac-
tions with the maingrid, and becoming more independent of the
maingrid. In summary, Configuration AllL has the least depen-
dency on the maingrid which is an important emergent behavior
stemming from our design modeling. Additionally, Configura-
tion AllL fairly allows all agent types to learn their best response
to each other and improve their performance. Hence, this con-
figuration is more suited for MGs having different owners for
DERs and customers since all agents individually are capable of
effectively learning to improve their performance in the MG and
adapt to each other. In short, the MG has the best performance
when both customers and DER agents are capable of learning
in terms of fairness and independence from the maingrid.
VI. CONCLUSION
A distributed multiagent approach for adaptive control of en-
ergy management in the MG framework was proposed and pre-
sented in this paper. The agents modeled in the MG include load
customers, energy supply resources such as DERs, and storage
agents. In this system, DERs and customers were modeled as au-
tonomous self-interested agents who can learn the best response
policy to increase their expected rewards in microgrid market
by adapting to the other agents in the MG. By adopting the
multiagent learning structure for supplier and customer agents,
supplier agents found the optimal solution for their bidding
problem at each time-slot t with learning strategies to effectively
improve their expected profits, while, customer agents learned
to make rational decisions about their consumption based on
current prices and their actual demands to achieve best response
strategies. This would not have been possible when the different
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
5757
types of agents–i.e., DERs and customers–been modeled and
investigated individually without considering that their coun-
terparts also capable of modeling and adapting in the same
environment. Furthermore, the MG’s electricity dependency on
the maingrid reduced to the lowest value as an emergent behav-
ior of the proposed model. Moreover, with stochastic model of
agents, the applied Q-learning, allowed all agents to infuse the
variability of customer and DER agents in their optimal policy.
The distributed RL learning approach offers a scalable solution.
Therefore, if an agent changes its behavior, i.e., the value of
electricity for a customer, it can update its model locally, based
on new information and learn its new best response without a
need of excessive communication to a central controller or other
agents. Thus, our proposed model provides a framework for MG
energy and load management taking into account the variability
of stochastic entities in the MG.
REFERENCES
[1] J. P. Lopes, N. Hatziargyriou, J. Mutale, P. Djapic, and N. Jenkins, “In-
tegrating distributed generation into electric power systems: A review of
drivers, challenges and opportunities,” Electric Power Syst. Res., vol. 77,
no. 9, pp. 1189–1203, 2007.
[2] P. Basak, S. Chowdhury, S. H. nee Dey, and S. Chowdhury, “A literature
review on integration of distributed energy resources in the perspective of
control, protection and stability of microgrid,” Renewable Sustain Energy
Rev., vol. 16, no. 8, pp. 5545–5556, 2012.
[3] M. Microgrids-Benefits, Barriers and Suggested Policy Initiatives for the
Commonwealth of Massachusetts. Burlington, MA, USA: KEMA Inc.,
2014.
[4] H. Farhangi, “The path of the smart grid,” IEEE Power Energy Mag.,
vol. 8, no. 1, pp. 18–28, Jan./Feb. 2010.
[5] H. Cai, G. Hu, F. L. Lewis, and A. Davoudi, “A distributed feedforward
approach to cooperative control of ac microgrids,” IEEE Trans. Power
Syst., vol. 31, no. 5, pp. 4057–4067, Sep. 2016.
[6] A. Pantoja and N. Quijano, “A population dynamics approach for the
dispatch of distributed generators,” IEEE Trans. Ind. Electron., vol. 58,
no. 10, pp. 4559–4567, Oct. 2011.
[7] V. Crespi, A. Galstyan, and K. Lerman, “Top-down vs bottom-up method-
ologies in multi-agent system design,” Auton. Robots, vol. 24, no. 3,
pp. 303–313, 2008.
[8] A. Belgana, B. P. Rimal, and M. Maier, “Open energy market strategies
in microgrids: A stackelberg game approach based on a hybrid multi-
objective evolutionary algorithm,” IEEE Trans. Smart Grid, vol. 6, no. 3,
pp. 1243–1252, May 2015.
[9] E. Mojica-Nava, C. Barreto, and N. Quijano, “Population games methods
for distributed control of microgrids,” IEEE Trans. Smart Grid, vol. 6,
no. 6, pp. 2586–2595, Nov. 2015.
[10] P. Shamsi, H. Xie, A. Longe, and J.-Y. Joo, “Economic dispatch for an
agent-based community microgrid,” IEEE Trans. Smart Grid, vol. 7, no. 5,
pp. 2317–2324, Sep. 2016.
[11] Y. Zhang, N. Gatsis, and G. B. Giannakis, “Robust energy management
for microgrids with high-penetration renewables,” IEEE Trans. Sustain.
Energy, vol. 4, no. 4, pp. 944–953, Oct. 2013.
[12] Y. F. Eddy, H. B. Gooi, and S. X. Chen, “Multi-agent system for distributed
management of microgrids,” IEEE Trans. Power Syst., vol. 30, no. 1,
pp. 24–34, Jan. 2015.
[13] R. Duan and G. Deconinck, “Multi-agent coordination in market environ-
ment for future electricity infrastructure based on microgrids,” in Proc.
IEEE Int. Conf. Syst., Man Cybern., 2009, pp. 3959–3964.
[14] D. Divényi and A. M. Dán, “Agent-based modeling of distributed genera-
tion in power system control,” IEEE Trans. Sustain. Energy, vol. 4, no. 4,
pp. 886–893, Oct. 2013.
[15] M. H. Cintuglu, H. Martin, and O. A. Mohammed, “Real-time implemen-
tation of multiagent-based game theory reverse auction model for micro-
grid market operation,” IEEE Trans. Smart Grid, vol. 6, no. 2, pp. 1064–
1072, Mar. 2015.
[16] B. An, N. Gatti, and V. Lesser, “Alternating-offers bargaining in one-
to-many and many-to-many settings,” Ann. Math. Artif. Intell., vol. 77,
no. 1–2, pp. 67–103, 2016.
 5758
[17] M. Rahimiyan and H. R. Mashhadi, “An adaptive-learning algorithm de-
veloped for agent-based computational modeling of electricity market,”
IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 40, no. 5, pp. 547–556,
Sep. 2010.
[18] D. Li and S. K. Jayaweera, “Distributed smart-home decision-making in
a hierarchical interactive smart grid architecture,” IEEE Trans. Parallel
Distrib. Syst., vol. 26, no. 1, pp. 75–84, Jan. 2015.
[19] S. Vandael, B. Claessens, D. Ernst, T. Holvoet, and G. Deconinck, “Re-
inforcement learning of heuristic fleet charging in a day-ahead electricity
market,” IEEE Trans. Smart Grid, vol. 6, no. 4, pp. 1795–1805, Jul. 2015.
[20] Y. Lim and H.-M. Kim, “Strategic bidding using reinforcement learning
for load shedding in microgrids,” Comput. Electr. Eng., vol. 40, no. 5,
pp. 1439–1446, 2014.
[21] E. Kuznetsova, Y.-F. Li, C. Ruiz, E. Zio, G. Ault, and K. Bell, “Rein-
forcement learning for microgrid energy management,” Energy, vol. 59,
pp. 133–146, 2013.
[22] B.-G. Kim, Y. Zhang, M. van der Schaar, and J.-W. Lee, “Dynamic pricing
and energy consumption scheduling with reinforcement learning,” IEEE
Trans. Smart Grid, vol. 7, no. 5, pp. 2187–2198, Sep. 2016.
[23] C. J. Watkins and P. Dayan, “Q-learning,” Mach. Learn., vol. 8, no. 3–4,
pp. 279–292, 1992.
[24] “Online,” [Online]. Available: https://www.sce.com/
[25] M. L. Littman, “Markov games as a framework for multi-agent reinforce-
ment learning,” in Proc. 11th Int. Conf. Mach. Learn., 1994, vol. 157,
pp. 157–163.
[26] Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An
Introduction. Cambridge, MA, USA: MIT Press, vol. 1, no. 1, 1998.
[27] L. P. Kaelbling, L. Pack, M. L. Littman, and A, W. Moore, “Reinforcement
learning: A survey,” J. Artif. Intell. Res., vol. 4, pp. 237–285, 1996.
[28] J. Hu and M. P. Wellman, “Nash Q-learning for general-sum stochas-
tic games,” J. Mach. Learn. Res., vol. 4, no. Nov, pp. 1039–1069,
2003.
[29] X. Wang and T. Sandholm, “Reinforcement learning to play an optimal
Nash equilibrium in team Markov games,” in Proc. 15th Int. Conf. Neural
Inf. Process. Syst., 2002 , pp. 1571–1578.
[30] M. L. Littman, “Markov games as a framework for multi-agent re-
inforcement learning,” in Proc. 11th Int. Conf. Mach. Learn., 1994,
vol. 157, pp. 157–163.
[31] E. Foruzan, S. Asgarpoor, and J. M. Bradley, “Hybrid system modeling
and supervisory control of a microgrid,” in Proc. North Amer. Power
Symp., 2016, pp. 1–6.
[32] Y. Atwa, E. El-Saadany, M. Salama, and R. Seethapathy, “Optimal re-
newable resources mix for distribution system energy loss minimization,”
IEEE Trans. Power Syst., vol. 25, no. 1, pp. 360–370, Feb. 2010.
Authorized licensed use limited to: Motilal Nehru National Institute of Technology. Downloaded on October 02,2020 at 16:42:07 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON POWER SYSTEMS, VOL. 33, NO. 5, SEPTEMBER 2018
Elham Foruzan (GS’17) received the B.Sc. degree
from the Ferdowsi University of Mashhad, Mashhad,
Iran, in 2008, and the M.Sc. degrees in electrical
and computer engineering and computer science and
engineering from the University of Tehran, Tehran,
Iran, and University of Nebraska—Lincoln, Lincoln,
NE, USA, in 2010 and 2017, respectively, where
she is currently working toward the Ph.D. degree.
Her research interests include smart grid, multi-agent
systems, microgrid, machine learning, and cyber-
physical systems.
Leen-Kiat Soh (M’98) received the B.S. degree (with
highest distinction) and the M.S. and Ph.D. degrees
(with honors) in electrical engineering from the Uni-
versity of Kansas, Lawrence, KS, USA. He is cur-
rently a Professor with the Department of Computer
Science and Engineering, University of Nebraska–
Lincoln, Lincoln, NE, USA. He has authored more
than 180 peer-reviewed journal and conference pub-
lications. His research interests include multiagent
systems and intelligent agents, computer-aided edu-
cation systems, computer science education, and in-
telligent image analysis.
He has applied his multiagent systems research to smart grids and distributed
GIS. He is a member of ACM and AAAI.
Sohrab Asgarpoor (S’80–M’86–SM’91) received
the B.S., M.S., and Ph.D. degrees in electrical en-
gineering from Texas A&M University, College Sta-
tion, TX, USA.
From 1986 to 1989, he was with ABB Net-
work Management Inc., as a Lead Engineer. Since
September 1989, he has been with the University of
Nebraska–Lincoln, Lincoln, NE, USA, where he is
currently the Interim Associate Dean of Undergrad-
uate Programs with the College of Engineering, and
a Professor with the Department of Electrical and
Computer Engineering. His research interests include reliability evaluation,
maintenance optimization, and advanced computer applications in security and
optimization of power systems.