Multi Agent Reinforcement Learning based
Distributed Optimization of Solar Microgrid
R Leo R S Milton
SSN College of Engineering SSN College of Engineering
leor@ssn.edu.in miltonrs@ssn.edu.in
Abstract—We consider grid connected solar microgrid system
which contains a local consumers, solar photo voltaic (PV)
systems, load and battery. The consumer as an agent continuously
interacts with the environment and learns to take optimal actions
through a model-free Reinforcement Learning algorithm, namely
Q Learning. The aim of the agent is to optimally schedule the
battery to increase the utility of the battery and solar photo
voltaic system and thereby aims for the long term objective
of reducing the power consumption from grid. Multiple agents
sense the states of environment components and make collective
decisions about how to respond to randomness in load and
intermittent solar power by using a Multi agent reinforcement
algorithm, namely Coordinated Q Learning (CQ Learning).
Each agent learns to optimize individually and contribute to
global optimization. Grid power consumed when solar PV system
operates individually, by using Q learning is compared with
operation of many such solar PV systems in a distributed
environment using CQ learning and it is proved that the grid
power requirement is considerably reduced in CQ learning than
in Q learning. Simulation results using real numerical data are
presented for a reliability test of the system.
Index Terms—Solar microgrid; Reinforcement learning; Q-
learning; CQ Learning; Battery scheduling; Distributed Opti-
mization.
I. I NTRODUCTION
Electricity is a crucial input to propel socio-economic
growth of a nation. Over the past decade, the electricity gen-
eration, transmission and distribution landscape has changed
significantly. Open electricity market, penetration of large
scale renewable generation etc. have further increases the
complexity in the traditional electricity grid towards providing
secure and reliable supply with quality. The key element
driving change is the emergence of smart grid. Renewable
energy plays a significant role in building a green and sus-
tainable environment. Solar and wind are the only solution
to the growing energy crisis in the world [1]. The smart
grid paradigm represents a transition towards an intelligent,
digitally enhanced, two way power delivery grids. The aim of
smart grid is to promote and enhance the efficient management
of operation of the power generation and delivery facilities
by incorporating advanced communications, Information tech-
nology, automation and control methodologies into power
grid. A microgrid is a group of interconnected loads and
distributed energy resources within clearly defined electrical
boundaries that acts as a single controllable entity with respect
to the grid. Microgrid is the building block of smart grid.
978-1-4799-3975-6/14/$31.00 ©2014 IEEE
A Kaviya
SSN College of Engineering
kaviyaindia@gmail.com
Integrating renewable energy in microgrid is the way forward
for economic and environmental optimization generating clean
and green energy and thereby providing solution to the global
warming [2]. A microgrid can connect and disconnect from
the grid to enable it to operate in both grid-connected or
island mode. Microgrids can provide quantifiable benefits
for utilities, their customers, and energy service companies,
regardless of ownership. Solar Photo Voltaic System (PV)
uptake by customers is increasing, due to incentives, improved
performanceand lower cost. Large utility customers increas-
ingly seek reliability, autonomy, cost certainty and resilience.
The importance of having more reliable, efficient, smart sys-
tems is getting more public attention. In the present scenario
the consumer wants not only intelligent but smart machines,
which can think and operate autonomously and optimally.
Smart energy management of microgrids using genetic al-
gorithm is discussed in [2] and [3]. Energy management
of microgrids using fuzzy logic is discussed in [4]. Energy
management of hybrid renewable energy generation using
constrained optimization was proposed in [5]. An agent-based
modeling approach is used to model microgrids and by simu-
lation the interactions between individual intelligent decision-
makers are analyzed in [2], [3] and [8]. Expert system and
other classical and heuristic algorithms for energy management
of microgrids are discussed in [6], [7] and [9]. Reinforcement
learning for optimization of wind energy system is discussed
in [10] and [15]. Coordination Q-Learning method in Multi-
Agent Reinforcement Learning is discussed in [12] and [16].
The aim of this paper is to introduce a smart decision making
system using a multi-agent reinforcement learning method,
called Coordination Q-Learning (CQ Learning), for optimiza-
tion of the distributed energy management in the microgrid.
The system behaves in strategic manner when dealing with
operational scenarios, by aiming to achieve the lowest possible
cost of power generation.
The rest of the paper is organized as follows. In section II,
solar microgrid is explained with the details of solar photo
voltaic system. Section III presents the modeling framework
of consumer agent with comprehensive framework of rein-
forcement learning. In section IV, Multi agent reinforcement
learning framework is explained with CQ Learning. In section
V, the performance improvement of solar microgrid by the
optimal scheduling of the battery to increase the utility of
the battery and solar power, to reduce the power consumption
 2014 IEEE International Conference on Computational Intelligence and Computing Research
from grid is discussed in detail. Also the performance of Solar
PV systems, when operating individually using Q learning
and when they operate in a distributed environment, using
CQ learning are analyzed and compared. Conclusions and
suggestion to possible improvements are given in the last
section.
II. M ODEL OF THE S OLAR M ICROGRID
Microgrids comprise low voltage distribution systems with
distributed energy resources (DER) (micro turbines, fuel cells,
PV, etc.) together with storage devices (flywheels, energy
capacitors and batteries) and flexible loads. Such systems can
be operated in a non-autonomous way, if interconnected to
the grid, or in an autonomous way, if disconnected from the
main grid. The operation of micro sources in the network can
provide distinct benefits to the overall system performance, if
managed and coordinated efficiently [2]. We consider a solar
microgrid with a battery and load. The solar microgrid involves
Switch DC/AC
DC/DC
DC/DC DC/AC
Fig. 1. Solar Microgrid
a consumer with a dynamically varying load Dt , a transformer
providing electricity power from the external grid, a solar
generator (solar photovoltaic system) with available power
output Psp and a storage facility with a level of battery charge
Rt . The architecture of the considered microgrid is shown in
Fig. 1. The consumer can cover his demand partly by using
the electricity produced by solar energy, store electricity in the
battery when the solar source is available and can use from the
storage when needed. The main aim of solar microgrid is to
satisfy the dynamic requirements of the load while maximizing
the utilization of the solar power and the battery. Since solar
PV system gives dc power, proper converters are used for
conversion.
A. Solar Microgrid Neural network model
A photo voltaic (PV) module is the basic element of each
photo voltaic system. A PV module is a clean energy source
used in power systems which absorb solar irradiation and
converts it into electric energy [13]. PV generation fits the load
demand very well since solar irradiation is higher in daytime.
Fig. 2. Neural network model of PV
A solar PV module consists of a number of solar cells
connected in series or parallel based on the requirement of
voltage and current. Modules may then be connected together
into a photo voltaic array. Solar irradiance (G) and temperature
(T ) are the two main factor which influences the solar power.
A simple feed forward neural network is used to train G and
T to get equivalent circuit parameters as shown in Fig. 2.
Once the neural network is trained with sufficient number
of examples then we can determine the current and the
voltage of the solar module for untrained values of G and
T by generalization [14]. The maximum power Psp is found
by Maximum Power Point Tracking (MPPT) algorithm. The
MPPT refers to the point with a maximum power output under
specific external temperature and solar irradiation [13].
B. Model for the battery storage
A simplified model of battery storage dynamics is adopted
by implementing a discrete time system for the power flow
dynamics over the time step interval Dt .
Rt = Rt−1 + Rtstore.charge + Rtstore.discharge (1)
In the equation 1, Rt and Rt−1 are the levels of energy
stored in the battery at time t and t − 1 respectively, and
Rtstore.charge , Rtstore.discharge are the power flows over the time
step interval δt from solar generator to battery, and from
battery to consumer load respectively [10].
III. M ODELING OF THE C ONSUMER AGENT
The dynamic variations of load, solar power and the battery
are considered to constitute the external environment. The
consumer is modeled as an individual agent who makes use of
reinforcement learning for its decision-making, action-taking
and moving towards its goal. Reinforcement learning deals
with learning in sequential decision making in the problems
with limited feedback. Markov Decision Process (MDP) has
become standard formalism for learning in sequential decision
 2014 IEEE International Conference on Computational Intelligence and Computing Research
making. In the MDP, the environment is modeled as set of
states and actions can be performed to control the system
state. The effect of an action taken in a state is dependent
only on that state and not on the prior history. The goal is
to control the system in such a way that some performance
criterion is maximized. This section presents the reinforcement
learning algorithm used by the consumer agent to interact,
adapt, and take decisions towards its goal defined in the form
of reward functions in the MDP environment, characterized
by the available solar power output Psp , the load Dt and the
level of battery charge Rt .
A. Markov Decision Process
Markov Decision Process (MDP) is a way to model a
sequential decision making under uncertainty. We formalize
an MDP, considering discrete states and actions. The initial
state is s0 and each state will have a reward r associated with
it. The transition function T (s|a, s ) indicates the probability
of transitioning from state s to s when action a is taken.
A discount factor γ in the range 0 . . . 1 is applied to future
rewards. This represents the notion that a current reward is
more valuable than one in the future. If it is near zero, future
rewards are almost ignored; a near one places great value
on future reward. The reward from a policy is the sum of
the discounted expected utility of each state visited by that
policy. The optimal policy is the policy that maximizes the
total expected discounted reward.
B. Reinforcement Learning
Reinforcement learning algorithm is used to model the
consumer’s adaptation to a dynamically changing environment
by performing actions of battery scheduling in an MDP envi-
ronment [10]. The agents observe the environment and take an
action. It gets a reward or punishment from the environment.
The agent takes the next action to optimize the reward in the
long run. After a number of interactions, the agent finds the
optimal policy to achieve long term objective. The goal of an
Environment
Action Reward State
Action Value
selection estimation
Agent
Fig. 3. Reinforcement Learning
agent is to find the optimal policy based on interactive learning
with the environment. Fig. 3 shows a simple reinforcement
learning scheme. The environment can be characterized by
the values of a certain number of its features, collectively
called its state, denoted by S(t) at time t. Every state has
an intrinsic value, based on reward or cost, denoted by R(t)
at time t. The agents observe the environment and take an
action. It gets a reward or punishment from the environment.
The training information is used to evaluate actions (in terms
of reward or punishment received from the environment) taken
by the agent. The agent takes the next action to optimize the
reward in the long run. After a number of interactions, with the
enough learning, the agent finds the optimal policy to achieve
long term objective. The agent’s choice of action is based on
its past experience of action taken in a certain state and the
concomitant reward/cost experienced, to update its decision
making process for future actions. In Dynamic Programming
(DP) value iteration and policy iterations are the two methods
to find the optimal policy. But these methods requires models
of the environment so we choose a model free Reinforcement
learning method [11].
C. Reward function
We optimize the battery scheduling of the solar microgrid
by reinforcement learning. This is a process of action-reward
dynamics, driven by quantitative performance indicators which
evaluate the action or sequence of actions undertaken and
feedback the value to adjust future scheduling decisions. The
optimization of the numerical reward is achieved through the
choice of the actions a0 and a1 of battery scheduling. The
consumer aims at increasing its performance by selecting an
optimal sequence of actions. The reward functions are the
response we get from the environment for the actions taken.
If it is charging (a1 ) then the reward function is minimum
of Psp and Bdif f erence and if it is discharging (a0 ) then
the reward function is minimum of Dt and Blevel . Here,
Bdif f erence is the difference between maximum possible
charge and the current battery level (Blevel ). The optimal
scheduling of the battery, and thus the increase of the solar
microgrid performance with respect to the consumer goals, is
done by Q learning.
D. Q learning
Q learning is a model-free reinforcement learning where
the agent explores the environment and finds the next reward
plus the best the agent can do from the next state. In Q
learning, the agent does not need to have any model of the
environment. It only needs to know what states exist and what
actions are possible in each state. We assign each state an
estimated value, called a Q value [11]. When we visit a state
and take an action we receive a reward. We use this reward to
update our estimate of the value of that action in the long run.
We visit the states infinitely often and the action values (Q
values) are continuously updated till it becomes convergent.
The Q learning algorithm is outlined in Algorithm 1 [11]. In
the algorithm, γ is the discount factor and α, learning rate.
The discounted factor contributes to determining the values of
future reward and the the learning rate influences the speed of
convergence to Q values.
 2014 IEEE International Conference on Computational Intelligence and Computing Research

Algorithm 1: Q-learning V. SIMULATIONS OF SOLAR MICROGRID WITH Q

LEARNING , WHEN OPERATING INDIVIDUALLY AND WITH
1 Set γ and rewards matrix R. CQ L EARNING , IN DISTRIBUTED OPERATION
2 Initialize Q(s, a) arbitrarily.
The forecast value of solar irradiance G is found from the
3 foreach episode do
National Renewable Energy Laboratory (NREL) for a 200
4 Initialize s arbitrarily.
KW and 150 KW solar PV systems in our campus electrical
5 repeat for each step of episode
engineering building and hostel respectively for the year 2014.
6 Select a in s using policy derived from Q
The temperature T is taken from meteorological department
7 Take action a, observe reward r, and next state s
for the same year. These two data are fed into already trained
8 Q(s, a) ←
neural network to predict the solar power Psp for the whole
Q(s, a) + α[ r + γ maxa Q(s , a) − Q(s, a) ]
year. Then the solar power distribution for the whole year in
9 s ← s
hourly basis is drawn as shown in the Fig. 4, Fig. 5. The
10 until s is terminal
load pattern of electrical engineering department and hostel in
11 end
our campus are considered. The hourly basis load graphs are
drawn for the department and the hostel as shown in Fig. 6
and Fig. 7. In the department peak loading happens in the
morning and in the hostel peak load happens in evening. The
IV. M ULTI AGENT R EINFORCEMENT L EARNING hourly basis solar power, Psp and the load, Dt feeds the Q-
Learning algorithm in both the agents (department and hostel)
for optimization. The explorer problem is solved using Monte
Carlo method where the actions are chosen stochastically and
A Multi-Agent System (MAS) is a loosely coupled network averaged over 50 times.
of software agents that interact to solve problems that are
beyond the individual capacities or knowledge of each problem
solver. MAS is used to design and develop decentralized
system for distributed optimization. Here autonomous control
process is assumed by each controllable element, namely,
diesel generator, storage and loads. MAS enhances overall
system performance, specifically along the dimensions of
reliability, computational efficiency and scalability. It is built
on two basic pillars namely the reinforcement learning and
the interdisciplinary work on game theory. The agents can
do reasoning, control, optimize and take collective decisions
about how to respond to user requests and unplanned con-
tingencies. Also the agents can be reactive, proactive and
social. Through a reward based system, the agent eventually
learns its optimal behavior by trial-and-error, where the reward
system provides feedback for each of the chosen actions (either
a reward or a penalty depending on the case). The agent Fig. 4. Solar power for 200 kw unit in a year
attempts to maximize its overall reward, and as a results it
converges in the end to an optimal solution. The problem The agent learns optimal single agent policy when acting
arises when several such RL based agents interact within the alone in the environment. Using this prior knowledge, every
same environment, transforming static environments (from a agent has a model of its expected rewards for every state action
single agent’s perspective) in non-stationary environments, as pair. The Q value table is learned by considering the agent as
each agent has a particular influence over the environment an isolated entity in the environment. Each agent learn to find
itself. All agents attempt to learn simultaneously to optimize the policy for local optimization and coordinate with other
individually. Distributed optimization of multiple agent is done agents for global optimization. The coordination q-learning is
through Coordinated Q Learning (CQ Learning). Here we applicable when the different agents, within which optimal
consider two solar microgrids with capacity of 200 KW and coordination have to be achieved, have already accumulated
150 KW for electrical department and hostel respectively. the anticipated Q-value table. The Coordination Q Learning
Each agent optimizes the scheduling of the battery to increase is implemented in three steps. First is to identify the safe and
the utility of the solar power and the battery to reach long dangerous states using a test followed by selection of actions
term objective of reducing the power from the grid. The two and then update the Q values. The test can be approximated
agents cooperate to manage the various scenarios like peak to be equal to the standard deviation for large samples [16].
load management, fault, etc., due to intermittent solar power To accumulate anticipated Q value table through single agent
supply and randomness in load. learning requires a large number of iteration hence the standard
 2014 IEEE International Conference on Computational Intelligence and Computing Research
Fig. 5. Solar power for 150 kw unit in a year
Fig. 6. Load pattern departmentl
deviation is taken. Thus together with the mean and deviation
we can identify whether the state is safe or dangerous and mark
it accordingly. The safe and dangerous states are maintained in
separate lists to decide which Q value table to use. The battery
levels of each indicate the present state of the system. Thus
the number of possible states is a chosen such as to increase
the productivity of the algorithm while still having a reduced
number of states. After having acquired the Q value table,
the agent maintains the mean and variance of various states
through single agent Q Learning method. Then the agents can
be made to undergo reinforced Coordinating Q Learning. The
algorithm then uses the test to detect if there are changes in
the observed rewards for the selected state action pair. If it
detects a change in the reward, algorithm observe this state as
join state, in which collision occurs and mark this as dangerous
state. State action pairs that did not cause collisions are marked
as safe state [15]. Each time an agent encounters a marked
state, it will check whether it is dangerous or safe and the
following update rule is used.
Fig. 7. load pattern hostel
Qjk (js, ak ) = Q(js, ak ) + α[r(js, ak )+
γmaxQ(sk , ak ) − Q(js, ak )] (2)
In the equation 2, Qk stands for the Q table containing the
independent states, and Qjk contains the joint states (js). The
second Q-table is initially empty and eventally gets updated
when the agent encounters joint state space. α is the learning
rate and γ is the discount factor. Python programming is used
to program CQ Learning algorithm and various scenarios the
agents encounters are analysed and the optimized solution is
found to meet the long term objective of reducing the cost
of energy consumption. Comparison is made between CQ
Learning, in a distributed environment and individual agent op-
timizing independently with Q Learning. CQ Learning proved
to be better way of optimizing the power. Three indicators
B0, S0 and Pg show the improvement in the performance
of the microgrid by using the Q Learning algorithm. The
average values of utility of solar power, battery and the power
consumption from the grid of the department are found as
given below and simulated for ten years as shown in Fig. 8,
Fig. 9 and Fig. 10. The increase in the utilization of electricity
from the battery is estimated by B0 which is defined as the
ratio of the cumulative power used from the battery to the
yearly cumulative load.

Battery to load
B0 = 
load(Dt )
The increase of the utilization rate of the solar PV system
evaluated by S0 is defined as the ratio of the yearly cumulative
power used from the solar PV system to the yearly cumulative
available solar power.

Solar to battery
S0 = 
solar power(Psp )
Finally, parameter Pg indicates the cumulative annual power
received from the external grid.
  
Pg = Grid = load(Dt ) − battery to load
 2014 IEEE International Conference on Computational Intelligence and Computing Research
Fig. 8. Utility of battery in department
Fig. 9. Utility of solar power in department
Fig. 11 shows the reduction in power from grid in the
hostel. The reduction in power consumption from the grid
due to both the units (department and the hostel) due to Q
Learning is observed in Fig. 14 and is compared with power
consumption from grid with Coordinated Q Learning in a
distributed environment.
It is found from the Fig. 15 that in Coordinated Q Learning,
the power consumed from grid is relatively reduced due to
distributed optimization. In the first year, the Grid power
required for the solar microgrid with individual Q learning
is 718000 KW. The Grid power required using CQ Learning
is 713000 KW. Thus 5000 KW power is saved in the first
year and the grid power requirement is considerably reduced
in the following years due to distributed optimization, using
Coordinated Q Learning method.
VI. C ONCLUSION
Thus the dynamic interactions between the consumer agent
and its environment is carried out for autonomous optimization
of battery scheduling to increase the utility of the battery,
utility of solar power and thereby the power consumption
Fig. 10. Grid power with QL in department
Fig. 11. Grid power with QL in hostel
from grid is reduced in the long run. Distributed optimization
of solar micro-grids is done with a Multi-Agent Reinforce-
ment Learning approach, namely Coordinated Q learning (CQ
Learning) and the performance is compared with optimiza-
tion of solar microgrid, operating independently, using Q
learning. A simulation model was developed using Python
programming to prove that CQ learning method requires less
power from the grid when compared to Q learning method
under intermittent solar power and randomness of load. The
proposed framework gives the intelligent consumer the ability
to explore and understand the stochastic environment and reuse
this experience for selecting the optimal energy management
actions to reduce dependency on the grid in a distributed
environment. Future work will focus on extension to multiple,
diverse renewable generators (solar and wind) using Multi-
Agent Reinforcement Learning with several intelligent con-
sumers with conflicting requirements. Generalization of state
space can be done through neural networks to reduce the
complexity of the problem.
 2014 IEEE International Conference on Computational Intelligence and Computing Research
Fig. 12. Utility of battery with CQL
Fig. 13. Utility of solar power with CQL
R EFERENCES
[1] Chen C, Duan S, Cai T, Liu B, Hu G, “Smart energy management
system for optimal microgrid economic operation,” Renewable Power
Generation, IET 5(3), pp.258-67, 2011
[2] Ross Guttromson, Steve Glover, “The advanced microgrid integration
and interoperability,” Sandia National Laboratories, Sandia report march
2014.
[3] Reddy P P, Veloso M M, “Strategy learning for autonomous agents in
smart grid markets,” In Twenty-second international joint conference on
artificial intelligence, pp. 1446-51, 2005.
[4] Hatziargyriou ND, European transactions on electrical power, Special
issue: “Microgrid and energy management,” pp. 1139-141, December
2010.
[5] Mohamed F A, Koivo H N, “System modelling and online optimal
management of MicroGrid with battery storage,” International Journal
on Electrical Power and Energy Systems, 32(5), pp.398-407, 2010.
[6] Colson C M, Nehrir M H, Pourmousavi S A, “Towards real-time mi-
crogrid power management using computational intelligence methods.”
IEEE, pp.1-8, 2010.
[7] Abdirahman M Abdilahi, M W Mustafa, G Aliyu, J Usman, “Au-
tonomous Integrated Microgrid (AIMG) System,” International Journal
of Education and Research, Vol.2, no.1, pp.77-82, January 2014.
[8] Jun Z, Junfeng L, Jie W, and Ng H, “A multi-agent solution to energy
management in hybrid renewable energy generation system,” Renewable
Energy, vol. 36, no. 5, pp.1352-63, 2011.
[9] Aymen Chaouachi, Rashad M, Kamel M, Ridha Andoulsi, and Ken
Nagasaka, “Multiobjective Intelligent Energy Management for a Mi-
Fig. 14. Grid power with individual Q Learning
Fig. 15. Grid power with CQ Learning
crogrid,” IEEE Transactions, Industrial Electronics, 60(4), pp.1688-99,
2013.
[10] Kuznetsova E, Li Y F, Ruiz C, Zio E, Ault G, and Bel L K, “Reinforce-
ment learning for microgrid energy management,” Energy Journal, 59,
pp.133-46, 2013.
[11] Sutton R S, Barto A G, “Reinforcement Learning: An introduction”.
London, England: The MIT Press, pp.1-398, 1998.
[12] De Hauwere, Y.M., Vrancx, P., Nowe, A, Learning Multi-Agent State
Space Representations In: Proceedings of the 9th International Confer-
ence on Autonomous Agents and Multi-Agent Systems, Toronto, pp.
715722, Canada,2010.
[13] Engin Karatepe, Mutlu Boztepe, Metin Cola, “Neural network based
solar cell model,” Energy Conversion and Management, 47, pp.1159-
78, 2006.
[14] Hiyama T, Kitabayashi K, “Neural network based estimation of maxi-
mum power generation from PV module using environmental informa-
tion,” IEEE Transactions, Energy Conversion; 12 (3), pp.241-52, 1997.
[15] Yujin Lim, Hak-Man Kim, “Strategic bidding using reinforcement
learning for load shedding in microgrid,” Computers and Electrical
Engineering, Elsevier, 2014.
[16] Marco Wiering and Martijn van Otterlo Ma, “Reinforcement Learning
State -of-the Art”, Springer-Verlag Berlin Heidelberg, 2012.