Bridging the gap between Markowitz planning and deep reinforcement learning

Eric Benhamou 1,2 , David Saltiel 1,3 , Sandrine Ungari 4 , Abhishek Mukhopadhyay 5
1
AI Square Connect, France, {eric.benhamou,david.saltiel}@aisquareconnect.com
2
MILES, LAMSADE, Dauphine university, France, eric.benhamou@lamsade.dauphine.fr
3
LISIC, ULCO, France, david.saltiel@univ-littoral.fr
4
Societe Generale, Cross Asset Quantitative Research, UK,
5
Societe Generale, Cross Asset Quantitative Research, France,
{sandrine.ungari,abhishek.mukhopadhyay}@sgcib.com

Abstract
While researchers in the asset management industry have
mostly focused on techniques based on financial and risk
planning techniques like Markowitz efficient frontier, min-
imum variance, maximum diversification or equal risk par-
ity, in parallel, another community in machine learning has
started working on reinforcement learning and more partic-
ularly deep reinforcement learning to solve other decision
making problems for challenging task like autonomous driv-
ing, robot learning, and on a more conceptual side games
solving like Go. This paper aims to bridge the gap be-
tween these two approaches by showing Deep Reinforcement
Learning (DRL) techniques can shed new lights on portfolio
allocation thanks to a more general optimization setting that
casts portfolio allocation as an optimal control problem that
is not just a one-step optimization, but rather a continuous
control optimization with a delayed reward. The advantages
are numerous: (i) DRL maps directly market conditions to
actions by design and hence should adapt to changing envi-
ronment, (ii) DRL does not rely on any traditional financial
risk assumptions like that risk is represented by variance, (iii)
DRL can incorporate additional data and be a multi inputs
method as opposed to more traditional optimization methods.
We present on an experiment some encouraging results using
convolution networks.
Introduction
In asset management, there is a gap between mainstream
used methods and new machine learning techniques around
reinforcement learning and in particular deep reinforcement
learning. The former methods rely on financial risk opti-
mization and solve the planning problem of the optimal
portfolio as a single step optimization question. The lat-
ter do not make any assumptions about risk, do a more
involving multi-steps optimization and solve complex and
challenging tasks like autonomous driving (Wang, Jia, and
Weng 2018), learning advanced locomotion and manipula-
tion skills from raw sensory inputs (Levine et al. 2015; 2016;
Schulman et al. 2015a; 2017; Lillicrap et al. 2015) or on a
more conceptual side for reaching supra human level in pop-
ular games like Atari (Mnih et al. 2013), Go (Silver et al.
Copyright
c 2020, Association for the Advancement of Artificial terest as recent breakthroughs has put growing emphasis on
Intelligence (www.aaai.org). All rights reserved.
2016; 2017), StarCraft II (Vinyals et al. 2019), etc ... One of
the reasons often put forward for this situation is that asset
management researchers have mostly been trained with an
econometric and financial mathematics background, while
the deep reinforcement learning community has been mostly
trained in computer science and robotics, leading to two dis-
tinctive research communities that do not interact much be-
tween each other. In this paper, we aim to present the various
approaches to show similarities and differences to bridge the
gap between these two approaches. Both methods can help
solving the decision making problem of finding the optimal
portfolio allocation weights.
Related works
As this paper aims at bridging the gap between traditional
asset management portfolio selection methods and deep re-
inforcement learning, there are too many works to be cited.
On the traditional methods side, the seminal work is
(Markowitz 1952) that has led to various extensions like
minimum variance (Chopra and Ziemba 1993; Haugen and
Baker 1991), (Kritzman 2014), maximum diversification
(Choueifaty and Coignard 2008; Choueifaty, Froidure, and
Reynier 2012), maximum decorrelation (Christoffersen et al.
2010), risk parity (Maillard, Roncalli, and Teı̈letche 2010;
Roncalli and Weisang 2016). We will review these works in
the section entitled Traditional methods.
On the reinforcement learning side, the seminal book is
(Sutton and Barto 2018). The field of deep reinforcement
learning is growing every day at an unprecedented pace,
making the citation exercise complicated. But in terms of
breakthroughs of deep reinforcement learning, one can cite
the work around Atari games from raw pixel inputs (Mnih
et al. 2013; 2015), Go (Silver et al. 2016; 2017), StarCraft II
(Vinyals et al. 2019), learning advanced locomotion and ma-
nipulation skills from raw sensory inputs (Levine et al. 2015;
2016) (Schulman et al. 2015a; 2015b; 2017; Lillicrap et al.
2015), autonomous driving (Wang, Jia, and Weng 2018) and
robot learning (Gu et al. 2017).
On the application of deep reinforcement learning meth-
ods to portfolio allocations, there is already a growing in-
this method. Hence, the field is growing very rapidly and
survey like (Fischer 2018) are already out dated. Driven ini-
tially mostly by applications to crypto currencies and Chi-
nese financial markets (Jiang and Liang 2016; Zhengyao et
al. 2017; Liang et al. 2018; Yu et al. 2019; Wang and Zhou
2019; Saltiel et al. 2020; Benhamou et al. 2020b; 2020a;
2020c), the field is progressively taking off on other as-
sets (Kolm and Ritter 2019; Liu et al. 2020; Ye et al. 2020;
Li et al. 2019; Xiong et al. 2019). More generally, DRL has
recently been applied to other problems than portfolio al-
location. For instance, (Deng et al. 2016; Zhang, Zohren,
and Roberts 2019; Huang 2018; Théate and Ernst 2020;
Chakraborty 2019; Nan, Perumal, and Zaiane 2020; Wu et
al. 2020) tackle the problem of direct trading strategies (Bao
and yang Liu 2019) handles the one of multi agent trading
while (Ning, Lin, and Jaimungal 2018) examine optimal ex-
ecution.

Traditional methods
We are interested in finding an optimal portfolio which Figure 1: Markowitz efficient frontier for the GAFA: returns
makes the planning problem quite different from standard taken from 2017 to end of 2019
planning problem where the aim is to plan a succession
of tasks. Typical planning algorithms are variations around
STRIPS (Fikes and Nilsson 1971), that starts by analysis gies and Σ the matrix of variance covariances of the l strate-
ending goals and means, builds the corresponding graph and gies’ returns. Let rmin be the minimum expected return. The
finds the optimal graph. Indeed we start from the goals to Markowitz optimization problem to solve is to minimize the
achieve and try to find means that can lead to them. New risk given a target of minimum expected return as follows:
work like Graphplan as presented in (Blum and Furst 1995)
uses a novel planning graph, to reduce the amount of search
Minimize wT Σw (1)
needed, while hierarchical task network (HTN) planning w
leverages the classification to structure networks and hence subject to µT w ≥ rmin ,
X
wi = 1, 1 ≥ w ≥ 0
reduce the number of graph searches. Other algorithms like
i=1...l
search algorithm as A∗ , B ∗ , weighted A∗ or for full graph
search, branch and bound and its extensions, as well as evo- It is solved by standard quadratic programming. Thanks
lutionary algorithms like particle swarm, CMA-ES are also to duality, there is an equivalent maximization with a given
used widely in AI planning etc.. However, when it comes to maximum risk σmax for wich the problem writes as follows:
portfolio allocation, standard methods used by practitioners
rely on more traditional financial risk reward optimization Maximize µT w (2)
w
problems and follows rather the Markowitz approach as pre- X
sented below. subject to wT Σw ≤ σmax , wi = 1, 1 ≥ w ≥ 0
i=1...l

Markowitz Minimum variance portfolio

The intuition of Markowitz portfolio is to be able to com-
pare various assets and assemble them taking into account
both return and risk. Comparing just returns of some finan-
cial assets would be too naive. One has to take into account
in her/his investment decision returns with associated risk.
Risk is not an easy concept. in Modern Portfolio Theory
(MPT), risk is represented by the variance of the asset re-
turns. If we take various financial assets and display their
returns and risk as in figure 1, we can find an efficient fron-
tier. Indeed there exists an efficient frontier represented by
the red dot line.
Mathematically, if we denote by w = (w1 , ..., wl ) the al-
location weights with 1 ≥ wi ≥ 0 for i = 0...l, summarized
by 1 ≥ w ≥ 0, with the additional constraints that these
Pl
weights sum to 1: i=1 wi = 1, we can see this portfolio
allocation question as an optimization.
Let µ = (µ1 , ..., µl )T be the expected returns for our l strate-
This seminal model has led to numerous extensions where
the overall idea is to use a different optimization objective.
As presented in (Chopra and Ziemba 1993; Haugen and
Baker 1991), (Kritzman 2014), we can for instance be in-
terested in just minimizing risk (as we are not so much in-
terested in expected returns), which leads to the minimum
variance portfolio given by the following optimization pro-
gram:
Minimize wT Σw (3)
w
X
subject to wi = 1, 1 ≥ w ≥ 0
i=1...l
Maximum diversification portfolio Denoting by σ the
volatilities of our l strategies, whose values are the diagonal
elements of the covariance matrix Σ: σ = (Σi,i )i=1..l , we
can shoot for maximum diversification with the diversifica-
T
What if we could cast this portfolio allocation planning
tion of a portfolio defined as follows: D = √ wT P
σ
. We question as a dynamic control problem where we have some
w w
market information and needs to decide at each time step the
then solve the following optimization program as presented
optimal portfolio allocation problem and evaluate the result
in (Choueifaty and Coignard 2008; Choueifaty, Froidure,
with delayed reward? What if we could move from static
and Reynier 2012)
portfolio allocation to optimal control territory where we
can change our portfolio allocation dynamically when mar-
wT σ ket conditions changes. Because the community of portfolio
Maximize p P (4) allocation is quite different from the one of reinforcement
w wT w
X learning, this approach has been ignored for quite some time
subject to wi = 1, 1 ≥ w ≥ 0 even though there is a growing interest for the use of rein-
i=1...l forcement learning and deep reinforcement learning over the
last few years. We will present here in greater details what
The concept of diversification is simply the ratio of the deep reinforcement is in order to suggest more discussions
weighted average of volatilities divided by the portfolio and exchanges between these two communities.
volatility. Contrary to supervised learning, reinforcement learning
do not try to predict future returns. It does not either try to
Maximum decorrelation portfolio Following (Christof- learn the structure of the market implicitly. Reinforcement
fersen et al. 2010) and denoting by C the correlation ma- learning does more: it directly learns the optimal policy for
trix of the portfolio strategies, the maximum decorrelation the portfolio allocation in connection with the dynamically
portfolio is obtained by finding the weights that provide the changing market conditions.
maximum decorrelation or equivalently the minimum corre-
lation as follows: Deep Reinforcement Learning Intuition
As it name stands for, Deep Reinforcement Learning (DRL)
Minimize wT Cw (5) is the combination of Reinforcement Learning (RL) and
w
X Deep (D). The usage of deep learning is to represent the pol-
subject to wi = 1, 1 ≥ w ≥ 0 icy function in RL. In a nutshell, the setting for applying RL
i=1...l to portfolio management can be summarized as follows:
Risk parity portfolio Another approach following risk • current knowledge of the financial markets is formalized
parity (Maillard, Roncalli, and Teı̈letche 2010; Roncalli and via a state variable denoted by st .
Weisang 2016) is to aim for more parity in risk and solve the • Our planning task which is to find an optimal portfolio
following optimization program allocation can be thought as taking an action at on this
market. This action is precisely the decision of the current
l
1 T 1X portfolio allocation (also called portfolio weights).
Minimize w Σw − ln(wi ) (6)
w 2 n i=1 • once we have decided the portfolio allocation, we observe
X the next state st+1 .
subject to wi = 1, 1 ≥ w ≥ 0
• we use a reward to evaluate the performance of our ac-
i=1...l
tions. In our particular setting, we can compute this re-
All these optimization techniques are the usual way to solve ward only at the the final time of our episode, making it
the planning question of getting the best portfolio allocation. quite special compared to standard reinforcement learn-
We will see in the following section that there are many ing problem. We denote this reward by RT where T is the
alternatives leveraging machine learning that remove cog- final time of our episode. This reward RT is in a sense
nitive bias of risk and are somehow more able to adapt to similar to our objective function in traditional methods.
changing environment. A typical reward is the final portfolio net performance. It
could be obviously other financial performance evaluation
criteria like Sharpe, Sortino ratio, etc..
Reinforcement learning
Following standard RL, we model our problem to solve
Previous financial methods treat the portfolio allocation
with a Markov Decision Process (MDP) as in (Sutton and
planning question as a one-step optimization problem, with
Barto 2018). MDP assumes that the agent knows all the
convex objective functions. There are multiple limitations to
states of the environment and has all the information to make
this approach:
the optimal decision in every state. The Markov property im-
• they do not relate market conditions to portfolio allocation plies in addition that knowing the current state is sufficient.
dynamically. MDP assumes a 4-tuple (S, A, P, R) where S is the set of
states, A is the set of actions, P is the state action to next
• they do not take into account that the result of the portfolio
state transition probability function P : S × A × S → [0, 1],
allocation may potentially be evaluated much later.
and R is the immediate reward. The goal of the agent is
• they make a strong assumptions about risk. to learn a policy that maps states to the optimal action
π : S →PA and that maximizes the expected discounted
∞
reward E[ t=0 γ t Rt ].
The concept of using deep network is to represent the
function that relates dynamically the states to the action
called in RL the policy and denoted by ~at = π(st ). This
function is represented by deep network because of the uni-
versal approximation theorem that states that any function
can be represented by a deep network provided we have
enough layers and nodes. Compared to traditional methods
that only solve a one step optimization, we are solving the
following dynamic control optimization program:
Maximize E[RT ]
π(.)
subject to at = π(st )
Note that we maximize the expected value of the cumu-
lated reward E[RT ] because we are operating in a stochastic
environment. To make things simpler, let us assume that the
cumulated reward is the final portfolio net performance. Let
us write Pt the price at time t of our portfolio, and its re-
turn at time t: rtP and the portfolio assets return vector at
time t: ~rt . The final net performance writes as PT /P0 − 1 =
QT P P
t=1 (1 + rt ) − 1. The returns rt is a function of our plan-
P
ning action at as follows: (1+rt ) = 1+h~at , ~rt i where h·, ·i
is the standard inner product of two vectors. In addition if we
recall that the policy is parametrized by some deep network
parameters, θ: at = πθ (st ), we can make our optimization
problem slightly more detailed as follows:
" T
Y
# computed as rtk =
Maximize E (1 + h~at , ~rt i)
θ
t=1
subject to at = πθ (st ).
It is worth noticing that compared to previous traditional
planning methods (optimization 1, 3, 4, 5 or 5), the underly-
ing optimization problem in RL 7 and its rewritting in terms
of deep network parameters θ as presented in 8 have many
differences:
• First, we are trying to optimize a function π and not sim-
ple weights wi . Although this function at the end is rep-
resented by a deep neural network that has admittely also
weights, this is conceptually very different as we are op-
timizing in the space of functions π : S → A , that is a
much bigger space than simply Rl .
• Second, it is a multi time step optimization at it involves
results from time t = 1 to t = T , making it also more
involving.
Partially Observable Markov Decision Process
If there is in addition some noise in our data and we are not
able to observe the full state, it is better to use Partially Ob-
servable Markov Decision Process (POMDP) as presented
initially in (Astrom 1969). In POMDP, only a subset of
the information of a given state is available. The partially-
informed agent cannot behave optimally. He uses a window
of past observations to replace states as in a traditional MDP.
Mathematically, POMDP is a generalization of MDP.
POMPD adds two more variables in the tuple, O and Z
where O is the set of observations and Z is the observation
transition function Z : S × A × O → [0, 1]. At each time,
the agent is asked to take an action at ∈ A in a particular
environment state st ∈ S, that is followed by the next state
st+1 with P(st+1 |st , at ). The next state st+1 is not observed
by the agent. It rather receives an observation ot+1 ∈ O on
the state st+1 with probability Z(ot+1 |st+1 , at ).
From a practical standpoint, the general RL setting is
modified by taking a pseudo state formed with a set of
past observations (ot−n , ot−n−1 , . . . , ot−1 , ot ). In practice
to avoid large dimension and the curse of dimension, it is
(7)
useful to reduce this set and take only a subset of these
past observations with j < n past observations, such that
0 < i1 < . . . < ij and ik ∈ N is an integer. The set
δ1 = (0, i1 , . . . , ij ) is called the observation lags. In our ex-
periment we typically use lag periods like (0, 1, 2, 3, 4, 20,
60) for daily data, where (0, 1, 2, 3, 4) provides last week ob-
servation, 20 is for the one-month ago observation (as there
is approximately 20 business days in a month) and 60 the
three-month ago observation.
Observations
Regular observations There are two types of observa-
tions: regular and contextual information. Regular observa-
tions are data directly linked to the problem to solve. In the
case of an asset management framework, regular observa-
tions are past prices observed over a lag period δ = (0 <
i1 < . . . < ij ). To normalize data, we rather use past returns
pk
pk
t
− 1 where pkt is the price at time t
t−1
(8) of the asset k. To give information about regime changes,
our trading agent receives also empirical standard devia-
tion computed over a q sliding estimation window denoted by
k 1
Pt 2
d as follows σt = d u=t−d+1 (ru − µ) , where the
empirical mean µ is computed as µ = d1 u=t−d+1 ru .
P t
Hence our regular observations is a three dimensional ten-
sor At = A1t , A2t
 1
rt−ij ... rt1
 1
σt−ij ... σt1
 
with A1t =  ... ... ... , A2t =  ... ... ... 
m
rt−i j
.... rtm m
σt−i j
.... σtm
This setting with two layers (past returns and past volatili-
ties) is quite different from the one presented in (Jiang and
Liang 2016; Zhengyao et al. 2017; Liang et al. 2018) that
uses different layers representing closing, open high low
prices. There are various remarks to be made. First, high
low information does not make sense for portfolio strate-
gies that are only evaluated daily, which is the case of all the
funds. Secondly, open high low prices tend to be highly cor-
related creating some noise in the inputs. Third, the concept
of volatility is crucial to detect regime change and is surpris-
ingly absent from these works as well as from other works
like (Yu et al. 2019; Wang and Zhou 2019; Liu et al. 2020;
Ye et al. 2020; Li et al. 2019; Xiong et al. 2019).
Context observation Contextual observations are addi-
tional information that provide intuition about current con-
text. For our asset manager, they are other financial data
not directly linked to its portfolio assumed to have some
predictive power for portfolio assets. Contextual observa-
tions are stored in a 2D matrix denoted by Ct with stacked
past p individual contextual observations. Among these ob-
servations, we have the maximum and minimum portfo-
lio strategies return and the maximum portfolio strategies
volatility. The latter information is like for regular obser-
vations motivated by the stylized fact that standard devi-
ations are useful features
 1to detect crisis.
 The contextual
ct ... c1t−ik
state writes as C t =  ... ... ... . The matrix nature
cpt .... cpt−ik
of contextual states Ct implies in particular that we will use
1D convolutions should we use convolutional layers. All in
all, observations that are augmented observations, write as
Ot = [At , Ct ], with At = [A1t , A2t ] that will feed the two
sub-networks of our global network.

Action
In our deep reinforcement learning the augmented asset
manager agent needs to decide at each period in which hedg-
ing strategy it invests. The augmented asset manager can in-
vest in l strategies that can be simple strategies or strategies
that are also done by asset management agent. To cope with
reality, the agent will only be able to act after one period.
This is because asset managers have a one day turn around
to change their position. We will see on experiments that this
one day turnaround lag makes a big difference in results.
As it has access to l potential hedging strategies, the out-
put is a l dimension vector that provides how much it invest
in each hedging strategy. For our deep network, this means
that the last layer is a softmax layer to ensure that portfolio
weights are between 0 and 100% and sum to 1, denoted by
(p1t , ..., plt ). In addition, to include leverage, our deep net-
work has a second output which is the overall leverage that
is between 0 and a maximum leverage value (in our experi-
ment 3), denoted by lvgt . Hence the final allocation is given
by lvgt × (p1t , ..., plt ).
Reward
In terms of reward, we are considering the net performance
of our portfolio from t0 to the last train date tT computed as
P Adversarial Policy Gradient
follows: PttT − 1.
0 To learn the parameters of our network depicted in 2, we
Multi inputs and outputs
We display in figure 2 the architecture of our network. Be-
cause we feed our network with both data from the strate-
gies to select but also contextual information, our network is
a multiple inputs network.
Additionally, as we want from these inputs to provide
not only percentage in the different hedging strategies (with
a softmax activation of a dense layer) but also the overall
leverage (with a dense layer with one single output neurons),
we also have a multi outputs network. Additional hyperpa-
rameters that are used in the network as L2 regularization
with a coefficient of 1e-8.
Figure 2: network architecture obtained via tensorflow plot-
model function. Our network is very different from standard
DRL networks that have single inputs and outputs. Contex-
tual information introduces a second input while the lever-
age adds a second output
Convolution networks
Because we want to extract some features implicitly with a
limited set of parameters, and following (Liang et al. 2018),
we use convolution network that perform better than simple
full connected layers. For our so called asset states named
like that because there are the part of the states that relates to
the asset, we use two layers of convolutional network with
5 and 10 convolutions. These parameters are found to be
efficient on our validation set. In contrast, for the contextual
states part, we only use one layer of convolution networks
with 3 convolutions. We flatten our two sub network in order
to concatenate them into a single network.
use a modified policy gradient algorithm called adversarial
as we introduce noise in the data as suggested in (Liang et
al. 2018).. The idea of introducing noise in the data is to
have some randomness in each training to make it more ro-
bust. This is somehow similar to drop out in deep networks
where we randomly perturb the network by randomly re-
moving some neurons to make it more robust and less prone
to overfitting. Here, we are perturbing directly the data to
create this stochasticity to make the network more robust. A
policy is a mapping from the observation space to the action
space, π : O → A. To achieve this, a policy is specified by a
deep network with a set of parameters θ.~ The action is a vec-
tor function of the observation given the parameters: ~at =
πθ~ (ot ). The performance metric of πθ~ for time interval [0, t]
is defined as the corresponding total reward function of the
interval J[0,t] (πθ~ ) = R ~o1 , πθ~ (o1 ), · · · , ~ot , πθ~ (ot ), ~ot+1 .
After random initialization, the parameters are continuously
updated along the gradient direction with a learning rate λ:
θ~ −→ θ~ + λ∇θ~ J[0,t] (πθ~ ). The gradient ascent optimization
is done with standard Adam (short for Adaptive Moment Es-
timation) optimizer to have the benefit of adaptive gradient
descent with root mean square propagation (Kingma and Ba
2014). The whole process is summarized in algorithm 1.
Algorithm 1 Adversarial Policy Gradient
1: Input: initial policy parameters θ, empty replay buffer D
2: repeat
3: reset replay buffer
4: while not terminal do
5: Observe observation o and select action a = πθ (o)
with probability p and random action with proba-
bility 1 − p,
6: Execute a in the environment
7: Observe next observation o0 , reward r, and done
signal d to indicate whether o0 is terminal
8: apply noise to next observation o0
9: store (o, a, o0 ) in replay buffer D
10: if Terminal then
11: for however many updates in D do
12: compute final reward R
13: end for
14: update network parameter with Adam gradient
ascent θ~ −→ θ~ + λ∇θ~ J[0,t] (πθ~ )
15: end if
16: end while
17: until convergence
In our gradient ascent, we use a learning rate of 0.01,
an adversarial Gaussian noise with a standard deviation of
0.002. We do up to 500 maximum iterations with an early
stop condition if on the train set, there is no improvement
over the last 50 iterations.
Experiments
Goal of the experiment
We are interested in planing a hedging strategy for a risky
asset. The experiment is using daily data from 01/05/2000
to 19/06/2020 for the MSCI and 4 SG-CIB proprietary sys-
tematic strategies. The risky asset is the MSCI world index
whose daily data can be found on Bloomberg. We choose
this index because it is a good proxy for a wide range of
asset manager portfolios. The hedging strategies are 4 SG-
CIB proprietary systematic strategies further described be-
low. Training and testing are done following extending walk
forward analysis as presented in (Benhamou et al. 2020b;
2020c; 2020a) with initial training from 2000 to end of 2006
and testing in a rolling 1 year period. Hence, there are 14
training and testing periods, with the different testing pe-
riod corresponding to all the years from 2007 to 2020 and
training done for period starting in 2000 and ending one day

before the start of the testing period.
Data-set description
Systematic strategies are similar to asset managers that in-
vest in financial markets according to an adaptive, pre-
defined trading rule. Here, we use 4 SG CIB proprietary
’hedging strategies’, that tend to perform when stock mar-
kets are down:
• Directional hedges - react to small negative return in eq-
uities,
• Gap risk hedges - perform well in sudden market crashes,
• Proxy hedges - tend to perform in some market config-
urations, like for example when highly indebted stocks
under-perform other stocks,
• Duration hedges - invest in bond market, a classical diver-
sifier to equity risk in finance.
The underlying financial instruments vary from put op-
tions, listed futures, single stocks, to government bonds.
Some of those strategies are akin to an insurance contract
and bear a negative cost over the long run. The challenge
consists in balancing cost versus benefits.
In practice, asset managers have to decide how much of
these hedging strategies are needed on top of an existing
portfolio to achieve a better risk reward. The decision mak-
ing process is often based on contextual information, such
as the economic and geopolitical environment, the level of
risk aversion among investors and other correlation regimes.
The contextual information is modeled by a large range of
features :
• the level of risk aversion in financial markets, or market
sentiment, measured as an indicator varying between 0 for
maximum risk aversion and 1 for maximum risk appetite,
• the bond equity historical correlation, a classical ex-
post measure of the diversification benefits of a duration
hedge, measured on a 1 month, 3 month and 1 year rolling
window,
• The credit spreads of global corporate - investment grade,
high yield, in Europe and in the US - known to be an early
indicator of potential economic tensions,
• The equity implied volatility, a measure if the ’fear factor’
in financial market,
• The spread between the yield of Italian government bonds
and the German government bond, a measure of potential
tensions in the European Union,
• The US Treasury slope, a classical early indicator for US
recession,
• And some more financial variables, often used as a gauge
for global trade and activity: the dollar, the level of rates
in the US, the estimated earnings per shares (EPS).
A cross validation step selects the most relevant features.
In the present case, the first three features are selected. The
rebalancing of strategies in the portfolio comes with trans-
action costs, that can be quite high since hedges use op-
tions. Transactions costs are like frictions in physical sys-
tems. They are taken into account dynamically to penalise
solutions with a high turnover rate.

Evaluation metrics Table 1: Models comparison over 2 and 5 years

2 Years
Asset managers use a wide range of metrics to evaluate the
return Sortino Sharpe max DD
success of their investment decision. For a thorough review
Risky asset 8.27% 0.39 0.36 - 0.34
of those metrics, see for example (Cogneau and Hübner DRL 20.64% 0.94 0.96 - 0.27
2009). The metrics we are interested in for our hedging prob- Markowitz -0.25% - 0.01 - 0.01 - 0.43
lem are listed below: MinVariance -0.22% - 0.01 - 0.01 - 0.43
• annualized return defined as the average annualized com- MaxDiversification 0.24% 0.01 0.01 - 0.43
MaxDecorrel 14.42% 0.65 0.63 - 0.21
pounded return, RiskParity 14.17% 0.73 0.72 -0.19
• annualized daily based Sharpe ratio defined as the ratio 5 Years
of the annualized return over the annualized daily based return Sortino Sharpe max DD
volatility µ/σ, Risky asset 9.16% 0.57 0.54 - 0.34
• Sortino ratio computed as the ratio of the annualized re- DRL 16.95% 1.00 1.02 - 0.27
Markowitz 1.48% 0.07 0.06 - 0.43
turn overt the downside standard deviation, MinVariance 1.56% 0.08 0.06 - 0.43
• maximum drawdown (max DD) computed as the max- MaxDiversification 1.77% 0.08 0.07 - 0.43
imum of all daily drawdowns. The daily drawdown is MaxDecorrel 7.65% 0.44 0.39 - 0.21
computed as the ratio of the difference between the run- RiskParity 7.46% 0.48 0.43 -0.19
ning maximum of the portfolio value defined as RMT =
maxt=0..T (Pt ) and the portfolio value over the running
maximum of the portfolio value. Hence the drawdon at
time T is given by DDT = (RMT − PT )/RMT while
the maximum drawdown M DDT = maxt=0..T (DDt ).
It is the maximum loss in return that an investor will incur
if she/he invested at the worst time (at peak).

Results and discussion

Overall, the DRL approach achieves much better results than
traditional methods as shown in table 1, except for the max-
imum drawdown (max DD). Because time horizon is impor-
tant in the comparison we provide risk measures for the last
2 and 5 years to emphasize that the DRL approach seems
more robust than traditional portfolio allocation methods.
When plotting performance results from 2007 to 2020 as
shown in figure 3, we see that DRL model is able to devi-
ate upward from the risky asset continuously, indicating a
steady performance. In contrast, other financial models are
not able to keep their marginal over-performance over time
with respect to the risky asset and end slightly below the
risky asset.

Allocation chosen by models

The reason of the stronger performance of DRL comes from
the way it chooses its allocation. Contrarily to standard fi-
nancial methods that play the diversification as shown in fig-
ure 4, DRL aims at choosing a single hedging strategy most
of the time and at changing it dynamically, should the fi- Figure 3: performance of all models
nancial market conditions change. In a sense, DRL is doing
some cherry picking by selecting what it thinks is the best
hedging strategy.
In contrast, traditional models like Markowitz, minimum
variance, maximum diversification, maximum decorrelation

Figure 4: weights for all models
and risk parity provides non null weights for all our hedg-
ing strategies and do not do cherry picking at all. They are
neither able to change the leverage used in the portfolio as
opposed to DRL model.
Adaptation to the Covid Crisis
The DRL model can change its portfolio allocation should
the market conditions change. This is the case from 2018
onwards, with a short deleveraging window emphasized by
the small blank disruption during the Covid crisis as shown
in figure 5. We observe in this figure where we have zoomed
over year 2020, that the DRL model is able to reduce lever-
age from 300 % to 200 % during the Covid crisis (end of
February 2020 to start of April 2020). This is a unique fea-
ture of our DRL model compared to traditional financial
planning models that do not take leverage into account and
keeps a leverage of 300 % regardless of market conditions.
Benefits of DRL
As illustrated by the experiment, the advantages of DRL are
numerous: (i) DRL maps directly market conditions to ac-
tions by design and hence should adapt to changing envi-
ronment, (ii) DRL does not rely on any traditional financial
risk assumptions, (iii) DRL can incorporate additional data
and be a multi inputs method as opposed to more traditional
optimization methods.
Figure 5: disallocation of DRL model
Future work
As nice as this work is, there is room for improvement as
we have only tested a few scenarios and only a limited set of
hyper-parameters for our convolutional networks. We should
do more intensive testing to confirm that DRL is able to bet-
ter adapt to changing financial environment. We should also
investigate the impact of more layers and other design choice
in our network.
Conclusion
In this paper, we discuss how a traditional portfolio alloca-
tion problem can be reformulated as a DRL problem, trying
to bridge the gaps between the two approaches. We see that
the DRL approach enables us to select fewer strategies, im-
proving the overall results as opposed to traditional methods
that are built on the concept of diversification. We also stress
that DRL can better adapt to changing market conditions and
is able to incorporate more information to make decision.
Acknowledgments
We would like to thank Beatrice Guez and Marc Pantic for
meaningful remarks. The views contained in this document
are those of the authors and do not necessarily reflect the
ones of SG CIB.
References
Astrom, K. 1969. Optimal control of markov processes with
incomplete state-information ii. the convexity of the loss-
function. Journal of Mathematical Analysis and Applica-
tions 26(2):403–406.
Bao, W., and yang Liu, X. 2019. Multi-agent deep rein-
forcement learning for liquidation strategy analysis.
Benhamou, E.; Saltiel, D.; Ohana, J.-J.; and Atif, J. 2020a.
Detecting and adapting to crisis pattern with context based
deep reinforcement learning. ICPR 2020.
Benhamou, E.; Saltiel, D.; Ungari, S.; and Mukhopadhyay,
A. 2020b. Aamdrl: Augmented asset management with deep
reinforcement learning. arXiv.
Benhamou, E.; Saltiel, D.; Ungari, S.; and Mukhopadhyay,
A. 2020c. Bridging the gap between markowitz planning
and deep reinforcement learning. ICAPS FinPlan workshop.
Blum, A., and Furst, M. 1995. Fast Planning Through Plan-
ning Graph Analysis. In IJCAI, 1636–1642.
Chakraborty, S. 2019. Capturing financial markets to apply
deep reinforcement learning.
Chopra, V. K., and Ziemba, W. T. 1993. The effect of errors
in means, variances, and covariances on optimal portfolio
choice. Journal of Portfolio Management 19(2):6–11.
Choueifaty, Y., and Coignard, Y. 2008. Toward maximum
diversification. Journal of Portfolio Management 35(1):40–
51.
Choueifaty, Y.; Froidure, T.; and Reynier, J. 2012. Proper-
ties of the most diversified portfolio. Journal of Investment
Strategies 2(2):49–70.
Christoffersen, P.; Errunza, V.; Jacobs, K.; and Jin, X. 2010.
Is the potential for international diversification disappear-
ing? Working Paper.
Cogneau, P., and Hübner, G. 2009. The 101 ways to measure
portfolio performance. SSRN Electronic Journal.
Deng, Y.; Bao, F.; Kong, Y.; Ren, Z.; and Dai, Q. 2016. Deep
direct reinforcement learning for financial signal representa-
tion and trading. IEEE Transactions on Neural Networks
and Learning Systems 28:1–12.
Fikes, R. E., and Nilsson, N. J. 1971. Strips: A new approach
to the application of theorem proving to problem solving.
Artificial Intelligence 2:189.
Fischer, T. G. 2018. Reinforcement learning in financial
markets - a survey. Discussion Papers in Economics 12.
Gu, S.; Holly, E.; Lillicrap, T.; and Levine, S. 2017. Deep
reinforcement learning for robotic manipulation with asyn-
chronous off-policy updates. In IEEE International Confer-
ence on Robotics and Automation (ICRA), 3389–3396.
Haugen, R., and Baker, N. 1991. The efficient market inef-
ficiency of capitalization-weighted stock portfolios. Journal
of Portfolio Management 17:35–40.
Huang, C. Y. 2018. Financial trading as a game: A deep
reinforcement learning approach.
Jiang, Z., and Liang, J. 2016. Cryptocurrency Portfolio
Management with Deep Reinforcement Learning. arXiv e-
prints.
Kingma, D., and Ba, J. 2014. Adam: A method for stochastic
optimization.
Kolm, P. N., and Ritter, G. 2019. Modern perspective on
reinforcement learning in finance. SSRN.
Kritzman, M. 2014. Six practical comments about asset
allocation. Practical Applications 1(3):6–11.
Levine, S.; Finn, C.; Darrell, T.; and Abbeel, P. 2015. End-
to-end training of deep visuomotor policies. Journal of Ma-
chine Learning Research 17.
Levine, S.; Pastor, P.; Krizhevsky, A.; and Quillen, D. 2016.
Learning hand-eye coordination for robotic grasping with
deep learning and large-scale data collection. The Interna-
tional Journal of Robotics Research.
Li, X.; Li, Y.; Zhan, Y.; and Liu, X.-Y. 2019. Optimistic bull
or pessimistic bear: Adaptive deep reinforcement learning
for stock portfolio allocation. In ICML.
Liang et al. 2018. Adversarial deep reinforcement learning
in portfolio management.
Lillicrap, T.; Hunt, J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa,
Y.; Silver, D.; and Wierstra, D. 2015. Continuous control
with deep reinforcement learning. CoRR.
Liu, Y.; Liu, Q.; Zhao, H.; Pan, Z.; and Liu, C. 2020. Adap-
tive quantitative trading: an imitative deep reinforcement
learning approach. In AAAI.
Maillard, S.; Roncalli, T.; and Teı̈letche, J. 2010. The prop-
erties of equally weighted risk contribution portfolios. The
Journal of Portfolio Management 36(4):60–70.
Markowitz, H. 1952. Portfolio selection. Journal of Finance
7:77–91.
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.;
Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013.
Playing atari with deep reinforcement learning. NIPS Deep
Learning Workshop.
Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.; Veness,
J.; Bellemare, M.; Graves, A.; Riedmiller, M.; Fidjeland,
A.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.;
Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg,
S.; and Hassabis, D. 2015. Human-level control through
deep reinforcement learning. Nature 518:529–33.
Nan, A.; Perumal, A.; and Zaiane, O. R. 2020. Sentiment
and knowledge based algorithmic trading with deep rein-
forcement learning.
Ning, B.; Lin, F. H. T.; and Jaimungal, S. 2018. Double deep
q-learning for optimal execution.
Roncalli, T., and Weisang, G. 2016. Risk parity portfolios
with risk factors. Quantitative Finance 16(3):377–388.
Saltiel, D.; Benhamou, E.; Ohana, J. J.; Laraki, R.; and Atif,
J. 2020. Drlps: Deep reinforcement learning for portfolio
selection. ECML PKDD Demo track.
Schulman, J.; Levine, S.; Moritz, P.; Jordan, M.; and Abbeel,
P. 2015a. Trust region policy optimization. In ICML.
Schulman, J.; Moritz, P.; Levine, S.; Jordan, M.; and Abbeel,
P. 2015b. High-dimensional continuous control using gen-
eralized advantage estimation. ICLR.
Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and
Klimov, O. 2017. Proximal policy optimization algorithms.
CoRR.
Silver, D.; Huang, A.; Maddison, C.; Guez, A.; Sifre, L.;
Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneer-
shelvam, V.; Lanctot, M.; Dieleman, S.; Grewe, D.; Nham,
J.; Kalchbrenner, N.; Sutskever, I.; Lillicrap, T.; Leach, M.;
Kavukcuoglu, K.; Graepel, T.; and Hassabis, D. 2016. Mas-
tering the game of go with deep neural networks and tree
search. Nature 529:484–489.
Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.;
Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton,
A.; Chen, Y.; Lillicrap, T.; Hui, F.; Sifre, L.; van den Driess-
che, G.; Graepel, T.; and Hassabis, D. 2017. Mastering the
game of go without human knowledge. Nature 550:354–.
Sutton, R. S., and Barto, A. G. 2018. Reinforcement Learn-
ing: An Introduction. The MIT Press, second edition.
Théate, T., and Ernst, D. 2020. Application of deep re-
inforcement learning in stock trading strategies and stock
forecasting.
Vinyals, O.; Babuschkin, I.; Czarnecki, W.; Mathieu, M.;
Dudzik, A.; Chung, J.; Choi, D.; Powell, R.; Ewalds, T.;
Georgiev, P.; Oh, J.; Horgan, D.; Kroiss, M.; Danihelka, I.;
Huang, A.; Sifre, L.; Cai, T.; Agapiou, J.; Jaderberg, M.;
and Silver, D. 2019. Grandmaster level in starcraft ii using
multi-agent reinforcement learning. Nature 575.
Wang, H., and Zhou, X. Y. 2019. Continuous-Time Mean-
Variance Portfolio Selection: A Reinforcement Learning
Framework. arXiv e-prints.
Wang, S.; Jia, D.; and Weng, X. 2018. Deep reinforcement
learning for autonomous driving. ArXiv abs/1811.11329.
Wu, X.; Chen, H.; Wang, J.; Troiano, L.; Loia, V.; and
Fujita, H. 2020. Adaptive stock trading strategies with
deep reinforcement learning methods. Information Sciences
538:142–158.
Xiong, Z.; Liu, X.-Y.; Zhong, S.; Yang, H.; and Walid, A.
2019. Practical deep reinforcement learning approach for
stock trading.
Ye, Y.; Pei, H.; Wang, B.; Chen, P.-Y.; Zhu, Y.; Xiao, J.; and
Li, B. 2020. Reinforcement-learning based portfolio man-
agement with augmented asset movement prediction states.
In AAAI.
Yu, P.; Lee, J. S.; Kulyatin, I.; Shi, Z.; and Dasgupta, S.
2019. Model-based deep reinforcement learning for finan-
cial portfolio optimization. RWSDM Workshop, ICML 2019.
Zhang, Z.; Zohren, S.; and Roberts, S. 2019. Deep rein-
forcement learning for trading.
Zhengyao et al. 2017. Reinforcement learning framework
for the financial portfolio management problem. arXiv.