ML and Model Risk in Finance

Samuel N. Cohen
University of Oxford and Alan Turing Institute

with Christoph Reisinger and Victor Wang

and Lukasz Szpruch and Derek Snow
Doesn’t machine learning free us from needing models?

16 March 2021 ML and Model Risk 2

Modelling and machine learning

▶ Machine learning provides great modelling flexibility

▶ Neural networks / tree based models / Kernel methods all are
tools for expressivity in approximating functions

▶ How we use these methods, and how we make them interact

with data, is a modelling decision.
▶ The results of these methods have hidden assumptions.

16 March 2021 ML and Model Risk 3

An example application

▶ Risk calculation needs models (under real world probabilities)

of risky outcomes.
▶ Options contain useful information, which may not be in basic
prices.
▶ Estimating risks of option portfolios is useful.

Index price

10000 EURO STOXX 50

DAX
5000

2 4 6 8 0 2 4 6 8
200 200 200 200 201 201 201 201 201

1M ATM call option price (normalised)

0.1
EURO STOXX 50
DAX

0.0
2 4 6 8 0 2 4 6 8
200 200 200 200 201 201 201 201 201

16 March 2021 ML and Model Risk 4

Let’s build a model!

▶ We have a collection of inputs

▶ Historical prices of the underlying index
▶ Historical prices of vanilla options
▶ We have desired outputs
▶ Price changes
▶ Hedging strategies
▶ Distributions of losses, value at risk

Data ML Algorithm Risk assessment

16 March 2021 ML and Model Risk 5

A more realistic pipeline

Data Models Outputs Neural Network

Traditional
Generator

Real Data-driven Price

Inputs
Simulated Handcrafted Volatility Outputs

Hybrid Hedging Strategy

Numerical Risk Assessments

Numerical

Environment Optimisation

We will discuss risks in each of these three columns.

16 March 2021 ML and Model Risk 6

Risks from data

Biased Data
▶ Backward looking.
▶ Spurious Correlations.
▶ Sample disparity (e.g. geographic bias).
▶ Imbalanced and insufficient data.

16 March 2021 ML and Model Risk 7

Risks from data

Errors and preprocessing

▶ Recording errors.
▶ Timing errors.
▶ Inconsistent data
▶ Heavy tails

Static arbitrage violations for CME traded USDEUR option EOD prices
2008–2018

16 March 2021 ML and Model Risk 8

Potential Solutions

▶ Domain knowledge – finance is not image processing!

▶ Common methodology and standardized test cases

▶ Synthetic data for quantity and bias correction

▶ Market simulation

▶ 80% of a data scientist’s time is spent on data preparation.

16 March 2021 ML and Model Risk 9

Example solutions

We have a toolbox to identify and remove arbitrage from

historic data (github.com/vicaws/arbitragerepair)
Major currencies Major currencies

75 AUDUSD 200 GBPUSD

EURUSD USDCAD
50 USDCHF
100
25 USDJPY

0 0
2008 2010 2012 2014 2016 2018 2008 2010 2012 2014 2016 2018
Time Time

Cross currencies Emerging market currencies

750
EURGBP USDBRL
100
USDKRW
500
USDMXN
50
250

0 0
2008 2010 2012 2014 2016 2018 2008 2010 2012 2014 2016 2018
Time Time

OTC Forex options on Bloomberg

16 March 2021 ML and Model Risk 10

Example solutions

If considering intraday data, the problem is even worse.

100
N ε,δ

ATM IV
0.35
N ε,δ

50

ATM IV 0.30
0

1.0%
3040
Futures prices

0.5%

Return
3020
0.0%
15min return (positive)
3000
Futures prices 15min return (negative) -0.5%

12:00 13:00 14:00 15:00 16:00

Time (Eastern Time)

12 June 2020, Emini S&P500 monthly European call options, 1 min resolution.
N ϵ,δ =#perturbations outside bid–ask spread needed to eliminate arbitrage.

16 March 2021 ML and Model Risk 11

Example solutions

Synthetic data
▶ Classical models have well understood performance.
▶ They allow us to use our experience in building models.
▶ We can also consider counterfactuals.
▶ Testing models on synthetic data is critical.

16 March 2021 ML and Model Risk 12

Risks from models and methods

Four typical sources of model risk:

▶ Structural risk
▶ Adversarial attack
▶ Sensitivity
▶ Model drift

16 March 2021 ML and Model Risk 13

Model risk – Structural

▶ ML can be misleading in surprising ways

▶ Model richness is constrained by numerical methods, rather
than explicit modelling choices
▶ The questions we ask often hide modelling assumptions
▶ Prediction set – Markov assumption
▶ Summed quadratic loss – no-correlation assumption
▶ Data driven using historical data – no market impact

16 March 2021 ML and Model Risk 14

Model risk – Structural

How we parameterize our models can be suprisingly important

1.0

0.8

0.6
t
t=T

0.4

0.2 K = 1.05
K = 1.1
K = 1.3
0.0 Traded strikes
0.15 0.10 0.05 0.00 0.05 0.10 0.15
mt = ln(K/Ft(T))
CME EURUSD options with expiry 6 March 2020 from first listing

16 March 2021 ML and Model Risk 15

Model risk – Structural

For our EuroSTOXX50 options model

G0 G1 G2

7.5 1
0.2 5.0
2.5 0
0.1
2 2 2
1 1 1
−0.2 0.0 −0.2 0.0 −0.2 0.0
τ

τ
0.2 0 0.2 0 0.2 0
m m m
G3 G4 G5

5 1 0.5
0 0.0
0
−1 −0.5
−5
2 2 2
1 1 1
−0.2 0.0 −0.2 0.0 −0.2 0.0
τ

τ
0.2 0 0.2 0 0.2 0
m m m

16 March 2021 ML and Model Risk 16

Model risk – Adversarial

▶ Arbitrage is a classic adversarial attack

▶ Dynamic and static arbitrage are different

0.10

0.05
ξ2

0.00

−0.05

−0.02 0.00 0.02 0.04 0.06 0.08 0.10

ξ1

16 March 2021 ML and Model Risk 17

Model risk – Sensitivity

▶ Performance depends on a wide range of inputs

▶ These are an opaque form of model selection

2500 2500
Benchmark Benchmark
0 NN depth = 2 0 NN width = 64
NN depth = 5 NN width = 512
−2500 −2500

−5000 −5000

0 5000 10000 15000 20000 0 5000 10000 15000 20000

Epochs Epochs
2500
Benchmark Benchmark
0 Sparsity ratio = 0 Activation = tanh
0
Sparsity ratio = 0.9 Activation = softmax
−2500 −2500

−5000 −5000

0 5000 10000 15000 20000 0 5000 10000 15000 20000

Epochs Epochs

16 March 2021 ML and Model Risk 18

Risk in outputs
▶ We learn a model for the drift and diffusion of the first
two factors, ensuring no static arbitrage
▶ Remaining factors are fitted as independent OU processes
0.06

0.04

0.02

ξ2
0.00

−0.02

−0.02 0.00 0.02 0.04 0.06 0.08

ξ1

Drift Diffusion
0.06 0.06

0.04 0.04

0.02 0.02
ξ2

ξ2

0.00 0.00

−0.02 −0.02

−0.02 0.00 0.02 0.04 0.06 0.08 −0.02 0.00 0.02 0.04 0.06 0.08
ξ1 ξ1

16 March 2021 ML and Model Risk 19

Risk in outputs

▶ The most fundamental risk is that a model will not calibrate

correctly, so ensuring output appears plausible is of first
importance.
▶ A good model should replicate the full range of observed data,
particularly when observed from directions which were not
used in training.

16 March 2021 ML and Model Risk 20

Risk in outputs – Plausible outputs
Implied Volatility surfaces

DAX 2002-05-29 DAX 2017-07-18 ES50 2008-10-17 ES50 2010-07-29

0.27 0.32
0.24
0.26
0.7 0.30
0.22
0.25
0.20 0.28

Implied volatility
Implied volatility

Implied volatility

Implied volatility
Real 0.24
0.18
0.6

0.26
0.23
data 0.22
0.16 0.5
0.24
0.14
0.21 0.4 0.22
0.12
0.20
0.3 0.20
0.10
−0.2 0.0 0.2 −0.2 0.0 0.2 −0.2 0.0 0.2 −0.2 0.0 0.2
Moneyness m Moneyness m Moneyness m Moneyness m

0.27 0.24 0.32

0.65

0.26 0.22 0.30

0.60

0.25 0.20
0.55
Implied volatility

Implied volatility

Implied volatility
0.28

Implied volatility
0.24 0.18
0.50
0.26
0.23 0.16
0.45
Simulation 0.22 0.14
0.40
0.24

0.12
0.21 0.22
0.35
0.10
0.20
−0.2 0.0 0.2 −0.2 0.0 0.2 −0.2 0.0 0.2 −0.2 0.0 0.2
Moneyness m Moneyness m Moneyness m Moneyness m

16 March 2021 ML and Model Risk 21

Risk in outputs – VIX

Volatility index calculated from CBOE VIX methodology

40

Real 20

data 0 1000 2000 3000 4000

Log-return of S
0.1

0.0

0 1000 2000 3000 4000

Time

Volatility index calculated from CBOE VIX methodology

30

20

10
0 1000 2000 3000 4000 5000
Simulation Log-return of S

0.0

−0.1
0 1000 2000 3000 4000 5000
Time

16 March 2021 ML and Model Risk 22

Risk in outputs – VIX
EURO STOXX 50 index options

0.04
Data
0.03 Real
Simulation
Log-return of S 0.02

0.01

0.00

−0.01

−0.02

−0.03

−0.04
10 20 30 40
Volatility index

16 March 2021 ML and Model Risk 23

Performance and Risk

▶ Given ML models are less explainable than traditional

approaches, they are only worthwhile if they result in a
performance gain.
▶ Performance can be measured both in terms of statistical fit
and predictive power, and in terms of reducing computational
cost
▶ Generally, ML methods will be much more expensive to train,
but may be cheaper to run.

16 March 2021 ML and Model Risk 24

Value at Risk

▶ We can use our model to compute risk profiles for various

options portfolios.
▶ Outright calls and Delta-hedged calls
▶ Delta spreads, Delta butterflies, Delta-neutral strangles
▶ Risk reversal, Calendar spreads, VIX
▶ We use the historical innovations (to allow for unmodelled and
higher-order correlation effects) from our training data
▶ For comparison, we also use a Filtered Historical Simulation
approach, from a time-series model on the Heston parameters.
▶ Given the neural-SDE approach does not involve additional
option pricing, it is much more computationally efficient.

16 March 2021 ML and Model Risk 25

1-day 0.99-Value at Risk

Neural-SDE Heston-EWMA FHS

Coverage ratio median 0.9921 0.9881
Coverage ratio mean 0.9887 0.9742
Kupiec PF (two-sided) 6.92% 25.23%
Kupiec PF (one-sided) 6.92% 25.23%
Christoffersen independence 0.70% 11.03%
Conditional coverage 3.53% 24.88%
Basel committee traffic light 69.1% 29.7% 0.5% 62.4% 25.9% 10.8%

16 March 2021 ML and Model Risk 26

1-day 0.99-Value at Risk

Long Short

Outright

Delta spread

Risk reversal

Long S Short S
VaR (long)
50 50
Realised PnL
0 0

−50 −50
VaR (short)
Realised PnL
−100 −100
2019-01-02 2019-03-13 2019-05-27 2019-08-05 2019-10-14 2019-12-23 2019-01-02 2019-03-13 2019-05-27 2019-08-05 2019-10-14 2019-12-23
Date Date

16 March 2021 ML and Model Risk 27

1-day 0.99-Value at Risk

Long Short

Delta butterfly

Delta-hedged option

Delta-neutral strangle

Calendar spread

Long VIX Short VIX

10 10
VaR (long)
5 Realised PnL
0 0

−5

−10 −10 VaR (short)

2019-01-02 2019-03-13 2019-05-27 2019-08-05 2019-10-14 2019-12-23 2019-01-02 2019-03-13 2019-05-27 2019-08-05 2019-10-14 2019-12-23
Date Date

16 March 2021 ML and Model Risk 28

1-day 0.99-Value at Risk (training sample)

300 VaR (long)

VaR (short)
200
Realised PnL
100

−100

−200

−300

2002-01-02 2006-01-09 2009-12-22 2013-11-21 2017-10-30

16 March 2021 ML and Model Risk 29

1-day 0.99-Value at Risk – procyclicality

1-day VaR0.99
Number of trading strategy

150 Neural-SDE
FHS
100

50

0
0.0 0.2 0.4 0.6 0.8

Trough-to-Peak ratio (1.0 = constant VaR)

16 March 2021 ML and Model Risk 30

1-day 0.99-Value at Risk – computational time

Simulate & revaluation Repair arbitrage

nSDE FHS (Heston) nSDE FHS (Heston)
Expected time (ms) 5.0 147.3 14.3 N.A.
Std. time (ms) 0.3 58.7 7.1 N.A.

Table: Average and standard deviation of the elapsed time for simulation
one risk scenario.

16 March 2021 ML and Model Risk 31

Conclusions

▶ ML is a powerful numerical tool, and gives flexible modelling

▶ Risk management still requires expert knowledge
▶ Benchmarking and testing critical
▶ Explainability and monitoring remains a challenge
▶ Potential benefits mean it’s certainly worth pursuing!

16 March 2021 ML and Model Risk 32

References

▶ SNC, C Reisinger, S Wang, Detecting and repairing arbitrage

in traded option prices, App Math Fin, 2021
(arXiv:2008.09454)
▶ SNC, C Reisinger, S Wang, Arbitrage-free neural-SDE market
models, 2021 (arXiv:2105.11053)
▶ SNC, C Reisinger, S Wang, Estimating risks of option books
using neural-SDE market models, 2022 (arXiv:2202.07148)
▶ SNC, D Snow, L Szpruch, Black-box model risk in finance,
2021, in Machine Learning and Data Science in Financial
Markets (to appear) (arXiv:2102.04757)

16 March 2021 ML and Model Risk 33