The Probability Frontier: Research

S B V
RESEARCH

The Probability Frontier
or, Covariance Crunch:
A New Paradigm for Mean‐Variance Optimization

Robert D. Stock, PhD
SBV Research
October, 2019
research@stockbob.com

WORKING PAPER

Abstract
Three innovative concepts are combined here to create a new and unique framework for
optimizing a portfolio of investments or bets. These inventions are:
The Probability Frontier, a generalization of the Markowitz Efficient Frontier;
The Positive Probability Estimate, which estimates the chance of a market to move up in
the coming period, regardless of the magnitude of the move; and
The Directional Covariance, which measures the tendency of markets to move together,
regardless of the size of the moves.
Looking at both market moves and their covariances in terms of pure directionality, treating their
magnitudes as “noise” on the premise (demonstrated decisively in this paper) that direction is
more robustly predictable, dramatically improves the optimized system performance. Indeed, if
our estimates can’t even get market directions right, what’s the point of quibbling over tenths of
a percentage point? Don’t shrink that covariance matrix; crush it!
Keywords: mean variance optimization; efficient frontier; covariance shrinkage; directional
covariance
Electroniccopy
Electronic copyavailable
available at:
at: https://ssrn.com/abstract=3516217
https://ssrn.com/abstract=3516217
research@stockbob.com SBV Research October, 2019
Introduction
It is well‐established that optimizers have difficulties in practice, because standard Markowitz1
mean‐variance optimization magnifies any noise or errors in the input estimates for returns and
covariances. Predicting market returns requires estimating a vector quantity (that is, both a
magnitude and direction). Doing both with accuracy is difficult. But the hypothesis of this paper
is that reducing the task to simply predicting a direction is more tractable. Our Positive
Probability Estimate and Directional Covariance can be thought of as an even more extreme (and
more successful) approach to solving these input estimation problems than resampling2 or
shrinkage3, by crushing out all potentially misleading magnitude information. While other
advanced methods4 have been proposed, this one is easy for practitioners to understand and
implement, without requiring an advanced degree in mathematics or computer science.
This probabilistic idea was inspired partly by Quantum Mechanics (supplying the probability
interpretation) and the Ising Model of atomic spins (suggesting the two‐state directionality
approach). Focusing on directionality produces more diverse (and thus robust) portfolios by
reducing the chance all bets are down at the same time.
The first section of this paper uses a simple example to demonstrate the calculation of the
Positive Probability Estimate and the Directional Covariance. The second section applies these
methods to a real example of optimizing a 7‐asset portfolio using daily data over the last 13 years,
showing how the Probability Frontier produces a more diverse portfolio with robustly superior
risk and return performance versus either a standard Markowitz‐optimized portfolio (with or
without covariance shrinkage) or to a basic buy‐and‐hold portfolio.
I. Probability Frontier
Given a covariance matrix, expected returns, and investment size constraints, the Markowitz
efficient frontier produces a set of portfolios that minimize the variance (interpreted as “risk”) at
a given desired level of return. For any average level of return, the minimum variance portfolio
creates the highest compounded geometric return over time, maximizing the growth of capital.
The Global Minimum Variance portfolio is the one on the efficient frontier with the least expected
variance of all, and represents the portfolio with the best expected return for the least expected
risk. As a unique point on the efficient frontier, we will use the Global Minimum Variance
portfolio for apples‐to‐apples comparisons of the different frontier‐generating methods.
In order to use the well‐developed mathematical algorithms for solving this constrained
minimization problem, it is necessary to find an analogy to the expected returns, and an

1
Markowitz, H. “Portfolio selection.” Journal of Finance, Vol. 7 (1952), pp. 77‐91.
2
Ledoit, O., and Wolf, M. “Improved Estimation of the Covariance matrix of Stock Returns With an Application to
Portfolio Selection.” Journal of Empirical Finance, Vol. 10, No. 5 (2003), pp. 603‐621.
3
Michaud, R. “The Markowitz optimization enigma: Is optimized optimal?” Financial Analysts Journal, Vol. 45
(1989), pp. 31‐42.
4
Lopez de Prado, M. “A Robust Estimator of the Efficient Frontier.” Working Paper (October 15, 2019). Available
at: https://ssrn.com/abstract=3469961.
1

Electroniccopy
available at:
appropriate “covariance” matrix to minimize. In place of the expected returns, we use the
expected frequency (based on a historical period, for example, or incorporating our forward
views) of daily positive returns, which is our estimate of the probability any given day’s return
will be positive for each asset. This weighted‐average odds of each portfolio asset rising will be
called the “combined odds.”
For covariance matrix C, vector of unknown portfolio weights w, vector pE of expected
probabilities of a positive return on any given day for each asset, and desired combined‐odds
scalar , the weights are found by minimizing

subject to
,
along with other possible weight constraints in the usual way (including the budget constraint
that the weights sum to a nominal net value of 100%). Then for a variety of values of , a
probabilistic frontier is generated. The minimum value of  should be at least 50% (as no
investment should be made in any portfolio without a better‐than‐even chance of winning, and
its maximum obviously must not exceed the highest value of pE available (for which the only
possible solution is to invest in that single asset). The pE can be estimated with some forward‐
looking views, or simply by the frequency of positive returns in the historical look‐back period (as
we will use for these examples).

Directional Covariance
There is a conceptual disconnect in using a traditional covariance matrix computed from
historical returns R(t, i) at time t for asset i, which have both magnitude (including noise) and
direction information in them, when the optimization is primarily using a directional‐only
constraint. For a unified paradigm, the “directional covariance” CD is introduced. If the standard
covariance is computed as
cov , ,
where cov is the usual covariance operator, then the directional covariance is defined as
cov ,
using a modified matrix of historical returns RD that simply represents the directions of the
market moves. So, define
1 if ,
,
1 if ,
The value of  can be set to zero, or a minimum return threshold to cover transaction costs. A
“three‐phase” matrix RD could also be defined where it is zero in a range near , but this paper
will just focus on the two‐state directional matrix. The task is then to minimize
2

Electroniccopy
available at:
.
In this formulation, the relative correlation rankings of different assets can change drastically,
and “choppier” markets, with nearly equal numbers of up and down moves, will be treated as
inherently “riskier” than markets which exhibited longer‐term trends in one direction or another.
Simple Example
To gain insight, here is a simple example. Assume there are three assets (A, B, and C). Let the
historical returns of these assets be as shown in Figure 1:
Figure 1: Asset returns and correlations from standard and directional viewpoints
Actual Asset Returns Modified Directional Returns
A B C A B C
0.4% 0.6% (0.3%) 1 1 (-1)
0.1% 0.5% 0.3% 1 1 1
0.2% 1.0% 0.2% 1 1 1
(0.3%) (0.4%) (0.1%) (-1) (-1) (-1)
0.1% 0.1% (0.3%) 1 1 (-1)
(0.2%) (0.1%) (0.4%) (-1) (-1) (-1)
(1.0%) (0.2%) 0.5% (-1) (-1) 1
0.3% 0.5% 0.3% 1 1 1
0.7% 0.2% 0.4% 1 1 1
(0.3%) (0.4%) 0.2% (-1) (-1) 1
0.5% 0.1% (0.4%) 1 1 (-1)
0.2% 0.6% (0.3%) 1 1 (-1)
0.1% (0.2%) 0.3% 1 (-1) 1

Correlations: Correlations:
A to B 0.55 A to B 0.84
A to C (0.24) A to C 0.05
B to C (0.05) B to C (0.10)

Using a traditional covariance formulation, assets A and C appear the most uncorrelated with
each other (and would thus get the most weight in a standard optimization), and even assets A
and B do not look too highly correlated. The directional covariance formulation, however, views
assets B and C as the most uncorrelated, and assets A and B appear much more similar! Clearly,
A and B move up and down at almost identical times, but this fact is hidden using traditional
correlation by the noisy magnitudes of the moves, while the directional correlation sees it.
With just the budget constraint of all weights adding up to one, the Global Minimum Variance
portfolio has a closed form (which depends only on the covariance matrix and not on any
assumed forward return performance), and calculating it to globally minimize the traditional
variance produces a very different “optimal” asset allocation versus globally minimizing the
directional covariance, as shown in figure 2:

3

Electroniccopy
available at:
Figure 2: Global Minimum Variance optimizations using standard and directional covariances
Asset Allocation A B C
Standard 31% 12% 57%
Directional 12% 41% 47%

Note how the allocation nearly reverses between assets A and B, using the directional
formulation! The standard method thinks C is definitely the best asset to overweight, but the
directional method wants to give it only about half the weight, with the other half divided
between the (to it) very similar investments A and B. This directional approach reduces the
chance of all bets being wrong at the same time.
The closed form solution used above for the Global Minimum Variance allocation is
1 ⁄ 1 1.
If we had needed forward combined‐odds estimates, we would use the past frequency of positive
returns to assign an expected daily positive probability of 69% to asset A, 62% to asset B, and
54% to asset C, in the absence of strong views on the future. This is the analogy to the expected
returns in the standard optimization framework.
II. Example: Optimized Trading System
To add more constraints in the usual way, the problem can be solved with a quadratic optimizer
and constraint matrix, where several efficient frontier portfolios would be calculated for a range
of desired forward returns (for standard optimization) or for a range of forward expected
combined‐odds of positive returns. Then, the actual past realized volatility of each portfolio is
calculated (using the true, non‐directionalized returns) over the look‐back period used for
generating the covariance matrix, to find the one with the lowest past realized variance. This
portfolio will be our constrained global minimum variance portfolio which will be used to
compare performance across methods for calculating the frontier.
For this realistic and implementable trading system test, we use daily returns from seven ETFs
from 2006 through late 2019, to cover some popular basic investment choices:
Figure 3: Assets used for optimization tests
Ticker Name Purpose

SPY SPDR S&P 500 U.S. Large Cap Equity
IWM iShares Russell 2000 U.S. Smallcap Equity
EFA iShares MSCI EAFE Foreign Developed Equity
EEM iShares MSCI Emerging Market Emerging Market Equity
AGG iShares Core U.S. Aggregate Core Bonds
TLT iShares 20+ Year Treasury Long-Term Treasuries
GLD SPDR Gold Shares Commodities (Gold)

4

Electroniccopy
available at:
In addition to the constraint that all weights add to 100%, we also used a long‐only constraint
that each weight must be between 0% and 100% (no shorting). We performed an optimization
on two portfolio rebalancing/updating frequencies: 20 trading days (about monthly) and 60
trading days (about quarterly). We also used two look‐back periods of 100 trading days and 200
trading days to compute the covariance, average return, and positive probability calculations.
The portfolios were updated with a one‐day lag (i.e. at the close of day N using information
through the close of day N‐1, for no look‐ahead bias), with transaction costs of 0.05% of the
volume traded per half‐turn. The covariance was computed both in the standard method and in
the directional method. The standard optimization was performed both with and without Ledoit5
covariance shrinkage. We report only the standard results using shrinkage here, as they were
better than not using it, but implementing shrinkage only improved the return/risk ratio of the
standard optimized portfolios by between +0.00 and +0.03 (i.e., nearly negligible improvement).
Results annualized over the period from September 18, 2006 through October 15, 2019 came out
as follows:
Figure 4: Trading system results
Standard
Directional
Parameters Covariance
Covariance
(shrunk)
Look-back (days) 200 200 200 200
Update frequency (days) 20 60 20 60
Return (ann.) 4.36% 3.87% 7.67% 9.57%
Volatility 4.92% 5.06% 7.85% 7.61%
Return/Vol Ratio 0.88 0.76 0.98 1.26
Look-back (days) 100 100 100 100
Update frequency (days) 20 60 20 60
Return (ann.) 4.86% 4.92% 3.83% 5.20%
Volatility 4.49% 5.31% 8.25% 9.08%
Return/Vol Ratio 1.08 0.93 0.46 0.57

While the standard method wins for the 100‐day look‐back, the Directional Covariance wins at
both update frequencies for the 200‐day look‐back, and has the best overall performance winner
for all of the trials (with the 200‐day look‐back with a 60‐day update frequency). Indeed, with
such short windows as 100/20 where the standard method wins, it is likely becoming simply a
short‐term trend‐follower especially as the standard method tends to put most of its eggs in the
best‐looking basket. The Directional method, on the other hand, is designed to specifically
produce more diverse portfolios, and the fact it wins using parameters that feel right for longer‐
term portfolios (updating every quarter, looking back about a year) is gratifying. Longer and
shorter look‐back lengths of 250 and 50 days were also spot‐tested, and performance degraded
for both methods.

5
Ledoit, O. and Wolf, M. “Honey I Shrunk the Sample Covariance Matrix.” UPF Economics and Business Working
Paper No. 691 (June, 2003). Available at: https://ssrn.com/abstract=433840.
5

Electroniccopy
available at:
We look at more details of the performance, now focusing on the best performance for each
method: 100/20 for the standard method, and 200/60 for the new Directional method, and
compare with a static 50% SPY, 50% AGG buy & hold portfolio as a benchmark with a similar long‐
term volatility to the Directional portfolio. The worst drawdown was ‐14.3% for the standard test
and ‐14.5% for the Directional case; this near identity in the worst drawdown is all the more
surprising as the Directional portfolio has 1.7x the volatility! For comparison, the 50‐50 buy &
hold portfolio had a ‐28.8% drawdown. The active Directional portfolio has only half the
drawdown of the (supposedly) similarly‐risky 50‐50 portfolio! Note: the naïve “equal weight”
portfolio underperformed the 50‐50 substantially.
For the following tables, we also switch to using monthly returns for the full months from
October, 2006 through September, 2019, inclusive, for a full 13 years.
Figure 5: Annual calendar returns versus 50‐50 static portfolio
Annual Returns
10/2006
Standard Directional
Through 50-50
100/20 200/60
9/2019
YTD 2006 1.9% 4.3% 4.9%
2007 7.0% 22.3% 6.4%
2008 8.1% 10.2% (15.8%)
2009 2.2% 8.1% 15.3%
2010 7.5% 16.3% 11.2%
2011 7.5% 12.6% 5.6%
2012 4.9% 6.1% 10.0%
2013 2.2% 6.5% 14.0%
2014 6.7% 10.6% 9.9%
2015 (0.6%) (1.2%) 1.2%
2016 3.5% 6.0% 7.4%
2017 7.0% 14.1% 12.4%
2018 (2.0%) 0.1% (1.9%)
YTD 2019 8.2% 11.3% 14.4%

Note the superior performance of the Directional method in the difficult years of 2008 and 2011,
and how the one year it underperformed (2015), it wasn’t trailing by much.
Figure 6: Trailing annualized returns versus 50‐50 static portfolio
Trailing
Through 50-50
100/20 200/60
9/2019
Return (Ann.)
1 Yr 8.8% 13.4% 7.6%
3 Yrs 3.4% 6.6% 8.2%
5 Yrs 3.6% 6.8% 7.2%
10 Yrs 4.5% 8.3% 8.6%
13 Yrs 4.8% 9.5% 6.9%
6

Electroniccopy
available at:
The Directional portfolio beats the best standard optimization over all trailing periods, and is a
good competitor against the 50‐50, providing a diversified return stream (indeed, the daily
correlation is 0.54 to the 50‐50 for the Directional, and 0.42 to the 50‐50 for the standard
optimization). On a risk‐adjusted basis, the Directional optimized portfolio is even more
dominant, and beats the 50‐50 portfolio for all periods except 10 years (where the performance
is very close).
Figure 7: Trailing annualized volatility versus 50‐50 static portfolio
Trailing
Through 50-50
100/20 200/60
9/2019
Volatility (ann., monthly data)
1 Yr 4.2% 3.5% 9.1%
3 Yrs 3.5% 4.4% 5.9%
5 Yrs 3.2% 4.9% 5.9%
10 Yrs 3.1% 6.4% 6.1%
13 Yrs 5.7% 7.1% 7.5%

Figure 8: Trailing annualized return‐to‐risk ratios
Trailing
Through 50-50
100/20 200/60
9/2019
Return / Volatility Ratio
1 Yr 2.07 3.77 0.84
3 Yrs 0.96 1.48 1.38
5 Yrs 1.14 1.38 1.24
10 Yrs 1.43 1.30 1.41
13 Yrs 0.85 1.33 0.92

The Directional optimization gets these results by being both active and more diversified than the
standard method. The daily average number of investments it holds out of the seven possible is 4.6,
versus 3.3 for the standard optimizer. Taking out token investments under a 10% allocation, the
Directional portfolio holds an average of 3.1 investments every day while the standard one has just 1.5
significant investments in its portfolio!
Averaged over time, the Directional portfolio was 32% equity (with a distribution among the four
possibilities), 56% bonds (balanced between core bonds and long‐term Treasuries), and a healthy 11% to
gold. The Standard optimized portfolio, however, is essentially all core bonds! It has just 11% in
equities (almost all in SPY), 87% in bonds, and a nominal 2% in gold.
Figure 9: Long‐term time‐averaged portfolios
Portfolio SPY IWM EFA EEM AGG TLT GLD
Directional 200/60 16% 9% 4% 3% 33% 24% 11%
Standard 100/20 7% 3% 1% 0% 86% 2% 2%

Figures 10 and 11 show how the optimal global minimum variance portfolios evolved over time.
7

Electroniccopy
available at:
Figure 10: Evolution of the Directional Covariance optimized portfolio
Directional Optimization Portfolio Evolution
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
SPY IWM EFA EEM AGG TLT GLD

Figure 11: Evolution of the Standard (shrunk) covariance optimized portfolio
Standard Optimization Portfolio Evolution
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
SPY IWM EFA EEM AGG TLT GLD

8

Electroniccopy
available at:
Figure 12 shows the growth of $100 for the various portfolios. Note how the active optimizations
adjusted quickly in 2008 versus the static 50‐50, but the Directional method wasn’t trapped into bonds
like the standard method was.
Figure 12: Growth of $100 for the various portfolios
Growth of $100 (10/2006 Through 9/2019)
350
$331.12
300
250
$240.29
200
$186.17
150
100
50
0
Sep‐06 Sep‐08 Sep‐10 Sep‐12 Sep‐14 Sep‐16 Sep‐18
Standard Optimizer (100/20) Directional Optimizer (200/60) 50‐50 SPY‐AGG

III. Conclusion and Further Work
This new approach of crushing out magnitude information (i.e., the noise) from the covariance
and forward return assumptions, and replacing them with just directional quantities, vastly
improves the diversification (and hence improves the out‐of‐sample performance) of the
optimized portfolios, without resorting to cumbersome resampling, severe weight constraints,
or other ad hoc methods often relied on to produce reasonable output in the face of estimation
uncertainty.
The case outlined here is for just one small set of assets over a limited period of time, but the
simulation was done in a realistic way accounting for trading costs and lags. Indeed, following
9

Electroniccopy
available at:
Lopez de Prado6, we only tried a very small number of cases using round numbers for our look‐
back and rebalancing parameters, to avoid oversampling bias. While more work is needed, the
outlook is extremely promising.

IV. Appendix: Data Sources
Daily data of total returns for the seven ETFs of Figure 3 is from Bloomberg. All calculations and
charts are by SBV Research.

V. Disclaimers
The observations and opinions of SBV Research (“SBV”) contained herein or elsewhere are
not intended as investment advice. Consult your own investment professional before making
any investment.
This presentation does not constitute an offer to sell, or a solicitation of an offer to buy, any
interest in any investment vehicle, and should not be relied on as such.
Past performance is neither indicative nor a guarantee of comparable future results. Predictions
include the reinvestment of all dividends, interest, income, and profits. The investments
discussed herein may fluctuate in price or value. Investors may get back less than they
invested. It is not possible to invest directly in an index.

6
Lopez de Prado, M. “Illegitimate Science: Why most Empirical Discoveries in Finance Are Likely Wrong, and What
Can Be Done About It”, Presentation Slides (April 25, 2015). Available at: https://ssrn.com/abstract=2599105.
10

Electroniccopy
available at:

The Probability Frontier: Research

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

The Probability Frontier: Research

Uploaded by

Copyright:

Available Formats

S B V

Ticker Name Purpose

SPY IWM EFA EEM AGG TLT GLD

SPY IWM EFA EEM AGG TLT GLD

Standard Optimizer (100/20) Directional Optimizer (200/60) 50‐50 SPY‐AGG

You might also like