5 views

Uploaded by Michael Kaplan

kedgeree recipe

save

- 3419.pdf
- 1602.07300v2
- n13.pdf
- Ch. 08
- Mendes 1994
- System Programming
- Patriarche Et Al. - 2007 - Stereological Analysis of Fractures in the Roselend Tunnel and Permeability Determination
- Estimation
- Standard Deviation
- Case Analysis 2
- Writing up research - a statistical perspective
- The Use of Geographically Weighted Regression
- Introduction Defining the Role of Statistics in Business
- Only Be Live Misa
- 054.pdf
- Adaptive Parametric Algorithms for Processing
- birchall1-article critique-april 14-2015-final2
- Bed-Thickness Structure of Multi-Sourced Ramp Turbidites; Devonian Brallier Formation, Central Appalachian Basin - JSR, Drummond, 1999.pdf
- Effect of Protests on Voting
- Brassier Genre is Obsolete
- Tausk The Influencinh Machine.pdf
- Estlund Beyond Fairness & Deliberation
- A Culture of Candor - HBR
- SHEG Evaluating Information Online
- Bringhurst & Chappell a Short History of the Printed Word
- Legitimating the Bailout
- Mooney Citizen Juries Handbook
- Scheuerman Civil Disobedience & Anti-Legal Turn
- Žižek Snowden Assange and Political Event
- Is Religion Above the Law
- How Christian Were the Founders
- Aronczyk Marketing Activism
- Critchley the Rigor of Love
- Five Minute Refresher Ezine
- Bellah American Civil Religion
- Ford & Nazis
- Clarke Ordinary People
- Gilgen Deconversion of de Vries
- ESTADÍSTICA BÁSICA
- Anteproyecto Residencia
- thesis by gaspar
- Final Exam
- Basic Numeracy Statistics
- Advance Probability and Statistics - 2014 - 2da edición - Ottman.pdf
- Semibatch Reaction Crystallization of Benzoic Acid
- Statistical Methods for Research Workers. Fisher, Ronald a. 2000
- Evaluación de la bondad de un estimador puntual
- Scenario Analysis
- Verma&Quiroz_2006_Tablas_Criticas.pdf
- GJBSS Paper Vol4 Issue 3 4 Paper3
- Elastic Stability of Thin- Walled Cylindrical and Conical Shells under Axial Compression
- Statistics 01
- (www.entrance-exam.net)-RPF Sub Inspectors Constables Exam Sample Paper 5
- Unidad_09
- 3360hyp
- How to Choose a Sample Size
- Imf
- Uncertainties
- Andrade Et Al. - 2014 - Maintaining High Activity Levels in Sedentary Adults With a Reinforcement-thinning Schedule
- 7.Hum-professional Commitment of Teacher Educators-dr. Nimisha Beri
- Trabajo final de est. 1. 2018-2 Semi presencial.docx
- Muest Reo
- Practice Test in Statistics
- Bringing Accelerated Math and English to your Campus
- MHPT IPM 4
- SDA 3E Chapter 4
- exam1
- Straightening Standard Deviation

You are on page 1of 20

**Pinker, the second a technical discussion of the
**

flaw in Pinker’s book forthcoming in Physica A: Statistical Mechanics and Applications, the third a technical discussion of what I call the

Pinker Problem, a corruption of the law of large numbers.

**The “Long Peace” is a Statistical Illusion
**

Nassim Nicholas Taleb

When I finished writing The Black Swan, in 2006, I was confronted with ideas of “great moderation”, by people who did not realize that the

process was getting fatter and fatter tails (from operational and financial, leverage, complexity, interdependence, etc.), meaning fewer but

deeper departures from the mean. The fact that nuclear bombs explode less often that regular shells does not make them safer. Needless to say

that with the arrival of the events of 2008, I did not have to explain myself too much. Nevertheless people in economics are still using the

methods that led to the “great moderation” narrative, and Bernanke, the protagonist of the theory, had his mandate renewed.

I had argued that we were undergoing a switch between the top graph (continuous low grade volatility) to the next one, with the process moving

by jumps, with less and less variations outside of jumps.

Position

140

130

120

110

100

90

80

50

100

150

200

250

Time

Position

140

130

120

110

100

90

80

50

100

150

200

250

Time

**My idea of current threats outside finance:
**

1. Loss of the Island Effect: My point now is the loss of island effect, from interconnectedness. The number one danger is a biological agent

that can travel on British Air and reach the entire planet. And it can be built in a high school lab, or, even, a garage.

2. Nuclear Potential: As I explain in Antifragile, risk management is about fragility, not naive interpretation of past data. If Fannie Mae is

sitting on a barrel of dynamite I would not use past statistical data for my current analysis. Risks are in the fragility.( Sensitivity to

counterfactuals is more important than past history, something Greenspan missed in his famous congressional testimony).

**The Pinker Argument
**

Now to my horror I saw an identical theory of great moderation produced by Steven Pinker with the same naive statistically derived discussions

(>700 pages of them!).

1. I agree that diabetes is a bigger risk than murder --we are victims of sensionalism. But our suckerdom for overblown narratives of violence

does not imply that the risks of large scale violent shocks have declined. (The same as in economics, people’s mapping of risks are out of

sync and they underestimate large deviations). We are just bad at evaluating risks.

2

Long Peace.nb

2. Pinker conflates nonscalable Mediocristan (death from encounters with simple weapons) with scalable Extremistan (death from heavy

shells and nuclear weapons). The two have markedly distinct statistical properties. Yet he uses statistics of one to make inferences about

the other. And the book does not realize the core difference between scalable/nonscalable (although he tried to define powerlaws). He

claims that crime has dropped, which does not mean anything concerning casualties from violent conflict.

3. Another way to see the conflation, Pinker works with a times series process without dealing with the notion of temporal homogeneity.

Ancestral man had no nuclear weapons, so it is downright foolish to assume the statistics of conflicts in the 14th century can apply to the

21st. A mean person with a stick is categorically different from a mean person with a nuclear weapon, so the emphasis should be on the

weapon and not exclusively on the psychological makup of the person.

4. The statistical discussions are disturbingly amateurish, which would not be a problem except that the point of his book is statistical. Pinker

misdefines fat tails by talking about probability not contribution of rare events to the higher moments; he somehow himself accepts

powerlaws, with low exponents, but he does not connect the dots that, if true, statistics can allow no claim about the mean of the process.

Further, he assumes that data reveals its properties without inferential errors. He talks about the process switching from 80/20 to 80/02,

when the first has a tail exponent of 1.16, and the other 1.06, meaning they are statistically indistinguishable. (Errors in computation of tail

exponents are at leat .6, so this discussion is noise, and as shown in [1], [2], it is lower than 1. (It is an error to talk 80/20 and derive the

statistics of cumulative contributions from samples rather than fit exponents; an 80/20 style statement is interpolative from the existing

sample, hence biased to clip the tail, while exponents extrapolate.)

5. He completely misses the survivorship biases (which I called the Casanova effect) that make an observation by an observer whose survival

depends on the observation invalid probabilistically, or to the least, biased favorably. Had a nuclear event taken place Signor Pinker would

not have been able to write the book.

6. He calls John Gray’s critique “anecdotal”, yet it is more powerful statistically (argument of via negativa) than his >700 pages of pseudostats.

7. Psychologically, he complains about the lurid leading people to make inferences about the state of the system, yet he uses lurid arguments

to make his point.

8. You can look at the data he presents and actually see a rise in war effects, comparing pre-1914 to post 1914.

3

9. Recursing a Bit (Point added Nov 8): Had a book proclaiming The Long Peace been published in 1913 it would carry similar arguments

4

to those in Pinker’s book.

**Is The World Getting Less Safe?
**

When writing The Black Swan, I was worried about the state of the system from the new type of fragility, which I ended up defining later. I had

noted that because of connectedness and complexity, earthquakes carried a larger and larger proportional economic costs, an analysis by

Zajdenwebber which saw many iterations and independent rediscoveries (the latest in a 2012 Economist article). The current deficit is largely

the result of the (up to) $3 trillion spent after September 11 on wars in Afganistan and Iraq.

But connectedness has an even larger threat. With biological connectedness, the Island Effect is gone both economically and physically: tails

are so fat ...

{Technical Appendix}

**Pinker’s Rebuttal of This Note
**

Pinker has written a rebuttal (ad hominem blather, if he had a point he would have written something

1

of

3

this, not 3 x the words). He still does

not understand the difference between probability and expectation (drop in observed volatility/fluctualtion ≠ drop in risk) or the incompatibility

of his claims with his acceptance of fat tails (he does not understand asymmetries-- from his posts on FB and private correspondence). Yet it

was Pinker who said “what is the actual risk for any individual? It is approaching zero”.

**Second Thoughts on The Pinker Story: What Can We Learn From It
**

It turned out, the entire exchange with S. Pinker was a dialogue de sourds. In my correspondence and exchange with him, I was under the

impression that he simply misunderstood the difference between inference from symmetric, thin-tailed random variables an one from asymmetric, fat-tailed ones --the 4th Quadrant problem. I thought that I was making him aware of the effects from the complications of the distribution.

But it turned out things were worse, a lot worse than that.

Pinker doesn’t have a clear idea of the difference between science and journalism, or the one between rigorous empiricism and anecdotal

statements. Science is not about making claims about a sample, but using a sample to make general claims and discuss properties that apply

outside the sample.

Take M* the observed arithmetic mean from the realizations (a sample path) for some process, and M the "true" mean. When someone says:

"Crime rate in NYC dropped between 2000 and 2010", the claim is about M* the observed mean, not M the true mean, hence the claim can be

deemed merely journalistic, not scientific, and journalists are there to report "facts" not theories. No scientific and causal statement should be

made from M* on "why violence has dropped" unless one establishes a link to M the true mean. M* cannot be deemed "evidence" by itself.

Working with M* cannot be called "empiricism".

What I just wrote is at the foundation of statistics (and, it looks like, science). Bayesians disagree on how M* converges to M, etc., never on this

point. From his statements, Pinker seems to be aware that M* may have dropped (which is a straight equality) and sort of perhaps we might not

be able to make claims on M which might not have really been dropping.

Now Pinker is excusable. The practice is widespread in social science where academics use mechanistic techniques of statistics without

understanding the properties of the statistical claims. And in some areas not involving time series, the differnce between M* and M is negligible. So I rapidly jot down a few rules before showing proofs and derivations (limiting M to the arithmetic mean). Where E is the expectation

operator under “real-world” probability measure P:

Asymmetry in Inference: Under both negative skewness and fat tails. So I rapidly jot down a few rules before showing proofs and derivations (limiting M to the arithmetic mean). .1. 5. the second 100 years dwarf the first. Where E is the expectation operator under “real-world” probability measure P: 1. dangerously close to 1 Fig 2: The Turkey Surprise: Now 200 years. 6. The distance between different values of M* one gets from repeated sampling of the process (say counterfactual history) increases with fat tails. Counterfactual Property: Another way to view the previous point. And in some areas not involving time series.Long Peace. Fig1 First 100 years (Sample Path): A Monte Carlo generated realization of a process of the “80/20 or 80/02 style” as described by Pinker’s in his book. (Casanova effect) 4. Survivorship Bias Property: E[M*-M ] increases under the presence of an absorbing barrier for the process. They are not “anecdotal”. μ[M*]. large deviations from the mean are vastly more informational than small ones. 3. fat tails tend to mask the distributional properties. that is tail exponent $= 1. 2. Tails Sampling Property: E[|M*-M|] increases in with fat-tailedness (the mean deviation of M* seen from the realizations in different samples of the same process). Power of Extreme Deviations (N=1 is OK): Under fat tails. In other words. Left Tail Sample Insufficiency: E[M*-M] increases with negative skeweness of the true underying variable. The practice is widespread in social science where academics use mechanistic techniques of statistics without understanding the properties of the statistical claims. seen with a longer widow. these are realizations of the exact same process. negative deviations from the mean are more informational than positive deviations. the differnce between M* and M is negligible. (The last two properties corresponds to the black swan problem).nb 3 Now Pinker is excusable.

that is. Nov 11 2014. as we will show in this discussion. bounded on the left and unbounded on the right. For exponents 1 ↵ 2. and so forth until one obtains the result that the top percent of the population will own about 53% of the total wealth. They are also inconsistent under aggregation and mixing distributions. depending on how it is measured. increases in the total sum need to be accompanied by increased sample size of the concentration measurement. but strictly above 1 and severe. remains significant. Dagum. Centre d’Economie de la Sorbonne ⇤ School Let us define the quantile contribution Abstract—Sample measures of top centile contributions to the total (concentration) are downward biased. in other words: ↵ (x) = ↵x↵ x xmin .N. for no other reason than expansion in sample space or aggregate value. +1) is a slowly varying function. thus providing the illusion of structural changes in concentration. though extremely slowly. the in-sample quantile contribution is a biased estimator of the true value of the actual quantile contribution. and also larger population and economic growth will make such a measure converge by increasing over time. There is little difference for small exceedance quantiles (<50%) between the various possible distributions such as Student’s t. We examine the estimation superadditivity and bias under homogeneous and mixed distributions. The core of the problem is that. It makes them particularly unfit in domains with power law tails. unstable estimators. II. it can be shown that under such fat tails. even for very large samples. that is. and such bias is in proportion to the fatness of tails and. as it diverges. so it is very likely that the true ratio of concentration of what Pareto observed. I NTRODUCTION Vilfredo Pareto noticed that 80% of the land in Italy belonged to 20% of the population. extremely sensitive to sample size and concave in accounting for large deviations. was closer to 70%. as the weighted average of concentration measures for A and B will tend to be lower than that from A [ B.R.S. Lévy ↵-stable. 1). or straight Pareto. Accepted Physica A:Statistical Mechanics and Applications This work was achieved through the Laboratory of Excellence on Financial Regulation (Labex ReFi) supported by PRES heSam under the reference ANR-10-LABX-0095. as observed in [6]. In addition. and min x 1 . +1) . q = q E[X|X > h(q)] E[X] where h(q) = inf{h 2 [xmin . that is: P(X > x) ⇠ L(x) x ↵ (1) where L : [xmin . where the random variable X 2 [xmin . used in most academic studies. The self-similarity at the core of the property of power laws [1] and [2] allows us to recurse and reapply the 80/20 to the remaining 20%. The problem is acute for ↵ around. for very fat tailed distributions. +1) ! (0.[4] Singh-Maddala distribution [5]. Bias and Convergence 1) Simple Pareto Distribution: Let us first consider ↵ (x) the density of a ↵-Pareto distribution bounded from below by ↵ 1 xmin > 0. and vice-versa. the law of large numbers operates. New York University † Riskdata & C. say wealth. for the class one-tailed fat-tailed random variables. It looks like such a measure of concentration can be seriously biased. defined as limx!+1 L(kx) L(x) = 1 for any k > 0. the share of the top percentile. hence changes year-on-year would drift higher to converge to such a level from larger sample.[3]. more complete samples resulting from technological progress.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES On the Super-Additivity and Estimation Biases of Quantile Contributions Nassim Nicholas Taleb⇤ . E STIMATION F OR U NMIXED PARETO -TAILED D ISTRIBUTIONS Let X be a random variable belonging to the class of distributions with a "power law" right tail. Labex ReFi. Raphael Douady† of Engineering. Paris. especially for low values of the exponent. n 1X ˆ h(q) = inf{h : n i=1 x>h q} We shall see that the observed variable bq is a downward biased estimator of the true ratio q . can be expressed. I. A. its "natural" estimator bq ⌘ q th percentile . thus both giving birth to the power law class of distributions and the popular saying 80/20. for. as shown in this article. P(X > h) q} is the exceedance threshold for the probability q. the one that would hold out of sample. For a given sample (Xk )1kn . In fact. These estimators can vary over time and increase with the population size. for ↵ = 1. total as Pn Xi ˆ i=1 Xi >h(q) Pn bq ⌘ i=1 Xi ˆ where h(q) is the estimated exceedance threshold for the probability q : Final version.

"small" meaning in practice 108 . X). a value chosen to be compatible with practical economic measures. when introducing one extra sample because of a slight increase in the expected value of ˆ the estimator h(q). the estimator is extemely sensitive to "small" samples. the new value S(n) Ah (n) h (n)+Xn+1 h bh (n + 1) ⇡ AS(n)+X = 1 S(n)+X . The intuition behind the estimation bias of q by bq lies in a difference of concavity of the concentration measure with respect to an innovation (a new sample value). where the exponent depends on ↵ and on the . which we assume to be below h(q). so that we have P(X > x) = x ↵ for some > 0. 2 so that we have q = h(q) .591506 0. so that S(n) frozen threshold h. although this effect is rather negligible).160244 0. one has: @2 bh (X t Y ) 0. ˆ ).117917 0. We also define the n-sample estimator: Pn X Pn Xi >h i bh ⌘ i=1 X i=1 i where Xi are n independent copies of X. such as the wealth distribution in the world or in a particular country.0601528 0.405235 0. 2) General Case: In the general case. even in very large sample. namely of the order of n /ln n. The concavity of the estimator constitutes a upper bound for the measurement in finite n.555997 0.367698 0. The fatter the right tail of the distribution. We ran up to a trillion simulations across varieties of sample sizes. so that uncertainty on Xn+1 reduces its value. though not under a mixture distribution for ↵. which is lower than the estimated exponents one can find in the literature – around 2 – is. as we shall se later. the cutpoint x of exceedance is h(q) = xmin q 1/↵ and we have: R1 ✓ ◆ x (x)dx ↵ 1 h(q) 1 ↵ h(q) q = R 1 = =q ↵ (2) xmin x (x)dx x min If the distribution of X is ↵-Pareto only beyond a cut-point xcut .516415 0.1 In such a case. which is n+1 h n+1 h now concave in Xn+1 . following [7]. q)n b(q)(↵ 1) where constants b(q) and c(↵. clipping large deviations. Table I shows the bias of bq as an estimator of q in the case of an ↵-Pareto distribution for ↵ = 1. hence this discussion focuses on the famed "one percent" and on low values of the ↵ exponent. whether Pn it falls below or above the threshold. Let Ah (n) = i=1 Xi >h Xi and Pn Ah (n) S(n) = bh (n) = and assume a i=1 Xi . TABLE I: Biases of Estimator of = 0.0461397 In view of these results and of a number of tests we have performed around them. q) need to be evaluated.581384 0. because of a higher concavity with respect to the variable. We face a different situation from the common small sample effect resulting from high impact from the rare observation in the tails that are less likely to show up in small samples. and also of a higher variability (whatever its measurement) of the variable conditionally to being above the threshold than to being below.1. The competition between these two opposite effects is in favor of the latter. Y = Xn+1 an extra single random observation. we can conjecture that the bias q bq (n) is "of the order of" c(↵. a lower estimate which cannot be excluded from the observations. For a better. We remark i=1 Xi + Y that. In practice. We have in fact the following: Proposition 1. whenever Y > h. It is also weaker outside the top 1% centile. let us fix the threshold h and define: h = P (X > h) E[X|X > h] E[X X>h ] = E[X] E[X] 1 This value. estimating the theoretical tail contribution q (ˆ ↵. even a sample size of 100 million remains severely biased as seen in the table.0931362 0.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES ↵ P(X > x) = xmin . and Pn X >h Xi + Y >h Y define: bh (X t Y ) = i=1 Pni . The value is convex S(n) + Xn+1 in Xn+1 so that uncertainty on Xn+1 increases its expectation. or at least its ratio to the expectation of X. then we still have h(q) = q 1/↵ and q = ↵ ↵ 1 E [X] q ↵ 1 ↵ The estimation of q hence requires that of the exponent ↵ as well as that of the scaling parameter . If a new sample value Xn+1 < h then the Ah (n) new value is bh (n + 1) = . the convergence is extremely slow. even with a proper estimator of ↵ and . At variance.458449 0. Let X = (X)ni=1 a random sample of size n > 1q .593667 STD across MC runs 0. one would need to use⇣a different ⌘ path: first estimating the distribution parameters ↵ ˆ . While 0. but the rather slow convergence of the estimator and of its standard deviation to 0 makes precise estimation difficult.539028 0.606513 0. @Y 2 ˆ X tY ) This inequality is still valid with bq as the value h(q. if the new sample value Xn+1 > h.0853593 0. the stronger the effect. and becomes weak in the neighborhood of 2 for a constant ↵.485916 0. including developped ones. we find E [Ah (n)] that E [b h (n)] = h (note that unfreezing the E [S(n)] ˆ threshold h(q) also tends to reduce the concentration measure estimate. Under these assumptions. adding to the effect. the contribution of very large rare events to q slows down the convergence of the sample estimator to the true value. Overall.01 ⇡ 0.575262 0. ˆ and only then.657933 From 1012 Monte Carlo Realizations b(n) b(103 ) b(104 ) b(105 ) b(106 ) b(107 ) b(108 ) Mean Median 0.657933. Falk [7] observes that. which leads to problems of aggregation as we will state below in Theorem 1. a bias which goes away by repetition of sample runs. whatever the value of ↵ and q. unbiased estimate. ˆ doesn’t depend on the particular value of Y > h(q. Simulations suggest that b(q) = 1. Naturally the bias is rapidly (and nonlinearly) reduced for ↵ further away from 1.

We define A = i=1 20 40 60 80 100 Y Fig.. . .. . the allocation of the total number of observations n between i and j does not matter so long as the total n is unchanged. . . Therefore i2J 00 Xi A S S0 0 00 + and we have: S 00 S 00 0 S S E [00 ] E 00 + E 00 0 S S . . and let S1 . random variables. .v. Then we have: m X ni E [b q (N )] E [b q (Ni )] n i=1 In other words. .. X 0 = (X10 .... and is exacerbated by high variance.m+n}. . . 1) and (X1 . measured by the Hellinger distance. and A0 = mq X 0 X[i] where i=1 0 X[i] is the i-th largest value of (X10 . We set S = Xi and 0. ..EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES kHSXi +YL 3 Theorem 1. .[Nm of respective sizes n1 .95 0. . .n}. In other words. .. in m m sub-samples of size ni each for a total of n = i=1 ni .. . We observe that: A= max X J⇢{1. If J ⇢ {1... . We also set (m+n)q S 00 = S + S 0 and A” = X 00 00 X[i] where X[i] is the i-th i=1 tolerance of the actual distribution vs. Let q 2 (0. . 2: Effect of additional observations on . Xm . Xm ). . . For simplicity..90 0.. |J 0 | = 00 0 0 qn. X1 . largest value of the joint sample (X1 .70 0. produces a downward biased estimate of the concentration measure of the full sample. .. Xn0 ) . . 2 The same concavity – and general bias – applies when the distribution is lognormal. . Xn0 ) and 00 0 0 X = (X1 . . . Partition the n data into m sub-samples Pm N = N1 [.. . an area of order 1/n mq X i=1 X[i] where X[i] is the i- i=1 th largest value of (X1 . . Xm ) and (X10 . . Xn0 ).2 Next we prove that global concentration as measured by bq on a broad set of data will appear higher than local concentration.65 20 000 40 000 60 000 80 000 Y 100 000 Fig. P (X)ji . ! 0 as ↵ ! 1. . A N I NEQUALITY A BOUT AGGREGATING I NEQUALITY For the estimation of the mean of a fat-tailed r. a theoretical Pareto. with i=1 ni = n.. The q-concentration measure for the samples X = (X1 . . nm .. m + n} . where we have denoted Xm+i = Xi0 for i = 1 .. . . Xn ) are: A A0 A00 0 = 0 00 = 00 S S S We must prove that he following inequality holds for expected concentration measures: 0 S S 00 E [ ] E 00 E [] + E 00 E [0 ] S S = III. hence A + A = .80 0. . . weighted by the total sum of each subsample.. .|J i2J 0 Xi and P 00 A = maxJ 00 ⇢{1.626 0. making the convergence really slow for low values of ↵. Xn0 ) be two samples of positive i.. Proof. 1: Effect of additional observations on kHSXi +YL If we further assume that the distribution of variables Xj is the same in all the sub-samples. .d.75 0. . . n. . Xm ). P then J = J00 [ J has cardinal m + n.i. for instance. i=1 Then we have: m X Si E [b q (N )] E E [b q (Ni )] S i=1 0. . .|J 00 |=q(m+n) i2J 00 Xi . and S = Si be that over the whole sample.. m} . |J| = ✓m and J 0 ⇢ {m + 1. .622 S0 = n X Xi0 . . . similarly A0 = maxJ 0 ⇢{1.624 0. An elementary induction reduces the question to the case of two sub-samples. we claim that the estimation bias when using bq (n) is even increased when dividing the sample into sub-samples and taking the weighted average of the measured values bq (ni ). . would give a bq higher than the average measure of concentration across countries – an "inequality about inequality".. we assume that m X both qm and qn are integers. averaging concentration measures of subsamples.m} |J|=✓m i2J Xi P 0 0 |=qn and.. .. Here the allocation of n samples between m sub-samples does matter because of the concavity of . . SX m be the sum of variables over each subm sample. In particular. X10 . whatever the particular sample.85 0. so aggregating European data. . we can see convexity on both sides of h except for values of no effect to the left of h. the Xi ’s having distributions p(dx) and the Xj0 ’s having distribution p0 (dx0 ). Xm .

We recall that p0 (ds0 ) is the distribution of S 0 and denote q(dt) that of T . The above theorem shows that. then we identically get for 0 : 0 0 0 0 S A S A E 00 0 = E 00 E 00 E S S S S0 hence we will have: E [00 ] E Theorem 2. thanks to the law of large numbers. X)] E E [b q (!i n. They are also independent of S 0 . We have: S E 00 = S ZZ a+b a pA (t. B and S .d.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES Let us now show that: S A E 00 = E 00 S S E whereas the n-sample ✓-concentration measure is bh (n) = A(n) . S E T. db) be the distribution of A and B respectively. Conditionally to T . . Similarly. This ends the proof of the theorem. hence their ratio also converges almost surely to h . 1).s. We remind the theoretical h-concentration measure. this ratio is bounded by 1. and we have: S S A E 00 E 00 E S S S Proof. X) i=1 S(!i n. Let X be a positive random variable and h 2 (0. the cdf of X↵ ⇣ normalize ⌘ is F↵ (x) = 1 x xmin ↵ and we have: q (X↵ ) = q ↵ 1 ↵ and 2 ↵ 1 (log q) d2 q (X↵ ) = q ↵ >0 2 3 d↵ ↵ Hence q (X↵ ) is a convex function of ↵ and we can write: q (X) m X i=1 !i q (X↵i ) q (X↵¯ ) Pm where ↵ ¯ = i=1 !i ↵. 1 n S(n) converges almost surely and in probability to E [X] and n1 A(n) converges almost surely and in probability to E [X X>h ] = P (X > h)E [X |X > h ]. defined as: P (X > h)E [X |X > h ] E [X] m X q (X) 0 h = a. da) and pB (t. each ratio If the two samples have the same distribution. we have: m X S(!i n. we have: S A 0 E T. For any n 2 N. i S0 n E S 00 = m+n . except that its tail decays like a power low with unknown exponent. in this case. An unbiased estimation of the . the parameter ↵ represents the tail exponent of the distribution. Suppose now that X is a positive random variable with unknown distribution. Moreover. A and B are i=1 independent: A is a sum if m✓ samples constarined to being above T . Lebesgue dominated convergence theorem concludes the argument about the convergence in probability. M IXED D ISTRIBUTIONS F OR T HE TAIL E XPONENT Consider now a random variable X. we have: E [b h (n)] < h and S S E [] + E 00 E [0 ] S 00 S S A = n!+1 i=1 Xi Xi <T . then we have: m n E [00 ] E [] + E [0 ] m+n m+n ⇥S ⇤ m Indeed. hence conditionally to T . X) surely to !i respectively. therefore we have the following convexity inequality: When n ! +1. X↵i )] S(n. IV. so that A = Xi Xi T and let B = m X 4 !i q (X↵i ) i=1 The case of Pareto distribution is particularly interesting. yielding the result. variables with the same distribution as X. it is valid for the unconditional expectation. But we also have E SS 00 = ⇥ ⇤ ⇥ ⇤ 1 1 h= (m + n)E SX00 therefore E SX00 = m+n . The above corrolary shows that the sequence nE [b h (n)] is super-additive. Let pA (t. h 00 i ⇥ S ⇤ i=1 i ⇥ X ⇤ hence E S 00 = mE S 00 . S A E S 00 S If this is the case. S = E T. X↵i ) converges almost S(n. hence E [b h (n)] is an increasing sequence. given T = t. we observe that E S 00 = m+n . B. while B is the sum of m(1 ✓) independent samples constrained to being below T . X↵i ) E [b q (n. and in probability lim bh (n) = h 0 Let T = X[mq] be the cut-off point (where [mq] is the m X integer part of mq). where A(n) and S(n) are defined as above for an nS(n) sample X = (X1 . Here. . B and S 0 . the distribution of which p(dx) is a mixture of parametric distributions with Pm different values of the parameter: p(dx) = i=1 !i p↵i (dx). S 0 0 A+B+S A+B This inequality being valid for any values of T . On the other hand. t and s0 . If we expectations to 1. S 0 S 00 A + B + S0 A+B A 0 E T. Xn ) of i. a ! a+b+s and a ! a+b 0 are two increasing functions of the same variable a. in this Pm case. B. . da) pB (t. db) q(dt) p0 (ds0 ) a + b + s0 a + b a+b a For given b.i. . A typical n-sample of X can be made of ni = !i n samples of X↵i with distribution p↵i . B. B. Indeed S = X and the Xi are identically distributed.

would lead to a downward biased estimate of q . we note a fallacious inference about the concentration of damage from wars from a bq with minutely small population in relation to the fat-tailedness. a distribution of possible true values around some average). this inequality also applies in the case of a mixture of distributions with a power decay. Robust methods. rejection of such theories also ignored the size effect. unlike other parameters. The convexity inequaly writes as follows: ⌘ 1 1 1 ⇣ 1 ↵+1 q ( ↵ ¯ ) = q 1 ↵¯ < q + q1 ↵ 2 So in practice. So it appears that both Pinker and the literature cited for the quantitative properties of violent conflicts are using a flawed methodology. Because the concentration measure only depends on the tail of the distribution.3 3 Accumulated k Hn=104 L 1. size or companies. Indeed. A L ARGER T OTAL S UM IS ACCOMPANIED BY I NCREASES IN bq There is a large dependence between the estimator bq and n X the sum S = Xj : conditional on an increase in bq the j=1 expected sum is larger. it can be shown that.7 0. May 23. additional wealth means more measured inequality. say. wealth concentration when it has been there all along at the level of the process.8 0.6 0. 3: Effect of additional wealth on ˆ Even the estimation of ↵ can be biased in some domains where one does not see the entire picture: in the presence of uncertainty about the "true" ↵. To show how unreliable the measures of inequality concentration from quantiles.4 0. [10]: "(Wars) followed an 80:2 rule: almost eighty percent of the deaths were caused by two percent (his emph. the one to use is not the probability-weighted exponents (the standard average) but rather the minimum across a section of exponents [6]. effectively making the dialogue on inequality uninformational statistically. since the distribution is of the class of fat tails under consideration. and dominated by the second 1 5 wealth is typically thicker tailed than income.. 5 Using Richardson’s data. an adjustment would rapidly invalidate such claims that violence from war has statistically experienced a decline. What is worse. we can easily get a historical illusion of rise in. one that produces a severe bias. as in [9]. would produce similar results as value of ↵ much closer ro 1. It did not escape our attention that some theories are built based on claims of such "increase" in inequality.5 ↵j The slightest uncertainty about the exponent increases the concentration index. V. as we used in the previous section. 0. Furthermore claims about the mean become spurious at low exponents. epidemics.) VI.25. see [8]. in the thesis by Steven Pinker [10] that the world is becoming less violent. in other words.) of the wars". and an expansion in the size of the unit measured can be part of the explanation.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES exponent. with necessarily some amount of uncertainty (i.0 0. alas. as the centile estimation has extremely large biases with fat-tailed wars. Under such dynamics. bq and S are positively correlated. A. without taking into account the true nature of q . but in small units or subsections we will observe a lower q . as in Equation 1: P(X > x) ⇠ N X !i Li (x)x (3) j=1 1 ↵+ (ln(q) 2(↵+ )) order effect ln(q)q . is quite absurd to assume that additional wealth will arise from the bottom or even the middle. .e. consider that a standard error of 0. C ONCLUSION AND P ROPER E STIMATION OF C ONCENTRATION Concentration can be high at the level of the generator.9 0. etc. Simply q (↵) is convex. 4 Financial Times. For the case in which the random variables under concern are wealth. For instance. an effect that is exac(↵+ )4 erbated at lower values of ↵. by countering with data of a different sample size.3 in the measurement of ↵ causes q (↵) to rise by 0. The very methodology of using concentration and changes in concentration is highly questionable. So examining times series. and promulgating theories about the "variation" of inequality without reference to the stochasticity of the estimation and the lack of consistency of q across time and sub-units. the maximum is of the same order as the sum. an estimated ↵ ¯ of around 3/2.4 The mistake appears to be commonly made in common inference about fat-tailed data in the literature. 2014 "Piketty findings undercut by errors" by Chris Giles. Robust methods and use of exhaustive data We often face argument of the type "the method of measuring concentration from quantile contributions ˆ is robust and based on a complete set of data". One must not perform analyses of year-on-year changes in bq without adjustment. as shown in theorem 1. sometimes called the "half-cubic" exponent. One can get an actual estimate of this bias by considering an average ↵ ¯ > 1 and two surrounding values ↵+ = ↵ + and ↵ = ↵ . we observe as in Figure 3 such conditional increase.5 Owing to the fat-tailedness of war casualties and consequences of violent conflicts.3 60 000 80 000 100 000 120 000 Wealth Fig. (The same argument can be applied to wars.

vol. no. Tech. no. Rep. “Inequality measures between income distributions with applications. “Silent risk: Lectures on probability. “The evolution of top incomes: a historical and international perspective. Taleb. vol. 6. Income distribution models. vol. Falk et al. [7] M. pp. Such methods of adjustment are less biased and do not get mixed up with problems of aggregation –they are similar to the "stochastic volatility" methods in mathematical finance that consist in adjustments to option prices by adding a "smile" to the standard deviation. Piketty and E. 1963. Albert Shiryaev. 1.” Available at SSRN 2392310. 1791–1803. vol 1.” National Bureau of Economic Research. vol. no. use Pareto interpolation for unsufficient information about the tails (based on tail parameter). methods that are extrapolative and nonconcave.. 111–115. a method of direct centile estimation is still linked to a static and specific population and does not aggregage. [10] S. as the methods for fitting Poisson jumps are interpolative and have proved to be easier to fit in-sample than out of sample. 2. in addition to the centile estimation problem.7 8 ACKNOWLEDGMENT The late Benoît Mandelbrot.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES tend to fail with fat-tailed data. [5] S. Take an insurance (or. . [6] N. “Capital in the 21st century. Accordingly. an anonymous referee. 1. pp. Bruno Dupire. no. Tech. Maddala. [8] X. 79–106. such techniques do not allow us to make statistical claims or scientific statements about the true properties which should necessarily carry out of sample. but we focus on the powerlaw case in the application. Branko Milanovic. [4] ——. 6 R EFERENCES [1] B.. such a method retains a significant bias. N. the problem here is worse: even if such "robust" methods were deemed unbiased. Piketty. see [6]. 23. Penguin. better. 2015.” International Economic Review. integrating the functions of ↵ across the various possible states. 1983. Mandelbrot. rather the exposure to tail (and other) events. How Should We Measure Concentration? Practitioners of risk managers now tend to compute CVaR and other metrics. we mean the built-in extension of the tail in the measurement by taking into account realizations outside the sample path that are in excess of the extrema observed.” International Economic Review. analyses that take into account the stochastic nature of the performance. B. But. “The stable paretian income distribution when the apparent exponent is near two. 1995. “On testing the extreme value index via the pot-method. Rep. 46. Dominique Guéguan. Here it would be "stochastic alpha" or "stochastic tail exponent"6 By extrapolative. the staff at Luciano Restaurant in Brooklyn and Naya in Manhattan. 1978. The better angels of our nature: Why violence has declined. “A function for size distribution of incomes: reply. [3] C. 7 Even using a lognormal distribution. pp. 48. as we saw in our extension to Theorem 2 and rederiving the corresponding . “The pareto-levy law and the distribution of income. the late Marc Yor. 8 We also note that the theorems would also apply to Poisson jumps. [9] T. The same with wars: we do not estimate the severity of a (future) risk based on past in-sample historical data. Singh and G. 7. more rigorously. works to some extent as a rise of the standard deviation extrapolates probability mass into the right tail. no. 2008.” Econometrica. pp. 4. or. Wiley Online Library. some authors such as [11] when dealing with censored data. Felix Salmon. filling-in the bracket with conditional average bracket contribution. reinsurance) company. taking the one closer to the lower bound of the range of exponents. such as the information from the ↵ exponent. see [6]. This difference between "accounting" (deterministic) and "economic" (stochastic) values matters for policy making.” Econometrica. [2] ——. 2013–2035. The "accounting" profits in a year in which there were few claims do not reflect on the "economic" status of the company and it is futile to make statements on the concentration of losses per insured event based on a single year sample. Dagum. The "accounting" profits are not used to predict variations year-on-year. 1960. [11] T. 6 Also note that. 2. by fitting the scale parameter. vol. particularly under fat tails.” 2014. Pinker. Gabaix. 1980. in addition. Saez. “Power laws in economics and finance..” National Bureau of Economic Research. which is not the same thing as using full power-law extension. in proportion to the variability of the parameter representing volatility and the errors in its measurement. 2006. 2011.” The Annals of Statistics.

. . . . . . . . . . . . . . . . . . 2 VII References 11 Y Ou observe data and get some confidence that the average is represented by the sample thanks to a standard metrified "n". . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . New York University This is extracted from Chapter 6 of Silent Risk. . . . . . . . III-A2 Speed of convergence . . . . . III-A1 Relative convergence . . . . . . . . . . . . . . . . . . 10 10 10 10 FT I . . . . . . . . . . . . . . . . . . . . . . III-B Stochastic Alpha or Mixed Samples . . 1 . 4 4 5 5 5 IV Symmetric NonStable Distributions in the Subexponential Class IV-A Symmetric Mixed Gaussians. . . . . . . . . . . VII-B Derivations using explicit E(|X|) . . . . . . . . . . . . . . . . . . 9 9 9 9 Acknowledgement VII-A Cumulants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D RA . . . Model Uncertainty The practical import is that model uncertainty worsens inference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV-B Half cubic Student T (Lévy Stable Basin) . . Stochastic Mean . . . . . . . . . . 9 9 9 VI Asymmetric Distributions in the Superexponential Class VI-A Mixing Gaussian Distributions and Poisson Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Now what if the data were fat tailed? How much more do you need? What if the model were uncertain –we had uncertainty about the parameters or the probability distribution itself? Main Results In addition to explicit extractions of partial expectations for alpha stable distributions. in a quantifiable way. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C ONTENTS -A The "Pinker Problem" . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . It has been slowed down by the author’s tinkering with explicit expressions for partial expectations of asymmetric alpha-stable distributions and the accidental discovery of semi-closed form techniques for assessing convergence for convolutions of one-tailed power laws. . . . one main result in this paper is the expression of how uncertainty about parameters (in terms of parameter volatility) translates into a larger (or smaller) required n. . . . . . . . . . . . . . . . . . . . . . . . . . . . . IV-C Cubic Student T (Gaussian Basin) . . . . .EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES Fat Tails. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Model Uncertainty and the Law of Very Large Numbers Nassim Nicholas Taleb School of Engineering. . . . . . . . . . . . . . . . . . . VI-B Skew Normal Distribution . . . . . . . . . . . . . . . . . . . V-B The Lognormal and Borderline Subexponential Class . . . . . . . . . . . . This chapter is in progress and comments are welcome. . . VI-C Super-thin tailed distributions: Subgaussians . 2 The problem of Matching Errors II Generalizing Mean Deviation as Partial Expectation III Class of Stable Distributions III-A Results . . . . VII-C Derivations using the Hilbert Transform and = 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 6 7 8 V Asymmetric NonStable Distributions in the Subexponetial Class V-A One-tailed Pareto Distributions . . . . . . . .

We get most of the intuitions from closed-form and semi-closed form expressions working with: • stable distributions (which allow for a broad span of fat tails by varying the ↵ exponent. say n = 30 or some other such heuristically used number. or be too unreliable (power laws with exponents > 2. etc. alas. Assuming we are confortable with such a number of summands. RA A. T HE PROBLEM OF M ATCHING E RRORS By the weak law of large numbers. consider a sum P of random variables X1 . that is finite variance but infinite kurtosis).EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES 2 FT Fig. But this is a very general problem with the (irresponsible) mechanistic use of statistical methods in social science and biology. 1: How thin tails (Gaussian) and fat tails (1< ↵ 2) converge to the mean. . And the idea is that we live with finite n. The "Pinker Problem" It is also necessary to debunk a fallacy: we simply do not have enough data with commonly discussed fat-tailed processes to naively estimate a sum and make series of claims about stability of systems.. How can people compare averages concerning street crime (very thin tailed) to casualties from war (very fat tailed) without some sample adjustment?1 Perhaps the problem lies at the core of the law of large numbers: the average is not as "visible" as other statistical dimentions. along with the asymmetry via the coefficient • stable distributions with mixed ↵ exponent. pathology of people reacting to risks. Instability of Mean Deviation Indexing with p the property of the variable X p and g for X g the Gaussian: ( ! !) np ng X X Xip mp Xig mg np : E =E np ng (1) 1 The Pinker Problem A class of naive empiricism. there is no sound statistical procedure to derive the properties of a powerlaw tailed data by estimating the mean – typically estimation is done by fitting the tail exponent (via.. • other symmetric distributions with fat-tails (such as mixed Gaussians. Xn independent and identically distributed with finite mean m. then n1 1in Xi converges to m in probability. Pinker [1] about the drop of violence that is based on such statistical fallacies since wars –unlike domestic violence –are fat tailed. It has been named so in reference to sloppy use of statistical techniques in social science and policy making. A surprising result: for the case with equivalent tails to the "Pareto 80/20 rule" (a tail exponent ↵ = 1. etc. based on a theory promoted by the science writer S. the Hill estimator or some other method). decisions. are based on. say. Take a certain sample size in the conventional Gaussian domain. how much larger (or smaller) n does one need for the same error under a different process? And how do we define errors in the absence of standard deviation which might not exist (power laws with exponents close to 2). X2 . D I. or simple stochastic volatility) More complicated situations entailing more numerical tinkering are also covered: Pareto classes.16) one needs 1011 more data than the Gaussian. lognormal. as n ! 1. yet it remains that many articles make comparisons about the mean since it is what descriptive statistics and. or dealing with extrema. It is strange that given the dominant role of fat tails nobody thought of calculating some practical equivalence table. that is E[Xi ] < 1. Gamma-Variance Gaussians...

they are extremely unstable. as in 106 realizations for an average with 100 summands. a stable distribution with tail exponent at 32 matched to exactly the same mean deviation as the Gaussian will deliver measurements of mean deviation 1. where K can be any arbitrary level.0 FT 1. Using mean absolute deviation for "volatility". one uses standard deviations of the observations to establish the sufficiency of n. The difference is not trivial. For instance. II. when they exist.6 1.5 3. we can be expected observe maxima > 4000 ⇥ the average while for a Gaussian we can hardly encounter more than > 5 ⇥. as a function of the tail exponent. 2: The ratio of cumulants for a symmetric powerlaw.7 Fig. worse. particularly in cases where kurtosis is infinite. and its mean deviation "volatility of volatility" expressed in the L1 norm. or C1 and C2 cumulant: C1 = E(|X m|) C2 = E (|X E(|X m|)|) D We can compare that matching mean deviations does not go very far matching cumulants. a risk function) in L1 as mean absolute deviation from the observed average. E(|X|) = E(X|X>µ ) + E( X|X<µ ) = 2E(X|X>µ ). The "volatility of volatility".0 1. in statistical inference. or the dispersion around the mean deviation increases nonlinearly as the tails get fatter. Let ✓ be the Heaviside theta function.0 1 RA And since we know that convergence for the Gaussian happens at speed n 2 . as shown in figures .5 2. We are expressing in Equation 1 the expected error (that is.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES C2 C1 3 ( ) 1. Using mean deviations of the samples (when these exist) doesn’t accommodate the fact that fat tailed data hide properties. to accommodate absence of variance –but assuming of course existence of first moment without which there is no point discussing averages. a sum of Gaussian variables will have its extreme values distributed as a Gumbel while a sum of fat tailed will follow a Fréchet distribution regardless of the the number of summands. the dispersion of the distributions of the mean deviations (and their skewness) would be such that a "tail" would remain markedly different in spite of a number of summands that allows the matching of the first order cumulant. G ENERALIZING M EAN D EVIATION AS PARTIAL E XPECTATION It is unfortunate that even if one matches mean deviations. as in powerlaw with tail exponent > 3. Typically. by convoluting the Fourier transforms: Z 1 @ E(X|X>K ) = i (t u) ✓ (u)du|t=0 @t 1 Mean deviation becomes a special case of equation 2.4 times as unstable as the Gaussian. 1. (2) . But in fat tailed data standard deviations do not exist. or.(see Appendix 1) Further. So we can match the special part of the distribution. we can compare to convergence of other classes.4 2. Since sgn(x) = 2✓(x) 1 Z 1 2ieiKt ✓ (t) = eitx (2✓(x K) 1) dx = t 1 And the special expectation becomes. Let (t) be the characteristic function of the random variable.5 1. the expectation > K or < K.

. 1]. .i.d. We write. . X ↵ = (Y ↵ under the stable distribution above. . but not fully. be the centered variable with a mean of zero. 0. But for convention we retain the same symbol as it corresponds to tail exponent but use it differently in more general non-stable power law contexts. . FT 0<↵<2 A. From Equation 2: Z 1 ⇣ ⇣ ⇡↵ ⌘ ⌘ ⇣ ⇣ ↵ sgn(u) exp |u | 1 RA E(X|X>K ) = 1 2⇡ with explicit solution: ↵ 1 ↵ |u| E(↵. 1). Samorodnitsky and Taqqu[4]: ⇣ ⇣ ⇣ ⇡↵ ⌘ ⌘⌘ ↵ |t | 1 i tan sgn(t) ↵ = exp iµt 2 which. promoted in Samorodnitsky and Taqqu [4] and Zolotarev[3]: !! ✓ ◆ 1 ⇣ ⌘ ⌘ 2↵ tan 1 tan ⇡↵ 1 1 ⇣ 2 2 ⇡↵ 2 E(|X|) = 2 1 tan +1 cos ⇡ ↵ 2 ↵ Which allows us to prove the following statements: . ) represents the stable distribution with tail index ↵ 2 (0. µ. Results Let X ↵ 2 S. using the expression for ↵ > 1. [We also note that the convergence is not clear across papers[2] but this doesn’t apply to symmetric distributions. . 2]. symmetry parameter 2 [0. C LASS OF S TABLE D ISTRIBUTIONS Assume alpha-stable the class S of probability distribution that is closed under convolution: S(↵.2 . The characteristic function (t) of a variable X ↵ with scale will be. and scale parameter 2 R+ . See Zolotarev[3]. random variates Xi the distribution function of which FX (x) has asymptotes 1 cx ↵ as x ! +1 and d( x) ↵ as x ! 1 weakly converges to the stable distribution c d .] Note that the tail exponent ↵ used in non stable cases is somewhat. 0) = ↵ 2 1 + i tan 1 ⇡↵ and semi-explicit generalized form: ⇣ ↵ 1 1 + i tan ↵ E(↵.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES 4 III. . E(↵. for an n-summed variable (the equivalent of mixing with equal weights). µ. different for ↵ = 2. K) ⌘ E(X ↵ |X ↵ >K ) 1 +1 k ↵ ✓ ⇡↵ 2 1/↵ ⇣ + 1 i tan i tan 2 ⌘ ⌘ sgn(u) + iKu du ⇣ ⇡↵ ⌘⌘1/↵ ◆ 2 (3) . ⌘ ( 1)k 1 + i tan 2⇡ ⇣ ⇡↵ ⌘ k 1 k! ⇡↵ 2 k 1 ↵ + 1 i tan ⇡↵ 2 k 1 ↵ ◆ (4) Our formulation in Equation 4 generalizes and simplifies the commonly used one from Wolfe [5] from which Hardin [6] got the explicit form. the Gaussian case where it ceases to be a powerlaw –the main difference is in the asymptotic interpretation. c+d We note that the characteristic functions are real for all symmetric distributions. K) = i K k k+↵ 1 ↵ D 1 X k + ✓ 1 ↵ ⇡↵ 2 2 ◆ ✓⇣ 1/↵ 1 + i tan + 1 tan2 ⇡↵ 2 k=1 ⇣ ⇡↵ ⌘⌘1/↵ 2 i tan 2⇡ 2 µ) . The Generalized Pn Central Limit Theorem gives sequences an and bn such that the distribution of the shifted and rescaled sum Zn = ( i Xi an ) /bn of n i. location parameter µ 2 R. becomes: ⇣ ⇣ ⇡↵ ⌘ ⌘⌘ ↵⇣ 1 n↵ t 1 i tan sgn(t) ↵ (t) = exp iµnt 2 S(^↵.

632 3 2 567 613 737 13 8 165 171 186 7 4 75 77 79 15 8 44 44 44 2 30. (1) the first derivative of the Poly Gamma function (x) = 0 [x]/ [x]. ↵ classes "nonstable".88 ⇥ 106 11 8 5.952 1. Proof. assuming so and so. =± 1 2 - n↵=±1 - 6.027 6.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES 5 1) Relative convergence: The general case with 6= 0: for so and so. (Sketch) From the characteristic function of the stable distribution. (precisions) etc. For so and so ⇠(X↵¯ ) where ↵ ¯= Pm i=1 !i ↵i and Pm i=1 m X !i ⇠(X↵i ) i=1 !i = 1. Stochastic Alpha or Mixed Samples Define mixed population X ↵ and ⇠(X ↵ ) as the mean deviation of . This is not the case for other Proof. 30 30 n↵ Fughedaboudit 9 8 n↵ D RA ↵ 1 1 Remark 1... 2). and Hn the nth harmonic number: ✓ ◆ ✓ ✓ ◆ ⇣ ⌘⇣ ⌘◆ 1 @2⇠ 2 ↵ 1 ↵ 1 (1) ↵ 1 1 + log(n) + 1 + log(n) + = n + H 2↵ H ↵ ↵ @↵2 ⇡↵4 ↵ ↵ . The ratio mean deviation of distributions in S is homogeneous of degree k . or how many for equivalent ↵-stable distribution. 2] m↵ i E n↵ ! ng ↵ 1 ↵ n↵ X X↵ C C A (6) ! 1 ↵↵ m↵ ! (7) 1 FT kn↵ X X↵ p 1 ◆ 1 1 ↵↵ /E i n↵ = k↵ 1 (8) Table I shows the equivalence of summands between processes.86 ⇥ 1014 5 4 574.002 8. 1 .09 ⇥ 1012 2.5772. The Gaussian case is the ↵ = 2. Other distributions need to converge to the basin S. Proposition 1. For the case with equivalent tails to the 80/20 one needs 1011 more data than the Gaussian. A sketch for now: 8↵ 2 (1. where is the Euler-Mascheroni constant ⇡ 0.634 895. B. TABLE I: Corresponding n↵ . ✓ ✓ ◆ ✓⇣ ⇣ ⇡↵ ⌘⌘ ↵1 ⇣ ⇣ ⇡↵ ⌘⌘ ↵1 ◆◆ ↵ ↵ ↵ ↵ 1 p 1 ↵ 2 2↵ ↵ 1 n↵ = 2 ⇡ ng 1 i tan + 1 + i tan ↵ 2 2 (5) with alternative expression: n↵ = ⇡ 2 Which in the symmetric case ↵ 2↵ 0 1 2 ⇡↵ 2 2 B sec B @ /↵ p sec ng ✓ tan 1 (tan( ⇡↵ 2 )) ↵ ↵ 1 ↵ = 0 reduces to: n↵ = ⇡ ↵ 2(1 ↵) 2) Speed of convergence: 8k 2 N+ and ↵ 2 (1.8 ⇥ 1013 1..

0 2. IV. With the perfectly symmetric case µ = 0 and sampling with equal . which makes it convenient to simulate fattailedness. keeping ↵ < 2 as it would no longer converge to the Stable basin. But mixing means has the opposite effect.0 Fig. and keeping the average µ ± with probability 1/2 each. S YMMETRIC N ON S TABLE D ISTRIBUTIONS IN THE S UBEXPONENTIAL C LASS A.5 -0.5 = 3/2 1. 4: Mixing distributions: the effect is pronounced at lower values of ↵. The implication is that one’s sample underestimates the required "n".8 Fig.5 3.4 1.0 which is positive for values in the specified range. For Xi Gaussian with mean µ. 3: Asymmetries and Mean Deviation. The situation is the result of multimodality.5 = 7/4 FT -1. Symmetric Mixed Gaussians. Stochastic Mean While mixing Gaussians the kurtosis rises. E = µ erf pµ2 + ⇡2 e 2 2 . (Commentary). as if it were more "stabilizing". noting that stable distributions q unimodal ⇣ ⌘ are µ2 (Ibragimov and Chernin) [7] and infinitely divisible Wolfe [8]. 2.0 0.5 2 2 300 250 200 150 100 D 50 RA 350 1. We can observe a similar effect of "thin-tailedness" as far as the n required to match the standard benchmark.0 1.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES 6 (|X|) = 5/4 3. as tail uncertainty creates more fat-tailedness.6 1. Which is also negative with respect to alpha as can be seen in Figure 4. 2.

Proof.08 3 3 2 0.04 1 0. (details). 4 . 0 E ⇣P kn E ⇣P n Xi↵ m↵ n↵ Xi↵ m↵ n↵ ⌘ 2 2 2 + erf 2 2 ⇣ p 2 ⌘⌘2 1 C A (9) ⌘ c2 1 c1 = k ↵ 1 ✓ ✓ ◆◆ 1 7/2 1/2 c2 = 2 ⇡ 4 2 D Note that because the instability of distribution outside the basin.2) . For all so and so.10 9 4 2 2 0. 5: Different Speed: the fatter tailed processes are not just more uncertain.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES 7 0. 0 ⇣q ⇣ ⌘1 2 p erf ⇡ e 2 B A erf @ e p + A p + p exp @ ⇡ 2 2⇡ 1 B.12 5 2 3 8 0. etc. equation 9 becomes an equality and k ! 1 we satisfy the equalities in ?? and 8. 3 2 7 4 4 |t| ⌘ 2x2 3 ⌘ . 2. n = 1. so at k = 2.06 2 0.02 20 000 40 000 60 000 80 000 100 000 20 000 40 000 60 000 80 000 100 000 probability: 1 (E+ + E 2 0 e )=@ p 2 2 2 2⇡ + 1 erf 2 ✓ p 2 ◆ FT Fig. (Sketch) The characteristic function for ↵ = 32 : (t) = 33/8 |t| 3/4 p 8 Leading to convoluted density p2 for a sum n = 2: p2 (x) = 5 4 2 F1 p 3 K 34 ⇣q 3 2 3 4 2 ⇣ 5 7 4 . they end up converging to SM in(↵. they also converge more slowly. Half cubic Student T (Lévy Stable Basin) Relative convergence: c1 where: 2 2 2 RA Theorem 1.

1 10 20 30 C. .4 0. not not for the general form: ( p ) p p p p p p 2 3 3 3 34 71 3 3138 3 899 710162 3 425331 3 33082034 5719087 3 p .EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES | 1 8 n n xi | 0. 6: Student T with exponent =3.6 0. p . . .3 FT 0. p .2 the pdf p(x) can be writen: ⇣ ⌘ n 1 2 n2 + x3 p p(x) = 3⇡ ✓ ⇣ ⌘ 1 2 k +n ◆ x2 2 n! n + Tk+1 3 n X k=0 (n k)! q 1 x2 3n2 +1 ! which allows explicit solutions for specific values of n. .. ⇡ 2⇡ 9 3⇡ 64⇡ 3125⇡ 324 3⇡ 823543⇡ 524288⇡ 14348907 3⇡ 7812500⇡ 2 With thanks to Abe Nassen and Jack D’Aurizio on Math Stack Exchange. {En }1 n<1 = .7 0.2 0. We can get a semi-explicit density for the Cubic Student T. p 6 3 p(x) = 2 ⇡ (x2 + 3) '(t) = E[eitX ] = (1 + hence the n-summed characteristic function is: '(t) = (1 + and the pdf of Y is given by: 1 ⇡ Z t e t p(x) = Z D using 1 0 k +1 (1 + p p 3 |t|) e 3|t|)n e p 3 t)n e p 3 |t| p n 3 |t| p n 3t cos(tx) dt 0 p T1+k (1/ 1 + s2 )k! cos(st) dt = (1 + s2 )(k+1)/2 where Ta (x) is the T-Chebyshev polynomial. .5 Fig. 0. . This applies to the general class of symmetric power law distributions... Cubic Student T (Gaussian Basin) 40 50 n we have: RA Student T with 3 degrees of freedom (higher exponent resembles Gaussian). .

V. Super-thin tailed distributions: Subgaussians P P Consider a sum of Bernoulli variables X.4 FT 0. 7: Sum of bets converge rapidly to Gaussian bassin but remain clearly subgaussian for small samples.2 long shot (1/100) 1 2 4 7 p= 1 100 0.3 0. Skew Normal Distribution This is the most untractable case mathematically.5 Betting against the 0.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES 0.2 0. 8: For asymmetric binary bets. Mixing Gaussian Distributions and Poisson Case B. C. at small values of p. convergence is slower. apparently though the most present when we discuss fat tails [9]. The Lognormal and Borderline Subexponential Class VI.3 4 6 8 10 p= 0. A SYMMETRIC D ISTRIBUTIONS IN THE S UPEREXPONENTIAL C LASS A.4 9 p= 7 1 8 Fig. Assuming np 2 N+ to simplify: ✓ ◆ X x n E (|⌃n |) = 2 (x np) p (1 p)n x x i0np . A SYMMETRIC N ON S TABLE D ISTRIBUTIONS IN THE S UBEXPONETIAL C LASS A. The average n ⌘ in xi follows a Binomial Distribution. Gaussian 0.1 RA Gaussian 40 60 80 100 D 20 Fig. One-tailed Pareto Distributions B.5 p= 1 2 4 0.

means principal value in the Cauchy sense. np + 2.v. np + 3. dx ⇡ x 1 t Here p. and H2 = 2 F1 ⇣ p ⌘◆ 2 q 4 ⇡ 1 5 3 2. Cumulants ✓ C2g = 1 erf( p + e ⇡ which is ⇡ 1.. D C.v. Pinelis [11]): Z 1 1 f (x) H(f ) = p.v. 4. [10]. and 10 p p 1 p p 1 ◆ ◆ VII. A. ⇡ Z 1 1 e |t |↵ t2 dt = 2 ⇡ ✓ ↵ 1 ↵ ◆ . P ROOFS . dFz (z) = dFX (x)(1 sgn(x)). ACKNOWLEDGEMENT Colman Humphrey. 4. 2. 2 = 5 4 . = lim lim a!1 b!0 1 E(|X|) = In our case: E(|X|) = b + a Z Z a b 1 @ 1 @ (z) H( (0)) = p..v. H 1 = 2 F1 ⇣ 3 1 +3 + 24⇡ 9/4 9/2 2 1 (H2 + 2) p 4 3 3 5 7 4. For a powerlaw distribution. cumulants are more unwieldy: ↵=3/2 C1 1 3 (⇡ 1 = p 2 + 2 ) 5/4 2 6⇡ 3/2 1 3 q q 5 4 3 4 2 2 + 1536 2 ⇡ 1 + 3 H2 + ⇡ ⇡ 21 + where 1 = ✓ = q 2 ⇡6 1/⇡ C1g 5 4 3 4 RA Move to appendix ↵ =3/2 C2 ◆ FT we have in the Gaussian case indexed by g: 3 4 . Derivations using explicit E(|X|) See Wolfe [5] from which Hardin got the explicit form[6]. dz|t=0 @t ⇡ @t t z 1 Z 1 1 (z) E(|X|) = p. given that for Z = |X|. that type of thing. B. 4. in other words Z 1 Z p.30 C1g . dz 2 ⇡ z 1 1 p.. 2 1 ⇡ 2 9/2 H 1 3 1 + 2 3 2 1 ⌘ .EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES E (|⌃n |) = 2(1 p)n( ✓ ✓ ◆ n (p 1) 1 np + 1 where: p)+n 2 np+1 p (np + 2) ◆ ◆ n p(np + 2) 2 np + 2 ✓ 1 ✓ =2 F˜1 1. Hilbert Transform for a function f (see Hlusel. n(p 1) + 2. Derivations using the Hilbert Transform and =0 Section obsolete since I found forms for asymmetric stable distributions.v. APPENDIX : M ETHODOLOGY. n(p 1) + 1. Some commentary on Hilbert transforms for symmetric stable distributions. ⇡ 2 1 2 3 ⌘ 2⇡ 9/4 2 2⇡ 3/4 H1 .v. 3 = 1 4 3 5/2 2 1 384⇡ 5/4 2 3 ⇣ p 3 2 . 2 ✓ =2 F˜1 2. E TC .

” Probability Theory and Related Fields. S.” 2011. 417–419. I.EXTREME RISK INITIATIVE —NYU SCHOOL OF ENGINEERING WORKING PAPER SERIES 11 R EFERENCES S. Taqqu. 2011. J. 2013. M. S. 4. I. Chernin.” Pure and Applied geophysics. 2005. Chance and stability: stable distributions and their applications. 4. Y. 329–335. D RA FT [1] [2] [3] [4] [5] [6] [7] [8] [9] . “Skewed stable variables and processes. vol. CRC Press. vol. 1978. G. vol. Wolfe.” Theory of Probability & Its Applications. Hlusek. pp. no. “On the characteristic function of the positive part of a random variable. “On distribution of absolute values. Uchaikin and V. pp. and F. 4. Ibragimov and K. “On the unimodality of geometric stable laws. Walter de Gruyter. 1973. 162. no.” arXiv preprint arXiv:1309. Penguin. Zolotarev.. 1.” The Annals of Probability. Zolotarev. American Mathematical Soc. Hardin Jr. Stable non-Gaussian random processes: stochastic models with infinite variance.5928. “On the local behavior of characteristic functions. Kagan. Pinker. D. 1187–1228. no. “Approximating the distribution of pareto sums. Schoenberg. 1959. 1984. 1986. M. Samorodnitsky and M. J. 1994.. The better angels of our nature: Why violence has declined. [11] I. Wolfe. S. 65. pp. Zaliapin. 6-7. pp. vol. vol. C. V. 45. V. One-dimensional stable distributions. 862–866. Tech. Pinelis. [10] M.” DTIC Document. “On the unimodality of infinitely divisible distribution functions. Rep. V. P. Y. 1999.

- 3419.pdfUploaded bypkj009
- 1602.07300v2Uploaded bysobaseki1
- n13.pdfUploaded byChristine Straub
- Ch. 08Uploaded byfour threepio
- Mendes 1994Uploaded byTainã Melo
- System ProgrammingUploaded byMuzaFar
- Patriarche Et Al. - 2007 - Stereological Analysis of Fractures in the Roselend Tunnel and Permeability DeterminationUploaded byAlex Tatomir
- EstimationUploaded byyashbhardwaj
- Standard DeviationUploaded byconglucy
- Case Analysis 2Uploaded byJoseph Nikolai Chioco
- Writing up research - a statistical perspectiveUploaded byJack Chen
- The Use of Geographically Weighted RegressionUploaded byLeonel Lara Estrada
- Introduction Defining the Role of Statistics in BusinessUploaded byDr Singh
- Only Be Live MisaUploaded bysfofoby
- 054.pdfUploaded bySynths
- Adaptive Parametric Algorithms for ProcessingUploaded bySampad Pradhan
- birchall1-article critique-april 14-2015-final2Uploaded byapi-289842236
- Bed-Thickness Structure of Multi-Sourced Ramp Turbidites; Devonian Brallier Formation, Central Appalachian Basin - JSR, Drummond, 1999.pdfUploaded byAlex

- Effect of Protests on VotingUploaded byMichael Kaplan
- Brassier Genre is ObsoleteUploaded byMichael Kaplan
- Tausk The Influencinh Machine.pdfUploaded byMichael Kaplan
- Estlund Beyond Fairness & DeliberationUploaded byMichael Kaplan
- A Culture of Candor - HBRUploaded byMichael Kaplan
- SHEG Evaluating Information OnlineUploaded byMichael Kaplan
- Bringhurst & Chappell a Short History of the Printed WordUploaded byMichael Kaplan
- Legitimating the BailoutUploaded byMichael Kaplan
- Mooney Citizen Juries HandbookUploaded byMichael Kaplan
- Scheuerman Civil Disobedience & Anti-Legal TurnUploaded byMichael Kaplan
- Žižek Snowden Assange and Political EventUploaded byMichael Kaplan
- Is Religion Above the LawUploaded byMichael Kaplan
- How Christian Were the FoundersUploaded byMichael Kaplan
- Aronczyk Marketing ActivismUploaded byMichael Kaplan
- Critchley the Rigor of LoveUploaded byMichael Kaplan
- Five Minute Refresher EzineUploaded byMichael Kaplan
- Bellah American Civil ReligionUploaded byMichael Kaplan
- Ford & NazisUploaded byMichael Kaplan
- Clarke Ordinary PeopleUploaded byMichael Kaplan
- Gilgen Deconversion of de VriesUploaded byMichael Kaplan

- ESTADÍSTICA BÁSICAUploaded byAlejandro Duvan Lopez Rojas
- Anteproyecto ResidenciaUploaded byJorge Sanchez Zaragoza
- thesis by gasparUploaded bykris_88
- Final ExamUploaded byBonjovi Hajan
- Basic Numeracy StatisticsUploaded byupscportal.com
- Advance Probability and Statistics - 2014 - 2da edición - Ottman.pdfUploaded byAdan Graus Rios
- Semibatch Reaction Crystallization of Benzoic AcidUploaded byFita Desti Senja
- Statistical Methods for Research Workers. Fisher, Ronald a. 2000Uploaded byJosé Daniel Rivera Medina
- Evaluación de la bondad de un estimador puntualUploaded byPedro' Cordero'
- Scenario AnalysisUploaded byajinkyan2000
- Verma&Quiroz_2006_Tablas_Criticas.pdfUploaded byItzel Olazaran
- GJBSS Paper Vol4 Issue 3 4 Paper3Uploaded byNanieta Nacar
- Elastic Stability of Thin- Walled Cylindrical and Conical Shells under Axial CompressionUploaded byChandra Prakash
- Statistics 01Uploaded byAmol
- (www.entrance-exam.net)-RPF Sub Inspectors Constables Exam Sample Paper 5Uploaded byjaya30061988
- Unidad_09Uploaded byyrisnoemi
- 3360hypUploaded bynorvin_tolinero
- How to Choose a Sample SizeUploaded bypoonam_gangwal
- ImfUploaded byAndreea Mierliţă
- UncertaintiesUploaded bydarklordsauron
- Andrade Et Al. - 2014 - Maintaining High Activity Levels in Sedentary Adults With a Reinforcement-thinning ScheduleUploaded byJohn
- 7.Hum-professional Commitment of Teacher Educators-dr. Nimisha BeriUploaded byImpact Journals
- Trabajo final de est. 1. 2018-2 Semi presencial.docxUploaded byMíGuell ÀnGél Jr.
- Muest ReoUploaded byTomasHam
- Practice Test in StatisticsUploaded byJo Joy
- Bringing Accelerated Math and English to your CampusUploaded by3CSN
- MHPT IPM 4Uploaded byYuda Pangestu
- SDA 3E Chapter 4Uploaded byxinearpinger
- exam1Uploaded byJordan JabbyJoe Koffy
- Straightening Standard DeviationUploaded bycasmusings