Order statistic
From Wikipedia, the free encyclopedia
Jump to: navigation, search
Probability density functions of the order statistics for a sample of size n = 5 from an exponential
distribution with unit scale parameter.
In statistics, the kth order statistic of a statistical sample is equal to its kth-smallest value.[1]
Together with rank statistics, order statistics are among the most fundamental tools in non-
parametric statistics and inference.
Important special cases of the order statistics are the minimum and maximum value of a sample,
and (with some qualifications discussed below) the sample median and other sample quantiles.
When using probability theory to analyze order statistics of random samples from a continuous
distribution, the cumulative distribution function is used to reduce the analysis to the case of
order statistics of the uniform distribution.
Notation and examples[edit]
For example, suppose that four numbers are observed or recorded, resulting in a sample of size 4.
If the sample values are
6, 9, 3, 8,
they will usually be denoted
where the subscript i in indicates simply the order in which the observations were recorded
and is usually assumed not to be significant. A case when the order is significant is when the
observations are part of a time series.
The order statistics would be denoted
where the subscript (i) enclosed in parentheses indicates the ith order statistic of the sample.
The first order statistic (or smallest order statistic) is always the minimum of the sample, that
is,
where, following a common convention, we use upper-case letters to refer to random variables,
and lower-case letters (as above) to refer to their actual observed values.
Similarly, for a sample of size n, the nth order statistic (or largest order statistic) is the
maximum, that is,
The sample range is the difference between the maximum and minimum. It is clearly a function
of the order statistics:
A similar important statistic in exploratory data analysis that is simply related to the order
statistics is the sample interquartile range.
The sample median may or may not be an order statistic, since there is a single middle value only
when the number n of observations is odd. More precisely, if n = 2m+1 for some integer m,
then the sample median is and so is an order statistic. On the other hand, when n is even, n
= 2m and there are two middle values, and , and the sample median is some function
of the two (usually the average) and hence not an order statistic. Similar remarks apply to all
sample quantiles.
Probabilistic analysis[edit]
Given any random variables X1, X2..., Xn, the order statistics X(1), X(2), ..., X(n) are also random
variables, defined by sorting the values (realizations) of X1, ..., Xn in increasing order.
When the random variables X1, X2..., Xn form a sample they are independent and identically
distributed. This is the case treated below. In general, the random variables X1, ..., Xn can arise by
sampling from more than one population. Then they are independent, but not necessarily
identically distributed, and their joint probability distribution is given by the Bapat–Beg theorem.
From now on, we will assume that the random variables under consideration are continuous and,
where convenient, we will also assume that they have a probability density function (that is, they
are absolutely continuous). The peculiarities of the analysis of distributions assigning mass to
points (in particular, discrete distributions) are discussed at the end.
Probability distributions of order statistics[edit]
Order statistics sampled from a uniform distribution[edit]
In this section we show that the order statistics of the uniform distribution on the unit interval
have marginal distributions belonging to the Beta distribution family. We also give a simple
method to derive the joint distribution of any number of order statistics, and finally translate
these results to arbitrary continuous distributions using the cdf.
We assume throughout this section that is a random sample drawn from a continuous
distribution with cdf . Denoting we obtain the corresponding random sample from
the standard uniform distribution. Note that the order statistics also satisfy .
The probability of the order statistic falling in the interval is equal to[2]
that is, the kth order statistic of the uniform distribution is a Beta random variable.[2][3]
The proof of these statements is as follows. For to be between u and u + du, it is necessary
that exactly k − 1 elements of the sample are smaller than u, and that at least one is between u
and u + du. The probability that more than one is in this latter interval is already , so we
have to calculate the probability that exactly k − 1, 1 and n − k observations fall in the intervals
, and respectively. This equals (refer to multinomial distribution for details)
and the result follows.
The mean of this distribution is k / (n + 1).
The joint distribution of the order statistics of the uniform distribution[edit]
Similarly, for i < j, the joint probability density function of the two order statistics U(i) < U(j) can
be shown to be
which is (up to terms of higher order than ) the probability that i − 1, 1, j − 1 − i, 1 and n − j
sample elements fall in the intervals , , , , respectively.
One reasons in an entirely analogous way to derive the higher-order joint distributions. Perhaps
surprisingly, the joint density of the n order statistics turns out to be constant:
One way to understand this is that the unordered sample does have constant density equal to 1,
and that there are n! different permutations of the sample corresponding to the same sequence of
order statistics. This is related to the fact that 1/n! is the volume of the region .
Order statistics sampled from an exponential distribution[edit]
For random samples from an exponential distribution with parameter λ the order statistics
X(i) for i = 1,2,3, ..., n each have distribution
where the Zj are iid standard exponential random variables (i.e. with rate parameter 1). This
result was first published by Alfréd Rényi.[4][5]
Order statistics sampled from an Erlang distribution[edit]
The Laplace transform of order statistics sampled from an Erlang distribution via a path counting
method.[6]
The joint distribution of the order statistics of an absolutely continuous distribution[edit]
If FX is absolutely continuous, it has a density such that , and we can use the substitutions
and
to derive the following probability density functions (pdfs) for the order statistics of a sample of
size n drawn from the distribution of X:
where
where
Application: confidence intervals for quantiles[edit]
An interesting question is how well the order statistics perform as estimators of the quantiles of
the underlying distribution.
A small-sample-size example[edit]
The simplest case to consider is how well the sample median estimates the population median.
As an example, consider a random sample of size 6. In that case, the sample median is usually
defined as the midpoint of the interval delimited by the 3rd and 4th order statistics. However, we
know from the preceding discussion that the probability that this interval actually contains the
population median is
Although the sample median is probably among the best distribution-independent point estimates
of the population median, what this example illustrates is that it is not a particularly good one in
absolute terms. In this particular case, a better confidence interval for the median is the one
delimited by the 2nd and 5th order statistics, which contains the population median with
probability
With such a small sample size, if one wants at least 95% confidence, one is reduced to saying
that the median is between the minimum and the maximum of the 6 observations with
probability 31/32 or approximately 97%. Size 6 is, in fact, the smallest sample size such that the
interval determined by the minimum and the maximum is at least a 95% confidence interval for
the population median.
Large sample sizes[edit]
For the uniform distribution, as n tends to infinity, the pth sample quantile is asymptotically
normally distributed, since it is approximated by
For a general distribution F with a continuous non-zero density at F −1(p), a similar asymptotic
normality applies:
where f is the density function, and F −1 is the quantile function associated with F. One of the first
people to mention and prove this result was Frederick Mosteller in his seminal paper in 1946.[7]
Further research lead in the 1960s to the Bahadur representation which provides information
about the errorbounds.
An interesting observation can be made in the case where the distribution is symmetric, and the
population median equals the population mean. In this case, the sample mean, by the central limit
theorem, is also asymptotically normally distributed, but with variance σ2/n instead. This
asymptotic analysis suggests that the mean outperforms the median in cases of low kurtosis, and
vice versa. For example, the median achieves better confidence intervals for the Laplace
distribution, while the mean performs better for X that are normally distributed.
Proof[edit]
It can be shown that
where
with Zi being independent identically distributed exponential random variables with rate 1. Since
X/n and Y/n are asymptotically normally distributed by the CLT, our results follow by
application of the delta method.
Dealing with discrete variables[edit]
Suppose are i.i.d. random variables from a discrete distribution with cumulative distribution
function and probability mass function . To find the probabilities of the order
statistics, three values are first needed, namely
The cumulative distribution function of the order statistic can be computed by noting that
Similarly, is given by
Note that the probability mass function of is just the difference of these values, that is to say
Computing order statistics[edit]
Main articles: Selection algorithm and Sampling in order
The problem of computing the kth smallest (or largest) element of a list is called the selection
problem and is solved by a selection algorithm. Although this problem is difficult for very large
lists, sophisticated selection algorithms have been created that can solve this problem in time
proportional to the number of elements in the list, even if the list is totally unordered. If the data
is stored in certain specialized data structures, this time can be brought down to O(log n). In
many applications all order statistics are required, in which case a sorting algorithm can be used
and the time taken is O(n log n). More sophisticated methods can reduce the time to O(n).
See also[edit]
