Common distribution

Lesson 0: Preliminaries
Le Thi Xuan Mai
The university of natural sciences

February 24, 2013

T.X.M. Le

Bayesian statistics

Common distribution

Discrete random variables

T.X.M. Le

Bayesian statistics

Common distribution

The binomial distribution

As its name suggests, the binomial distribution refers to random variables with two outcomes. Example: smoking status: a person does or does not smoke healthy insurance coverage: a person does or does not have health insurance. We now consider the rst example to demonstrate the calculation of binomial probabilities. Suppose that four adults have been randomly selected and asked whether or not they currently smoke. The random variable of interest in this example is the number of persons who respond yes to the question about smoking (denoted by X ).

T.X.M. Le

Bayesian statistics

Common distribution

Suppose that in the population, the proportion of people who would respond yes to this question is , and the probability of a response of no is then 1 . Since each person is independent of all the other persons, the binomial probability can be calculated as follows P (X = 4) = 4 (the number of 4 responses is 4 ) P (X = 3) = 43 (1 ) (since there are four occurrences of three yes responses) P (X = 2) = 62 (1 )2 P (X = 1) = 4(1 )3 P (X = 0) = (1 )4

T.X.M. Le

Bayesian statistics

Common distribution

Suppose is 0.25. Then the probability of each outcome is as follows

0 P (X = 4) = C4 (0.25)4 (0.75)0 = 0.0039 = P{0 no responses} 1 P (X = 3) = C4 (0.25)3 (0.75)1 = 0.0469 = P{1 no response} 2 P (X = 2) = C4 (0.25)2 (0.75)2 = 0.2109 = P{2 no response} 3 P (X = 1) = C4 (0.25)1 (0.75)3 = 0.4219 = P{3 no response} 4 P (X = 0) = C4 (0.25)0 (0.75)4 = 0.3164 = P{4 no responses}

This lead us to dene a binomial random variable and calculate its probability distribution.

T.X.M. Le

Bayesian statistics

Common distribution

A random variable Y {0, 1, . . . , n} has a binomial (n, ) distribution if [0, 1] and

y y P (Y = y |, n) = Cn (1 )ny for y {0, 1, . . . , n}.

For this distribution, E [Y |] Var[Y |] mode[Y |] p (y |, n) = n ; = n(1 ); = [(n + 1)]; = dbinom(y , n, ).

If Y1 binomial(n1 , ) and Y2 binomial(n2 , ) are independent, then Y = Y1 + Y2 binomial(n1 + n2 , ). When n = 1 this distribution is called the Bernoulli distribution. The binomial(n, ) model assume that Y is a sum of independent binary() random variables.
T.X.M. Le Bayesian statistics

Common distribution

Figure : Binomial distribution with n = 10 and = 0.2.

T.X.M. Le

Bayesian statistics

Common distribution

Figure : Binomial distribution with n = 10 and = 0.8.

T.X.M. Le

Bayesian statistics

Common distribution

Figure : Binomial distribution with n = 100 and = 0.2.

T.X.M. Le Bayesian statistics

Common distribution

Figure : Binomial distribution with n = 100 and = 0.8.

T.X.M. Le

Bayesian statistics

Common distribution

The negative binomial distribution

A random variable Y {0, 1, . . .} has a negative binomial(r , ) distribution for any positive integer r if 0 1 and
y y r P ( y | r , ) = Cy +r 1 (1 ) .

For this distribution, E [Y |r , ] Var[Y |r , ] mode[Y |r , ] p (y | )

= r /(1 ); = r /(1 )2 ; = [(1 )(r 1)/] if r > 1, mode[Y |r , ] = 0 if r 1; = dnbinom(y , ).

T.X.M. Le

Bayesian statistics

Common distribution

Figure : Negative binomial distribution.

T.X.M. Le

Bayesian statistics

Common distribution

The name negative binomial can be replaced by inverse binomial because it arises from a series of Bernoulli trials where the number of successes is xed at the outset. In this case, Y is the number of failure preceding the r th success. This distribution is sometimes used to model countably innite random variables having variance substantially larger than the mean (so that the Poisson model would be inappropriate). The geometric is the special case of the negative binomial having r = 1. It is the nuber of failures preceding the rst success in a series of Bernoulli trials.

T.X.M. Le

Bayesian statistics

Common distribution

The Poisson distribution

A random variable Y {0, 1, . . .} has a Poisson() distribution if > 0 and P (Y = y |) = For this distribution, E [Y | ] Var[Y |] mode[Y |] p (y |) = ; = ; = []; = dpois(y , ). y e for y {0, 1, . . .}. y!

If Y1 Poisson(1 ) and Y2 Poisson(2 ) are independent, then Y = Y1 + Y2 Poisson(1 + 2 ). If it is observed that a sample mean is very dierent than the sample variance, then the Poisson model may not be appropriate.
T.X.M. Le Bayesian statistics

Common distribution

Figure : Poisson distribution with = 1, 4, 10.

T.X.M. Le

Bayesian statistics

Common distribution

Continuous random variables

T.X.M. Le

Bayesian statistics

Common distribution

The univariate normal distribution

A random variable Y R has a normal(, 2 ) distribution if 2 > 0 and p (y |, 2 ) = For this distribution, E [Y |, 2 ] Var[Y |, 2 ] = ; = 2; 1 2 2 e 2 ( y )
1 2 / 2

for < y < .

mode[Y |, 2 ] = ; p (y |, 2 ) = dnorm(y , theta, sigma).

2 ) and X normal( , 2 ) are If X1 normal(1 , 1 2 2 2 independent, then 2 2 aX1 + bX2 + c normal(a1 + b 2 + c , a2 1 + b 2 2 ).

T.X.M. Le Bayesian statistics

Common distribution

Figure : Some univariate normal distributions.

T.X.M. Le

Bayesian statistics

Common distribution

The t distribution
A random variable Y R has a t distribution( ) if > 0, and y2 [( + 1)/2] 1 + ]( +1)/2 . p (y | ) = (/2) For this distribution, E [Y | ] Var[Y | ] mode median p (y | ) = = = = = 0 if > 1; /( 2) if > 2, Var[Y | ] = if 1 < 2; 0 0 dt(y , nu).

The parameter is referred to as the degrees of freedom and is usually taken to be a positive integer, though the distribution is proper for any positive real number . The t is a common heavy-tailed (but still symmetric and unimodal) alternative to the normal distribution.
T.X.M. Le Bayesian statistics

Common distribution

Figure : Some t distributions.

T.X.M. Le

Bayesian statistics

Common distribution

The continuous uniform probability distribution

If a continuous r.v Y can assume any value in the interval a y b and only these values, and if its probability density function p (y |a, b ) is constant over that interval and equal to zero elsewhere, then Y is said to be uniformly distributed and its distribution is called a continuous uniform probability distribution or continuous rectangular distribution. Its density distribution is of the form 1 for a y b b a p (y |a, b ) = (1) 0 else where For this distribution, E [Y ] =
a+b 2 1 12 (b a+b 2

Var [Y ] = median = mode

T.X.M. Le

a)2

= any value in[a, b ].

Bayesian statistics

Common distribution

Figure : Uniform distribution.

T.X.M. Le

Bayesian statistics

Common distribution

The beta distribution

A random variable Y [0, 1] has a beta(a, b ) distribution if a > 0, b > 0 and p (y |a, b ) = E [Y |a , b ] Var[Y |a, b ] = = (a + b ) a1 y (1 y )b1 for 0 y 1. (a)(b )
a a+b ; ab = E [Y ] E [1 Y ] (a+b +1)(a+b )2 a1 (a1)+(b 1) if a > 1 and b > 1;

1 a+b +1 ;

mode[Y |a, b ] = p (y |a, b ) =

dbeta(y , a, b ).

The beta distribution is closely related to the gamma distribution. A multivariate version of the beta distribution is the Dirichlet distribution.
T.X.M. Le Bayesian statistics

Common distribution

Figure : Beta distribution.

T.X.M. Le

Bayesian statistics

Common distribution

The gamma and inverse-gamma distributions

A random variable Y (0, ) has a gamma(a, b ) distribution if p (y |a, b ) = E [Y |a, b ] = b a a1 by y e for y > 0, a > 0, b > 0. (a)
a b; a 1 b

Var[Y |a, b ] =

a ; b2

mode[Y |a, b ] = p (y |a, b ) =

if a 1, mode[Y |a, b ] = 0 if 0 < a < 1;

dgamma(y , a, b ).

If Y1 gamma(a1 , b ) and Y2 gamma(a2 , b ) are independent, then Y1 + Y2 gamma(a1 + a2 , b ) and Y1 /(Y1 + Y2 ) beta(a1 , a2 ). If Y normal(0, 2 ) then Y 2 gamma(1/2, 1/2 2 ). The chi-square distribution with degrees of freedom is the same as a gamma(/2, 1/2) distribution. If Y normal(0, 1) then Y 2 chi-square(1).
T.X.M. Le Bayesian statistics

Common distribution

Figure : Gamma distributions.

T.X.M. Le

Bayesian statistics

Common distribution

A random variable Y (0, ) has an inverse-gamma(a, b ) distribution if 1/Y has a gamma(a, b ) distribution. In other words, if X gamma(a, b ) and Y = 1/X , then Y inverse-gamma(a, b ). The density of Y is p (y |a, b ) = For this distribution, E [Y |a , b ] Var[Y |a, b ] = =
b a 1

b a a1 b/y y e for y > 0. (a)

if a 1, E [Y |a, b ] = if 0 < a < 1; if a 2, Var[Y |a, b ] = if 0 < a < 2;

mode[X |a, b ] =

b2 (a1)2 (a2) b a+1 .

Note that the inverse-gamma density is not simply the gamma density with y replaced by 1/y . The inverse-2 ( ) is a special case of the inverse-gamma distribution, with a = /2 and b = 1/2.
T.X.M. Le Bayesian statistics

Common distribution

The scaled inverse chisquare distribution

The family of scaled inverse chi-squared distributions is closely related to two other distribution families: the inverse-chi-squared distribution and the inverse gamma distribution. Compared to the inverse-chi-squared distribution, the scaled distribution has an extra parameter 2 , which scales the distribution horizontally and vertically. The relationship between the inverse-chi-squared distribution the scaled distribution is that if X inv-2 ( ) 2 The probability density function of the scaled inverse chi-squared distribution extends over the domain x > 0 and is X Scale-inv-2 (, 2 ) then f (x , , 2 ) = 2 ( 2 /2)/2 /21 x exp( ) (/2) 2x

where is the degrees of freedom parameter and 2 is the scale parameter.

T.X.M. Le Bayesian statistics

Common distribution

EX Var (X )

= =

mode(X ) =

2 for > 2 2 2 2 4 for > 4 ( 2)2 ( 4) 2 +2 X inv-2 ( ) 2 2 , 2 2

If X Scale-inv-2 (, 2 ) then kX Scale-inv-2 (, k 2 ) If X Scale-inv-2 (, 2 ) then

If X Scale-inv-2 (, 2 ) then X Inv-Gamma

T.X.M. Le

Bayesian statistics

Common distribution

Exponential distributions
A random variable Y > 0 has an exponential distribution() if > 0 and p (y |) = exp(y ). For this distribution, E [Y |] Var[Y |] = 1/; = 1/2 ;

mode[Y |] = 0; p (y |) = dexp(y , lambda).

The exponential distribution is a Gamma(1, ) distribution.

T.X.M. Le

Bayesian statistics

Common distribution

Figure : Exponential distributions.

T.X.M. Le

Bayesian statistics

Common distribution

The double exponential distribution (Laplace distribution)

A random variable Y R has a Laplace distribution(, 2 ) if < < , 2 > 0 and p (y |, 2 ) = For this distribution, E [Y |, 2 ] Var[Y |, 2 ] = ; = 2 2 ; |y | 1 exp( ). 2

mode[Y |, 2 ] = ; p (y |, 2 ) = dlaplace(y , mu, sigma).

This distribution is symmetric and unimodal, but has heavier tails and a somewhat dierent shape, being strictly concave up on both sides of .
T.X.M. Le Bayesian statistics

Common distribution

Figure : Laplace distributions.

T.X.M. Le

Bayesian statistics

Common distribution

The logistic distribution

A random variable Y R has a Logistic distribution(, 2 ) if < < , 2 > 0 and p (y |, 2 ) = For this distribution, E [Y |, 2 ] Var[Y |, 2 ] = ; = ( 2 /3) 2 ;
exp( y ) 2 [1 + exp( y )]

mode[Y |, 2 ] = ; p (y |, 2 ) = dlogis(y , mu, sigma).

The logistic is another symmetric unimodal distribution, more similar to the normal in appearance than the double exponential, but with even heavier tails.
T.X.M. Le Bayesian statistics

Common distribution

Figure : Some logistic distributions.

T.X.M. Le

Bayesian statistics

Common distribution

Cauchy distribution

A random variable Y R has a Cauchy distribution (, 2 ) if > 0, < < , 2 > 0 and p (y |, 2 ) = 1 2 . [1 + ( x ) ]

For this distribution, E [Y ] and Var[Y ] do not exist, though is the median of this distribution. This is a special case of the t distribution t (1, , 2 ) which has the heaviest possible tails.

T.X.M. Le

Bayesian statistics

Common distribution

Figure : Some Cauchy distributions.

T.X.M. Le

Bayesian statistics

Common distribution

The multivariate normal distribution

A random vector Y R p has a multivariate normal distribution normal(, ) distribution if is a positive denite p p matrix and 1 p (y |, ) = (2 )p/2 ||1/2 exp{ (y )T (y )} for y R p . 2 For this distribution, E [Y |, ] Var[Y |, ] = ; = ;

mode[Y |, ] = .

T.X.M. Le

Bayesian statistics

Common distribution

If Y1 normal(1 , 1 ) and Y2 normal(2 , 2 ) are independent, then aY1 + bY2 + c normal(a1 + b 2 + c , a2 1 + b 2 2 ). If Z is the vector with elements Z1 , . . . , Zp i.i.d. normal(0, 1) then Y = + AZ multivariate normal(, ), where and AAT = (A is the Choleski factorization of ). The following R-code will generate an n p matrix such that the rows are i.i.d samples from a multivariate normal distribution: Z Y = matrix(rnorm(n p ), nrow = n, ncol = p ) = t (t (Z % %chol(Sigma)) + c (theta)).

T.X.M. Le

Bayesian statistics

Common distribution

The Wishart and inverse-Wishart disttributions

A random p p symmetric positive denite matrix Y has a Wishart(, M ) distribution if the integer p , M is a p p symmetric positive denite matrix and

p (Y |, M ) = [2 p/2 p (/2)|M |/2 ]1 |Y |( p1)/2 exp{tr (M 1 Y /2)}, where

p

p (/2) =

p (p 1)/4 j =1

[( + 1 j )/2], and aj ,j ).

exp{tr (A)} = exp( For this distribution, E [Y |, M ] Var[Yi ,j |, M ] = M;

= (mi2,j + mi ,i mj ,j );

mode[Y |, M ] = ( p 1)M .
T.X.M. Le Bayesian statistics

Common distribution

The Wishart distribution is a multivariate version of the gamma distribution.

If X1 , . . . , X i.i.d. multivariate normal(0, Mpp ), then Xi XiT Wishart (, Mpp ).

i =1

The following R-code is used to generate a Wishart distributed random matrix: X Z Y

= matrix(rnorm( p ),nrow=nu,ncol=p) #standard normal matr = X % %chol(M) = t (Z )% %Z # #Wishart matrix

T.X.M. Le

Bayesian statistics

Common distribution

A random p p symmetric positive denite matrix Y has an inverse-Wishart(, M ) distribution if X 1 has a Wishart(, M ) distribution. In other words, if X Wishart(, M ) and X = Y 1 , then Y inverse-Wishart(, M ). The density of Y is p (Y |, M ) =[2 p/2 p (/2)|M |/2 ]1 |Y |( +p+1)/2 exp{tr (M 1 Y 1 /2)}. For this distribution, E [Y |, M ] = ( p 1)1 M 1 ;

mode[Y |, M ] = ( + p + 1)1 M 1 . If inverse-Wishart(, S 1 ), we have mode[Y |, S ] = ( + p + 1)1 S . If 0 were the most probable value of a priori, then we could set S = (0 + p + 1)0 , so that inverse-Wishart(, [( + p 1)0 ]1 ) and mode[|, S ] = 0 .
T.X.M. Le Bayesian statistics