You are on page 1of 6

# Exam S

PROBABILITY REVIEW
PROBABILITY REVIEW
PROBABILITY REVIEW
PROBABILITY REVIEW PARAMETER ESTIMATION
PARAMETER ESTIMATION
PARAMETER ESTIMATION PARAMETER ESTIMATION HYPOTHESIS TESTING
HYPOTHESIS TESTING
HYPOTHESIS TESTING HYPOTHESIS TESTING
Probability Functions
Probability Functions
Probability Functions Estimator Quality
Estimator Quality
Estimator Quality Hypothesis Testing
Hypothesis Testing
Hypothesis Testing
&& &
Bias Bias
Bias Type I error: Reject ??
Type I error: Reject ??
Type I error: Reject ?? ää even though ??
even though ??
ä even though ?? ää is true
is true
ä is true
???? ?????? ??=== ???? ?????? d?? ??d??d??
==Pr =PrPr ????≤ ??≤??≤?? ??
'('( '(
bias
biasbias
vv ?? v?? ??==E=E??E?? ?? −−?? −?? ?? Type II error: Accept ??
Type II error: Accept ??
Type II error: Accept ?? ää even though ??
even though ??
ä even though ?? RR is true
is true
R is true
???? ?????? ??==1=1−1 −??− ?? ?????? ??==Pr =PrPr ????> ??>??>?? ?? An estimator is asymptotically unbiased if the bias ????=
An estimator is asymptotically unbiased if the bias
An estimator is asymptotically unbiased if the bias ??=Pr
=PrPr Type I error
Type I error
Type I error==significance level =significance level
significance level
???? ?????? ?? of an estimator goes to 0 as the sample size goes to ????=
of an estimator goes to 0 as the sample size goes to
of an estimator goes to 0 as the sample size goes to ??=Pr
=PrPr Type II error
Type II error
Type II error
ℎℎ ??ℎ?? ??=== infinity.
infinity.
infinity. 11−1 −??− ??= ??=Pr(=Pr( Pr(Reject ??
Reject ??
Reject ?? ää?? ??RR is true
?? is true
R is true ==power =power
power
???? ?????? ?? ä
&& & Consistency
Consistency
Consistency ??-value: Probability of the observations if ??
??-value: Probability of the observations if ??
??-value: Probability of the observations if ?? ää is true
is true
ä is true
?????? ???? ??=== ℎℎ ??ℎ?? d?? ??d??d??
==− =−ln −ln??ln ; ;????;?????? ??==??=??'8
?? ?????? ?? '8'8
?? && & ?? is consistent if:
?? is consistent if:
?? is consistent if: Confidence Intervals and Sample Size
Confidence Intervals and Sample Size
Confidence Intervals and Sample Size
'( '( '( ?? ? ?? ?
lim
lim lim
PrPrPr????bb??−− If ??~??(??,
If ??~??(??,
If ??~??(??, ???? ) and ??
??) and ??
) and ?? is known, then the
is known, then the
is known, then the 11−1 −??−
?? %
??% %
Moments
Moments
Moments b ?? − ?? ??
<<??<?? ??==1 for all ??
=1 for all ??
1 for all ?? >>0 >0 0 ôô ô
b→(
b→( b→(
(( ( confidence interval for ?? is ??
confidence interval for ?? is ??
confidence interval for ?? is ?? ±±??± ??ó/? ??ó/? where
ó/? where
where
EE??E?? ?????? ??=== ???? ?????? ?????? ?????? ?? d??
d?? d?? •• Sufficient but not necessary for consistency:
•Sufficient but not necessary for consistency:
Sufficient but not necessary for consistency: bb b

bias
bias bias →→0 and Var
→0 and Var
0 and Var ???? ??→→0 as ??
→0 as ??
0 as ?? →→∞ →∞ ∞ Pr (??
Pr (??
Pr (?? <<??<??ó/? ??ó/?
ó/? ))== ) ??/2.
=??/2.
??/2.
'('( '(
kk th thk raw moment: ??
raw moment: ??
th raw moment: ?? == =
<<= =E=E??E??<<?? <
< Relative Efficiency
Relative Efficiency
Relative Efficiency Confidence Intervals for Means
Confidence Intervals for Means
Confidence Intervals for Means
?? ?
kk th thk central moment: ??
central moment: ??
th central moment: ?? = = E=E E?? ??− ??
− ??−?? << <
?? The relative efficiency of ??
The relative efficiency of ??
The relative efficiency of ?? χ χ χ distribution
distribution
distribution
<< < RR with respect to ??
with respect to ??
R with respect to ?? ?? ?
Variance: ??
Variance: ??
Variance: ?? ?? ?
==??=??????== ?E =E E????− ??
−??− ?? ???? = ?
=E=E??E??????− ?
−??− ???? ?? ? If ??
If ??
If ??===ûiüR û û ?? ?
????i i, ?? has a ??
??, ?? has a ?? ?? ?
i , ?? has a ?????? distribution. ?? distribution.
distribution.
Var
Var Var ????????? iüR iüR
???? ?? === For ??
For ??
For ??i ~??
i ~??i ~??
?? ?
??,??,????,
?? ??: : :
Coefficient of variation: ????
Coefficient of variation: ????
Coefficient of variation: ???? === VarVarVar ????RR??R
???? ?? •• • biüR b b
iüR iüR????i i??
−− i ??
− ?? ???? ?????????~?? ? ?? ?
~?? ~?????? ??
????SS??S Mean Square Error
Mean Square Error
Mean Square Error • • • bb b
???? ??
− − ??
− ?? ?? ? ?? ? ?? ?
?? ???? ~??
?? ~?? ~??????−?? −1− 1 1
Skewness: ??
Skewness: ??
Skewness: ?? = =
RR R SS S = ? ? ? ?? ? iüR
iüR iüRi i i
???? ?? MSE
MSE MSEvv ?? v?? ??==E=E E????−??−??− ?? ?? ==Var =Var Var ???? ??
+++bias bias biasvv ??v?? ?? Student’s t distribution
Student’s t distribution
Student’s t distribution
????VV??V
Kurtosis: ??
Kurtosis: ??
Kurtosis: ?? ??= =
? =VV V Uniformly Minimum Variance Unbiased Estimator ?? has a ?? distribution if ??
Uniformly Minimum Variance Unbiased Estimator
Uniformly Minimum Variance Unbiased Estimator ?? has a ?? distribution if ??
?? has a ?? distribution if ?? ==??=?? ?? ?? ???? ?? with
?? with
?? with
???? ?? ?? ?
Covariance: Cov
Covariance: Cov
Covariance: Cov ??,
??,????,?? ??==E=E E????− ??
−??− ??YY??Y????−?? −??−??ZZ??Z (UMVUE)
(UMVUE)
(UMVUE) ??~??(0,1) and ??~??
??~??(0,1) and ??~??
??~??(0,1) and ??~?? (??).
(??).
(??).
==E=E???? E???????? −−E− E??E??E??E??E?? ?? A UMVUE is an unbiased estimator that has the
A UMVUE is an unbiased estimator that has the
A UMVUE is an unbiased estimator that has the Testing the mean of a normal population
Testing the mean of a normal population
Testing the mean of a normal population
Cov
Cov Cov ??,??,??
??,
?? ?? lowest variance of any unbiased estimator
lowest variance of any unbiased estimator
lowest variance of any unbiased estimator For a sample of size ?? from a normal population,
For a sample of size ?? from a normal population,
For a sample of size ?? from a normal population,
Correlation coefficient: ??
Correlation coefficient: ??
Correlation coefficient: ?? YZYZYZ === regardless of the true value of the estimated
regardless of the true value of the estimated
regardless of the true value of the estimated the ?? statistic to test if the mean is ?? is:
the ?? statistic to test if the mean is ?? is:
the ?? statistic to test if the mean is ?? is:
????YY????ZYZ??Z
parameter.
parameter.
parameter. ????− −??−
?? ?? ??
Percentiles
Percentiles
Percentiles ????=??==
The 100p
The 100p
The 100p percentile, ??
percentile, ??
percentile, ?? Maximum Likelihood
Maximum Likelihood
Maximum Likelihood ???? ?? ???? ??
]], is the lowest point where
, is the lowest point where
] , is the lowest point where
thth th
Steps to Calculating MLE:
Steps to Calculating MLE:
Steps to Calculating MLE: ????
????????
==??=??−?? −1 −1 1
???? ??????]]??]≥≥??. If ?? is continuous, ??
≥??. If ?? is continuous, ??
??. If ?? is continuous, ?? ]] is a unique point
is a unique point
] is a unique point 1.1.??1.?? ?????? ??=== ???? ?????? ?? 3.3.Set ??
3.Set ?? == =
Set ?? ???? ??==0 =0 0
satisfying ??
satisfying ??
satisfying ?? ????]]??]==??. =??. ??. Testing the mean of a Bernoulli population
Testing the mean of a Bernoulli population
Testing the mean of a Bernoulli population
2.2.??2.?? ?????? ??==ln =ln??ln ?? ?????? ?? 4.4.Solve for ??
4.Solve for ??
Solve for ?? For a sample of size ?? from a Bernoulli population
For a sample of size ?? from a Bernoulli population
For a sample of size ?? from a Bernoulli population
Mode
Mode Mode The maximum likelihood estimate equals the
The maximum likelihood estimate equals the
The maximum likelihood estimate equals the with mean ??, the ?? statistic to test if ??
with mean ??, the ?? statistic to test if ??
with mean ??, the ?? statistic to test if ?? ==??=??ää is:
?? is:
ä is:
Mode is x that maximizes ??
Mode is x that maximizes ??
Mode is x that maximizes ?? ???? .
??. . method of moments estimate for:
method of moments estimate for:
method of moments estimate for: ???? ?? ????−??−??− ??ää??ä
Moment Generating Function
Moment Generating Function
Moment Generating Function •• •Binomial with fixed ??
Binomial with fixed ??
Binomial with fixed ?? ????=??==
????Y??Y ??Y?? = ??=E=E??E??aYaY ?? aY
•• •Poisson
Poisson
Poisson ????ää??1ä1−1 −??− ??ää??ä???? ??
????Y??
bb b
==E=E??E??bb??, where ??
b bb b ??: number of successes
??: number of successes
??: number of successes
Y Y00 0 , where ??
, where ?? is the n
is the n
is the n derivative
derivative
th derivative
thth
•• •Negative binomial with fixed ??
Negative binomial with fixed ??
Negative binomial with fixed ??
Conditional Probability
Conditional Probability
Conditional Probability Testing the difference of means from two
Testing the difference of means from two
Testing the difference of means from two
•• •Exponential
Exponential
Exponential
PrPr
Pr ????∩??∩??∩ ?? ?? populations
populations
populations
•• •Gamma with fixed ??
Gamma with fixed ??
Gamma with fixed ??
PrPrPr ????∣??∣????∣ ??===

•• •If the two populations ?? and ?? are normally
If the two populations ?? and ?? are normally
If the two populations ?? and ?? are normally
Pr PrPr ???? ?? •• •Normal
Normal
Normal
???? ??,
????,????,
?? ?? distributed with unknown variances that are
distributed with unknown variances that are
distributed with unknown variances that are
Variance of Maximum Likelihood Estimator
Variance of Maximum Likelihood Estimator
Variance of Maximum Likelihood Estimator
????
YY?? ??Y??∣??∣????∣ ??=== assumed to be equal, the test statistic is:
assumed to be equal, the test statistic is:
assumed to be equal, the test statistic is:
???? ?????? ?? Fisher information matrix
Fisher information matrix
Fisher information matrix
1.1.Two unpaired normal populations
1.Two unpaired normal populations
Two unpaired normal populations
Bayes’ Theorem
Bayes’ Theorem
Bayes’ Theorem ????????lnln ?
??ln
?? x;
??x;??x;
?? ?? ?????????? ?????;????;????;
?? ??
I I ????
I ??==−??E =−??E−??E ==−E =−E −E ????− −??−
?? ?? ??
PrPr
Pr ????∣??∣????∣ ?? PrPrPr ???? ?? d??
d??d??
?? ? d?? d??d??
? ? ? ????= ??== , where
, where
, where
PrPrPr ????∣??∣????∣ ??=== 11 1 11 1
PrPrPr ???? ?? Rao-Cramè
Rao-Cramè
Rao-Cramè rr lower bound for the variance
lower bound for the variance
r lower bound for the variance ???? ?? +++
???? ????YY??Y ????ZZ??Z
ZZ?? ??Z??∣??∣????∣ ??????
YY?? ??Y?? ?? of an estimator
of an estimator
of an estimator
????
YY?? ??Y??∣??∣????∣ ??=== 'R'R'R ????YY??−Y−1− 1 ??1??Y?Y???+Y+
?
+????ZZ??−−
Z1 −1 ??1??Z?Z???Z?
????
ZZ Z??
?? ?? ?? Var
Var Var ?? ?? ??= = =I I ?? ??
I ?? ?? ?
???? ??===
Law of Total Probability
Law of Total Probability
Law of Total Probability Sufficient Statistics
Sufficient Statistics
Sufficient Statistics ????YY??+Y+??+ ??ZZ??−−
Z2 −22
•• •Discrete
Discrete
Discrete ????
???? ????
= = ??
= ?? ??
+ + ??
+?? ??
−− 2
−2 2
•• •A statistic ?? is sufficient for a parameter ?? if the
A statistic ?? is sufficient for a parameter ?? if the
A statistic ?? is sufficient for a parameter ?? if the YY Y ZZ Z
2.2.Two paired normal populations
2.Two paired normal populations
Two paired normal populations
Pr PrPr ???? ??=== Pr PrPr ????∩?? ∩??∩ ??i i??= = = PrPr Pr?? ?? ??PrPr Pr
?? ??∣??
∣ ????
∣ ??
likelihood of the sample, given ??, does not
likelihood of the sample, given ??, does not
likelihood of the sample, given ??, does not
i ii i ii i ???? ??
ii i ii i
depend on the parameter:
depend on the parameter:
depend on the parameter: ????= ??== , where
, where
, where
•• •Continuous
Continuous
Continuous ???? ??
???? ,??

,
RR R bb b…,??
…?? ;??;?? ??;??
??
?? ??= = ℎ
= ℎ ??
ℎ?? ,??

, … ,
RR R bb b??…?? ?? ?? ?? ?? ???? ??
•• •A statistic ?? is sufficient for a parameter ?? if and
A statistic ?? is sufficient for a parameter ?? if and
A statistic ?? is sufficient for a parameter ?? if and ???? ??
i i i i i i i i i
= = ??
= ?? −
??− ??
−?? ??

Pr PrPr ???? ??=== Pr PrPr ???? ?????? ?????? ?????? ?? d?? d?? d?? only if the following factorization is possible:
only if the following factorization is possible:
only if the following factorization is possible: ????= ??=sample mean of the ??
=sample mean of the ??
sample mean of the ?? i s
i s i s

Conditional Expectation Formula
Conditional Expectation Formula
Conditional Expectation Formula ???? ??????RR,??…
, R…,??…
??bb;??;??b??; ??==?? =?? ?????? ??⋅ ⋅ℎℎ⋅ ??ℎ??RR,??…
, R…,??…
??bb?? b ????=??=sample standard deviation of the ??
=sample standard deviation of the ??
sample standard deviation of the ?? i s
i s
i s

EEYYE??Y?? ??==E=EZZEEZEYYE??Y??∣??∣????∣ ?? • • Rao-Blackwell Theorem: For an unbiased
• Rao-Blackwell Theorem: For an unbiased
Rao-Blackwell Theorem: For an unbiased ????
????????
= = ??
= ??− ??
− 1
− 1 1
Variances
Variances
Variances estimator ?? and a sufficient statistic ??, ??
estimator ?? and a sufficient statistic ??, ??
estimator ?? and a sufficient statistic ??, ?? ?? is is •• •For Bernoulli data for two samples, the ??
?????????? is For Bernoulli data for two samples, the ??
For Bernoulli data for two samples, the ??
an unbiased estimator with variance that is less
an unbiased estimator with variance that is less
an unbiased estimator with variance that is less statistic to test for the difference of means of
statistic to test for the difference of means of
statistic to test for the difference of means of
•• •Variance of the Sample Mean
Variance of the Sample Mean
Variance of the Sample Mean
than or equal to ??????
than or equal to ??????
than or equal to ?????? ???? . ??. . two populations with ??
two populations with ??
two populations with ?? YY and ??
and ??
Y and ?? ZZ successes
successes
Z successes
For ?? independent and identically distributed
For ?? independent and identically distributed
For ?? independent and identically distributed
respectively is:
respectively is:
respectively is:
(i.i.d.) random variables, the variance of the
(i.i.d.) random variables, the variance of the
(i.i.d.) random variables, the variance of the The maximum likelihood estimator is a function of
The maximum likelihood estimator is a function of
The maximum likelihood estimator is a function of ????YY??Y ????ZZ??Z
lmn lmn lmn
YY Y a sufficient statistic.
a sufficient statistic.
a sufficient statistic. −−−
sample mean is Var
sample mean is Var
sample mean is Var ???? ??=== . . . ????YY??Y ????ZZ??Z
bb b
????=??
= = , where
, where
, where
•• •Bernoulli Shortcut
Bernoulli Shortcut
Bernoulli Shortcut 11 1 11 1
???? 1??1−1 −??− ?? ?? +++
?? probability
?? probability
?? probability ==??
=??
?? ????YY??Y ????ZZ??Z
????= ??== ????YY??+Y+??+ ??ZZ??Z
?? probability
?? probability
?? probability ==1=1−1 −??− ?? ?? ????=??=overall sample mean
=overall sample mean
overall sample mean ===
Var Var Var ???? ??===????−?? −??− ?? ?????????1??1−1 −??− ?? ?? ????YY??+Y+??+ ??ZZ??Z
•• •Conditional Variance Formula
Conditional Variance Formula
Conditional Variance Formula
Var VarVarYY ?? Y?? ?? ==Var =Var VarZZ E ZEYYE?? Y??∣?? ∣????∣ ??++E+ EZZEVar ZVar Var
YY ??
Y??∣?? ∣????∣ ??