Permutation Inference

Kosuke Imai

Harvard University

S TAT 186/G OV 2002 C AUSAL I NFERENCE

Fall 2019

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 1 / 13

The California Alphabet Lottery (Ho and Imai, 2006, J. Am. Stat. Assoc.)

Randomization sometimes occurs in the real world

Started in 1975: “[B]oth the ‘incumbent first’ and ‘alphabetical
order’ procedures are constitutionally impermissible.” Gould v.
Grubb, 14 Cal. 3d 661, 676.
A random alphabet is drawn for every statewide election that
applies to all statewide offices

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 2 / 13

California Elections Code 13112(a)

Each letter of the alphabet shall be written on a separate slip of

paper, each of which shall be folded and inserted into a capsule.
Each capsule shall be opaque and of uniform weight, color, size,
shape, and texture. The capsules shall be placed in a container,
which shall be shaken vigorously in order to mix the capsules
thoroughly. The container then shall be opened and the capsules
removed at random one at a time. As each is removed, it shall be
opened and the letter on the slip of paper read aloud and written
down. The resulting random order of letters constitutes the
randomized alphabet, which is to be used in the same manner as
the conventional alphabet in determining the order of all
candidates in all elections. For example, if two candidates with the
surnames Campbell and Carlson are running for the same office,
their order on the ballot will depend on the order in which the
letters M and R were drawn in the randomized alphabet drawing.

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 3 / 13

Permutation Test for the Natural Experiment

Take into account the complex lottery procedure

1 Randomize alphabet
2 Sort candidates by randomized alphabet
3 Rotate the candidate order from the first district
Impossible via model-based inference

The 2003 CA Gubernatorial Recall Election

135 candidates including a billionaire and a porn star
Ballot order differs across 80 districts

Null hypothesis: no causal effect of ballot order

Permutation test for each candidate
Test statistic: difference-in-means between being on the first ballot
page and being on other ballot page
Reference distribution via Monte Carlo

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 4 / 13

 5 / 13
Fall 2019

Ranken ●
●
Distribution of Exact p-values across Candidates

Ramirez
Weber ●
Weir ●
Hoffmann ●
Hanlon ●
Henderson ●
Winters ●
Watts ●
Hamidi ●
Robinson ●
●
Stat186/Gov2002

Mortensen
Vann ●
Valdez ●
Hall ●
Wozniak ●
McMahon ●
Bock ●
Kimball ●
Richter ●
WalkerC ●
McClain ●
Walton ●
Quinn ●
Roscoe ●
Jackson ●
Hernandez ●
WalkerM ●
Vaughn ●
Mednick ●
Hickey ●
Rushford ●
Richards ●
Miller ●
Sproul ●
Mailander ●
Badiozamani ●
Mock ●
Rainforth ●
Angelyne ●
Renz ●
Green ●
Simon ●
Mobley ●
Sylvester ●
McCarthy ●
Rightmyer ●
Anderson ●
Mannheim ●
Macaluso ●
Guzzardi ●
McNeilly ●
Zellhoefer ●
Beyer ●
Cullenbine ●
●

Permutation Inference
Brown
Bhola ●
Adams ●
Margolin ●
Russell ●
AlexStJames ●
●
Candidate

Mariano
Gosse ●
Strauss ●
Adam ●
Vandeventer ●
Edwards ●
Tilley ●
Tsangares ●
Beard ●
Bajwa ●
Musilli ●
Simmons ●
Britton ●
Korevaar ●
Gorman ●
Coleman ●
Leonard ●
Arif ●
Martorana ●
Sprague ●
Schmier ●
Taylor ●
Kunzman ●
Grisham ●
Sams ●
Huffington ●
Gallagher ●
Vo ●
Bustamante ●
Gruener ●
BlyChester ●
Schwarzenegger ●
Nave ●
Price ●
Tracy ●
Scheidle ●
Burton ●
Templin ●
McClintock ●
Smith ●
Issa ●
Mehr ●
Schwartzman ●
Chambers ●
NewmanII ●
Cook ●
Safford ●
Fontanes ●
Foss ●
Peters ●
Prady ●
Farrell ●
Davis ●
Friedman ●
Palmieri ●

Kosuke Imai (Harvard)

Camejo ●
Lewis ●
Kennedy ●
Carson ●
Pawlik ●
Padilla ●
Flynt ●
Kessinger ●
Dole ●
Louie ●
Ueberroth ●
Knapp ●
Kelly ●
Clements ●
Lane ●
Pineda ●
Cheli ●
Forte ●
Feinstein ●
0.2 0.4 0.6 0.8 1.0
p−value
Placebo Tests

Placebo tests: used when effects are known to be zero

Null hypothesis is assumed to be true
Ballot order should not affect pre-election covariates

Registered Voters Proportion Male Rep. Gub. Vote (2002)

140

140

140
●
●● ●
●● ●
●
● ●● ●
●●
●● ●● ●
●● ●● ●●
●● ●● ●
●
● ●
120

120

120
●● ●
● ●●
●
● ●
●● ●
●
●● ● ●
●
●● ●● ●
●
●● ●
●● ●
●
●
●● ●● ●
●●
●
●● ●● ●●
● ●● ●
●
●
● ●
●● ●●
● ●
100

100

100
●
● ●
●● ●
●
●
● ●● ●
●
●● ● ●
●
●
● ●● ●●
●
● ●● ●●
●
● ●
● ●
●
Candidate

Candidate

Candidate
●
● ●● ●
●
●● ●● ●●
●
●● ●
●● ●
●
●● ●● ●●
● ● ●●
80

80

80
●● ●● ●
●
●
●● ●
●● ●●
● ● ● ●
●
● ●● ●●
●
● ●● ●
●
●● ●● ●
●
●
● ●● ●●
●● ●● ●●
●●● ●●● ●
●
●● ● ●●
60

60

60
●● ●● ●
●●
● ●
●● ●
●● ● ●●
●
●● ●
● ●●●
●● ●●● ●
● ● ●
●●
●● ●
●● ●
●● ● ●
●●
●● ●● ●●
●● ●
●● ●
40

40

40
●● ●● ●●
●
● ● ●●
●● ●
● ●●
●
●● ●● ● ●
● ●
● ●
●●
●● ●● ●
●● ●
●● ●
●
●●● ● ●
●●
● ●●● ●●
●
● ● ● ●
20

20

20
● ● ● ●●
●
●● ●
● ●●
●● ●● ●●
● ●
● ●●
●● ●
● ●
●●
●● ●● ●
●● ●
● ●●
●
● ●
● ●
●●
●● ●
● ●●
●● ●
● ●
●● ●● ●●
0

0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
p−value p−value p−value

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 6 / 13

Inverting Permutation Tests for Confidence Intervals

Permutation tests are useful but tell us nothing about effect size

Invert a α-level test generates (1 − α) × 100% confidence set

Idea: if we cannot reject a null hypothesis with a particular effect
size, then the confidence interval should include it.

Inverting a permutation test:

1 Consider the constant additive effect model

Yi (1) − Yi (0) = τ0 for all i

2 Collect all null values τ0 that cannot be rejected by α-level test

The resulting (1 − α) × 100% confidence set is

Aα = {τ0 : Pr(f ({Yi , Tiobs }ni=1 , Tiobs , τ0 ) ≤ f ({Yi , Tiobs }ni=1 , Ti , τ0 )) ≥ α}

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 7 / 13

Revising the California Alphabet Lottery

For each candidate, we test H0 : Yi (1) − Yi (0) = τ0 for a range of

τ0 values using the permutation test at the 0.05 level
We collect all τ0 values where we cannot reject the null

Page Effect on Major Candidates Page Effect on Minor Candidates

Republican ● Republican ●
Schwarzenegger Ranken
Democratic ● Democratic ●

Republican ● Republican ●
Bustamante Ramirez
Democratic ● Democratic ●

Republican ● Republican ●
McClintock Roscoe
Democratic ● Democratic ●

−0.4 −0.2 0.0 0.2 0e+00 5e−04 1e−03

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 8 / 13

Heterogeneous Effects for Binary Outcome
Can’t assume a constant additive model for binary outcome
Back to Fisher’s exact test for 2 × 2 table
Long-term impact of class size:
Small class Regular-sized class

Graduate 754 892

Not graduate 148 189

Total 902 1081

Monotonicity assumption: Yi (1) ≥ Yi (0) for all i
Attributable effects = the number of successes in the treatment
group caused by the treatment:
n
X
A = Ti (Yi (1) − Yi (0))
i=1
Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 9 / 13
Attributable Effects (Rosenbaum. 2001. Biometrika)
Attributable effects as a pivot:
n
X n
X
S−A = Ti Yi (0), where S = Ti Yi (1)
i=1 i=1

Adjusted 2 × 2 table:
Treated (T = 1) Control (T = 0)
Pn Pn
Success (Y = 1) i=1 Ti Yi (0) i=1 (1 − Ti )Yi (0)
Pn Pn
Failure (Y = 0) i=1 Ti (1 − Yi (0)) i=1 (1 − Ti )(1 − Yi (0))

Total n1 n0
The reference distribution hypergeometric
Inversion gives a confidence interval on the attributable effect
STAR example: p-value = 0.55, 95% conf. int. = [0, 41]
Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 10 / 13
Heterogenous Effects for Continuous Outcome

Rank-sum test for continuous outcome with no ties

The reference distribution does not depend on unit index:

1
Pr(each set of ranks) = n


n1

Population inference is identical to sample inference

Non-sharp null hypothesis:

H0 : Pr(Y (1) ≤ y ) = Pr(Y (0) ≤ y ) for all y

The population shift model:

H0 : Pr(Y (0) ≤ y ) = Pr(Y (1) − τ0 ≤ y )

Inverting a rank-sum test a confidence interval

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 11 / 13
Point Estimation under the Population Shift Model
Hodges and Lehmann estimator (Hodges & Lehmann. 1963. Ann. Math. Stat.)
Idea: choose τ̂ to remove the treatment effect such that
Pr(Yi − τ̂ ≤ y | Ti = 1) = Pr(Yi ≤ y | Ti = 0) for all y

Wilcoxon’s rank-sum test:

n
X
Sτ = Ti Ri (Y − τ T)
i=1

Estimator:
n1 (n + 1)
Sτ̂ = E(S | On ) =
2
Computation: Sτ is non-increasing in τ
No exact solution:
1
τ̂ = [inf{τ : E(S | On ) > Sτ } + sup{τ : E(S | On ) < Sτ }]
2

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 12 / 13

Summary

Inference about effect size, going beyond hypothesis test

Inverting a permutation test gives a confidence interval
no asymptotic approximation

Overcoming the limitations of permutation inference

Heterogenous treatment effects and population inference

Reading: P. Rosenbaum (2010) Design of Observational Studies,

Springer. Chapter 2.

Kosuke Imai (Harvard) Permutation Inference Stat186/Gov2002 Fall 2019 13 / 13