Case Questions:

1. Compute the percentage of supporters who pledge their support on social media for
each of the three variants. Are there any variants statistically better than Variant 1 at a
2.5% level?
Hint: You probably want to use an Excel Pivot Table to compute all the relevant
information needed for the formula we used in class.

Percentage of supporters who pledge their support on social media (Variant 1): 6.4%
Percentage of supporters who pledge their support on social media (Variant 2): 6.4%
Percentage of supporters who pledge their support on social media (Variant 3): 6.6%

There are no variants statistically better than Variant 1 at a 2.5% threshold.

2. Repeat the computation in Question 1, but separately for each party affiliation. Now
are any methods statistically better than Variant 1 at a 2.5% level? If so, which Variants
are better than Variant 1 for which parties?

There are two methods statistically better than Variant 1 at a 2.5% level. For Independent-
identifying voters Variant 2 leads to an increase in shares, suggested by a .0006 p-value. For
other-identifying voters Variant 3 led to an increase suggested by a p value of 5.03699E-05.
These p values are lower than the tested significance level, demonstrating that they are
statistically significant values.

3. Think carefully about what you’ve done in Question 1 and Question 2. What does this
tell you about parties as confounding variables?

Because there are mixed results and inconsistent p-values, we can infer that parties are a
confounding variable. If they had the same/similar p values than the general population, they
would most likely not be confounding.

4. Your very smart data scientist friend points out that because of multiple testing, it’s not
the case that both your conclusions in parts 1 and 2 hold tRather she says, they hold at a
higher significance level. Do you agree? What level? Why?

I agree that they hold at a higher significance level because we would have to add all
significance thresholds according to how many tests we have conducted. In other words, the
new significance level would be 20% because we conducted 8 tests and each test has a
significance threshold of 2.5. The chance of making one mistake in all of the tests needs to be
under 20% for us to reject the null hypothesis.

5. Compute the probability a supporter donates under each variant. Do your analysis
separately for each party. Are any variants statistically better than Variant 1? If so,
how, and at what significance level?
Democratic
Probability Variant 1 = .055
Probability Variant 2 = .0703
Probability Variant 3 = .0475
Independent
Probability Variant 1 = .0523
Probability Variant 2 = .0505
Probability Variant 3 = .0366
Other
Probability Variant 1 = .0466
Probability Variant 2 = .045
Probability Variant 3 = .0319

Yes, Variant 2 for Democratic-identifying voters is statistically better than Variant 1

because it has a p value of .0035 at a significance level of 5%, or any threshold higher than 3.5%

6. Compute the average amount a supporter donates under each variant. Do your analysis
separately for each party. Are any variants statistically better than variant 1? If so, how,
and at what significance level?
Democratic
Average Amount Variant 1 = $20.9
Average Amount Variant 2 = $21.22
Average Amount Variant 3 = $37.22
Independent
Average Amount Variant 1 = $21.33
Average Amount Variant 2 = $20.87
Average Amount Variant 3 = $21.61
Other
Average Amount 1 = $20.39
Average Amount 2 = $21.08
Average Amount 3 = $26

Yes, Variant 3 for Democratic and Other voters is statistically better than Variant 1 because they
have p values of 2.0906E-173 and .0007 respectively. They hold at any threshold above those
values or at a normal 2.5% significance level.

7. Compute the probability a supporter signs up for the newsletter under each variant. Do
your analysis separately for each party. Are any variants statistically better than variant
1? If so, how, and at what significance level?
Democratic
Probability Variant 1 = .0328
Probability Variant 2 = .0156
Probability Variant 3 = .0145
Independent
Probability Variant 1 = .0219
Probability Variant 2 = .0187
Probability Variant 3 = .0178
Other
Probability Variant 1 = .0324
Probability Variant 2 = .0189
Probability Variant 3 = .0206

No, there are no variants that are statistically better than Variant 1 for this data type. All p-
values are higher than a usable threshold.

8. Compare all of your answers above. Which variant would you recommend and why?
Justify your opinion quantitatively.

We would recommend variant 3, as it led to benefits with donation amount and in sharing. It led
to an increase in “Other” voters’ sharing and held at a 2.5% significance level. It also led to an
increase in donation amounts for Democrat and “Other” voters, both holding at a 2.5%
significance level.