Worksheet 8

Electricity Pricing Policy Experiment

Many countries face electricity supply shortages that severely affect economic activities.
Developing counties such as China and India are experiencing rapid increases in energy
demand. It usually takes substantial time and investment to build new power plants to meet
such demand. Developed countries such as the US and Japan have substantial electricity
generation capacity, but these counties also often experience “electricity crises” when there is a
demand shock (e.g. heat wave) or a supply shock (e.g. accidents at nuclear power plants) that
result in interrupted electricity service. Importantly, in any electricity system, supply has to
meet demand all of the time. Otherwise, large-scale blackouts happen.

In this handout, we will learn how to use experiments to make policy recommendations. You
are a policymaker in the Ministry of Economy in Japan. There is a serious shortage of electricity
because many nuclear reactors were shut down after the Fukushima nuclear disaster. You are
trying to decide if a new electricity pricing policy can motivate households to conserve
electricity.

The new electricity policy charges a lower price per Wh (20) if the total consumption of the
household is smaller than a threshold and higher if it is above (30). The threshold varies with
the number of people in the household. Assume under the old policy you pay 25 regardless of
your consumption.

You have been asked to provide a policy recommendation whether this new pricing is sufficient
to avoid blackouts in light of the following facts:
- Blackouts can be avoided if households reduce their hourly electricity consumption by
200Wh on average.
- If the reduction is less than 200Wh, there will be large-scale blackouts.

First Experiment: Experiment with self-selected treatment group

You decide to run a field experiment to analyze the effect of the new pricing policy. Out of the
entire Japanese population, you choose a random sample of 1,000 households.

In this experiment, you ask your sample households whether they wanted to try the new
electricity pricing or not. Households signing up for the program receive a rebate check to
incentivize their sign up. 500 households voluntarily sign up for the new pricing policy
(treatment group) and the remaining 500 households declined (control group).

You run the experiment and collect electricity consumption data (hourly electricity
consumption in kWh) during the experiment. You run a simple regression:
^
Consumptioni=b 0 +b1 Treatment i ( 1 )
where Consumptioni is household i’s electricity consumption and Treatmenti is an indicator
variable that equals 1 if household i is in the treatment group and 0 otherwise.

1. Based on your intuition, what types of households are more likely to sign up? What types of
households are less likely to sign up? Do you think it creates bias for our experiment?

Environmentally orientated households are more likely to sign up for the new pricing
policy. Household income may also be a factor that would influence households signing
up to try the new pricing policy. Larger households are less likely to sign up. Yes, the
omission of these variables in a self-selected sample will likely bias the treatment
coefficient.
2. Open the dataset Resources→Datasets→Handout 17 Electricity Experiments.xlsx. Use the
sheet “Self-selected treatment.” First, make an indicator variable for Treatment, which = 1 if
the household is in the treatment group and = 0 if the household is in the control group.

Regression 1 with Self-Selected Treatment Group

Dependent Variable: Electricity consumption

Coefficient Standard Error t Stat P-value

Intercept 1181.805 11.109 106.388 ≈0

Treatment -519.813 15.710 -33.089 ≈0

N = 1000 , R2 = 0.523

Interpret the coefficient on Treatment.

On average, households that self-selected into the treatment group used approximately 520
Wh less electricity than the control group.

Calculate the average consumption for the treatment group and for the control group.

Avg consumption treatment group: On average, “treated” households used 520 Wh less
electricity than those in the control group. Therefore, the households who volunteered to be
subject to the new pricing policy, used on average 662 Wh (1182Wh-520Wh) – 44% less than
the control group.

Avg consumption control group: On average, the control households used 1182 Wh.

3. Provide the 95% confidence interval (CI) for the coefficient on Treatment.

The 95% confidence interval is from -489 to -551 Wh. I.e., we are 95% confident that
treated household would consume between 489 and 551 kWh less electricity compared to
non-treated households.

4. A major concern for this experiment is that you do not know if the treatment group’s
consumption is lower (1) because of the effect of the new electricity pricing or (2) because
electricity consumption is different between the treatment and control groups for reasons
unrelated to the new electricity pricing. To figure out who would “self-select” into the
treatment group, run regressions to compare the mean of the following variables for the
treatment and control group. Each regression has the form:
Predicted Variable = bo+b1*Treatment
Average differences in variables for Treatment and Control Groups in the first experiment
Coefficient on Treatment Statistically significant?
Number of air conditioners -0.56 Yes, t=-6.38

Square feet of the house -134 Yes, t=-16.23

Energy efficient house 0.32 Yes, t=10.67

(0 or 1)

5. Remember that blackouts can only be avoided if households reduce their hourly electricity
consumption by 200Wh on average. Describe your policy recommendation based on this
analysis.

Although 200 falls outside the confidence interval (and thus satisfies the policy
requirement), I wouldn’t make any recommendation as the effects are clearly biased.
Therefore, one cannot really tell based on the estimates if the policy would actually
reduce consumption by 200 if implemented.
 Second Experiment: Experiment with randomly-assigned treatment group
Understanding the problems associated with self-selected group assignment, you decide to do
another experiment in which you randomly assign treatment. That is, you randomly assign 500
households to the treatment group and assign the other 500 households to the control group.

The advantage of a “random” assignment is that in theory, the two groups are statistically
equivalent except for their treatment status. Let’s see if this is “actually” true for our data.

1. Let’s look at the sample means in a different worksheet of the same variables as earlier. In
the dataset Resources→Datasets→Handout 17 Electricity Experiments.xlsx, use the sheet
“Randomized_Treatment.” Fill in the table below.

Average differences in variables for Treatment and Control Groups in randomized

experiment
Coefficient on Treatment Statistically significant?
Number of air conditioners 0.016 No, t=0.17

Square feet of the house -6.04 No, t=-0.15

Energy efficient house -.004 No, t=-0.12

(0 or 1)

Run a regression of Energy Consumption on whether or not the household is assigned to the
treatment group

Regression 2 with “Randomly assigned” Treatment Group

Dependent Variable: Electricity consumption

Coefficient Standard Error t Stat P-value

Intercept 1029.803 15.344 67.114 ≈0

Treatment -215.81 21.70 -9.945 ≈0

N = 1000 , R2 = .0893

2. Interpret the coefficient on Treatment.

Based upon the regression shown in Q2, the coefficient for the treatment group indicates
that, on average, those households used 216 Wh less electricity than did the households
in the control group.
3. Provide the 95% confidence interval (CI) for the coefficient on Treatment

The 95% confidence interval is from -173 to -258 Wh. I.e., we are 95% confident that the
true impact of the new pricing policy is a reduction in electricity usage between 173 and
258 Wh.

4. Describe your policy recommendation based on Regression 2.

Since the 95% confidence interval includes numbers less than 200 Wh it is unclear that the
new pricing policy will result in the required electricity savings. Therefore, we cannot be
certain in recommending the new pricing policy.

5. In Regression 2, we did not include any control variables other than “Treatment”. Do you
expect that omitting these variables will bias the results? Why or why not?

We do not expect any omitted variable bias. This is because the “treatment” is randomly
assigned. For example, see your pivot table above in Q1. All characteristics are balanced
between the treatment group and the control group. Intuitively, we can interpret that the
two groups are statistically very similar in their characteristics so that “their only
difference” is the treatment status (i.e. one group gets new pricing and another group
does not). That is why in theory, there is no concern for bias.

Lessons from this exercise:

Conducting a proper randomized experiment is very important because, otherwise, you might
make a wrong decision about a very important policy question. In many business, governmental
and not-for-profit enterprises, policy decision-making requires an accurate measure of the
effect of X on Y. More and more organizations are using randomized experiments to obtain
unbiased estimates of the impacts of their decisions.