Topics in Business Strategy: Experiments in Organizations

Spring 2020

PROFESSOR: Sally Sadoff

EMAIL: ssadoff@ucsd.edu
PHONE: (858) 822-7456
OFFICE: Wells Fargo Hall, Room 4W121
OFFICE HOURS: By email

TA: Ariel Fridman

EMAIL: ariel.fridman@rady.ucsd.edu
OFFICE HOURS: By email

DESCRIPTION

Businesses are increasingly using experimentation in a wide range of areas including product
pricing, marketing strategies, employee incentives and shaping consumer behavior. These
experiments can be particularly effective if the interventions tested incorporate insights from
behavioral economics. However, many companies currently lack the knowledge necessary to
take advantage of experimentation. An understanding of how to design and run a good
experiment will position students at the cutting edge of decision-making in business.

The goals of the course are for students to understand: (1) the role of experiments in business
(2) how insights from behavioral economics can be incorporated into experimental interventions
and (3) how to design and run an experiment in an organization.

We will begin by discussing the role of experiments in organizations and examining some recent
examples. Each student will be responsible for presenting a paper that describes an experiment
run in a company or non-profit organization.

Similar to the Lab to Market model, students will then work individually or in teams to develop an
experimental question and design a randomized experiment to test their question of interest. In
the design phase, we will discuss methods of experimental design as well as strategies for
working with an organization to implement experiments successfully. Students will run pilot
studies to inform their designs. At the end of the course, students will present a proposed
experimental design and implementation strategy.

About half of class time will consist of lecture and case discussion. The other half will be spent
discussing students’ proposed experiments and providing feedback at each phase of the design.
The course is designed to give students the practical tools they need to successfully run
experiments in organizations. Guest speakers will share their experience designing and
implementing experiments in a wide range of companies.
OBJECTIVES
At the close of this course you will be able to:

• Understand the role of experimentation in business

• Incorporate insights from behavioral economics into experimental interventions
• Design and implement randomized experiments
• Develop experiments that address the interests and concerns of companies and non-profits

MATERIALS
The materials listed below are required readings for Flex Evening and Flex Weekend students.
They are recommended readings for MSBA and Full Time students.

Nudge, Richard H. Thaler and Cass R. Sunstein (ISBN 9780143115267)

The Why Axis, Uri Gneezy and John List (ISBN 1610393112)

Running randomized evaluations: A practical guide, Rachel Glennerster and Kudzai Takavarasha
(ISBN 9780691159270), referred to below as GT

Additional papers used in the course will be made available electronically.

GUEST SPEAKERS
We will have guest speakers running experiments with organizations including Google, Lyft and
HP.

ASSIGNMENTS

Article presentation: Students will present a paper describing an experiment run in an

organization. The reading list includes example papers. You may present individually or in a
group. You will also provide feedback to presenters.

Experimental proposal: Over the course of the class students will work individually or in groups
to develop a proposal for an experiment to be run in an organization. During the last week of
classes, students will make a formal presentation of their experimental “pitch” as if presenting to
the key decision maker in the organization. Your grade will be based both on your final proposal
and your development of the proposal throughout the class.

Pilot study: Over the course of the class, students will design, run and analyze a pilot study to
provide initial feedback for their experimental proposals. The lessons learned from the pilot should
be incorporated into the final proposal. You will also provide feedback to other groups’ pilot
studies.

GRADING
• Participation in course discussion 20%
• Lead discussion of an experiment 20%
• Pilot study design and presentation 30%
• Experimental presentation and proposal 30%
ACADEMIC INTEGRITY

Integrity of scholarship is essential for an academic community. As members of the Rady School,
we pledge ourselves to uphold the highest ethical standards. The University expects that both
faculty and students will honor this principle and in so doing protect the validity of University
intellectual work. For students, this means that all academic work will be done by the individual to
whom it is assigned, without unauthorized aid of any kind.

The complete UCSD Policy on Integrity of Scholarship can be viewed at:

http://senate.ucsd.edu/manual/appendices/appendix2.pdf

STUDENTS WITH DISABILITIES

A student who has a disability or special need and requires an accommodation in order to have
equal access to the classroom must register with the Office for Students with Disabilities (OSD).
The OSD will determine what accommodations may be made and provide the necessary
documentation to present to the faculty member.

The student must present the OSD letter of certification and OSD accommodation
recommendation to the appropriate faculty member in order to initiate the request for
accommodation in classes, examinations, or other academic program activities. No
accommodations can be implemented retroactively.

Please visit the OSD website for further information or contact the Office for Students with
Disabilities at (858) 534-4382 or fosorio@ucsd.edu.
COURSE SCHEDULE
Below is an overview of the course schedule. Depending on the progress in class, some changes
may occur during the quarter. All times listed below are Pacific time.

Class Topic(s) Reading(s) Deliverables(s)

1 Introduction to Activity: Discuss topic of
April 2 experiments https://80000hours.org/articles/can- experimental interest
you-guess/

Article: “A step-by-step guide to

smart business experiments”

Running randomized evaluations:

Chapter 2, modules 2.1-2.3 (pp.
24-47)
2 Behavioral Nudge, Introduction and Part I
April 9 economics in
experimental The Why Axis, Chapter 1
design

3 Randomization Running randomized evaluations: April 17, 5pm: Submit

April and reading Chapter 4, modules 4.1-4.2 (pp. choice of article to
16 experiments 98-120), 4.4-4.5 (pp. 140-161) present and sign up for
presentation slot.
4 Pilot studies & Article: “The surprising power of April 22, 12pm: Submit
April online online experiments” group members and
23 experiments experimental question
Article: “Online field experiments: A
selective survey of methods” April 24, 5pm: Submit
article presentation and
sign up for presentation
feedback slot.

5 Guest Speaker: April 29, 12pm: Submit

April Michael Perry, 1-page pilot study
30 Google description (1 per group)

Pilot Design
Workshop
6 Statistical Running randomized evaluations: May 5, 12pm: Submit
May 7 analysis for Chapter 6, modules 6.1-6.5 pilot study feedback
experiments
May 8, 5pm: Submit
revised 1-page pilot study
description (1 per group).
 7 Ethics of 3 Articles: OK Cupid, Facebook, May 13, 12pm: Submit ½
May experimentation Election page progress report
14
8 Perfecting your May 20, 12pm: Submit ½
May pitch page progress report
21
9 Presentation May 27, 12pm: Submit
May preparation your pilot results (2-3
28 figures)

10 Present experimental proposals

June 4
June 3, 12pm: Submit your experimental proposal

SCHEDULE

Week 1 (April 2) – Introduction to experiments in organizations

We will discuss the role of experimentation in companies, the basic elements of experimentation
and how to read research papers describing experiments.

Students will present an experimental question they would like to explore and receive feedback
from the class.

Assignment:

Activity: https://80000hours.org/articles/can-you-guess/
Go to think link above and click on “Start the Game.” Go through the ten interventions and
choose whether you think they worked.

Reading: “A Step-by-Step Guide to Smart Business Experiments,” Eric T. Anderson

and Duncan Simester, Harvard Business Review, March 2011: This article provides a
useful introduction to running experiments in companies. You should read the article and be
prepared to discuss it. The article will be made available electronically.

Reading: Running randomized evaluations: Chapter 2, modules 2.1-2.3 (pp. 24-47). This
chapter discusses both experimental and non-experimental methods for measuring causal
impact. You should understand what causal impact is and be prepared to discuss the
advantages and disadvantages of the different methods.

Deliverable: Experimental question: You should be prepared to discuss at least one topic
or question you would like to explore using an experiment.

Week 2 (April 9) – Behavioral economics in experimental design

We will discuss how incorporating insights from behavioral economics can strengthen your
experimental design. Students will workshop the design of experiments that draw on concepts
from behavioral economics.
Assignment:

Readings: Nudge, Introduction and Part I, and The Why Axis, Chapter 1. The readings
will provide the basis for the workshop on incorporating insights from behavioral economics
into experimental designs. Students should be prepared to apply the concepts from the
readings.

Week 3 (April 16) – From experimental idea to experimental testing

We will discuss moving from an experimental idea to actual testing. We will begin with the basics
of experimental design with a focus on randomization. And we will discuss how to read and
present experimental papers in preparation for students to present their own experimental articles.

Assignment:

Reading: Glennerster and Takavarasha (GT) Chapter 4, modules 4.1-4.2 (pp. 98-120),
4.4-4.5 (pp. 140-161) This chapter will help students identify opportunities for experimentation
and provides a critical overview of randomization. Students should be able to apply the
concepts in the reading to the evaluation and design of experiments.

Deliverable: Submit your first and second choice of article to present; and sign up for
a presentation slot: You will lead discussion of an experiment run in a company chosen from
the list of suggested articles. The TAs or I can give you guidance on which article to choose
and we can also work together to find additional papers. You may present individually or in a
group of up to 2 people. Submit your first and second choice article. Articles will be assigned
first come, first served. Submission are due no later than Friday April 17 at 5pm (1 submission
per group). You should also sign up for a presentation slot.

Week 4 (April 23) – Pilot studies and online experiments

We will discuss the design and implementation of pilot studies. Students will participate in a
workshop on running experimental studies online. Students will also discuss their proposed pilot
study design.

Assignment:

Reading: “The surprising power of online experiments,” Ron Kohavi and Stefan
Thomke, September-October 2017. “Online field experiments: A selective survey of
methods,” Yan Chen and Joseph Konstan, Journal of the Economic Science
Association 1(1), 2015. These articles provide an overview of how experiments can be
conducted online, which will be the focus of the class. It will be made available electronically.

Deliverable: Submit your group members and experimental question: Students should
come to class prepared to discuss the experimental question that they will work on for the
remainder of the class either individually or in groups of up to 4. Submissions are due
Wednesday April 22 at 12pm (1 submission per group).
 Deliverable: Submit your article presentation and sign up for a presentation feedback
slot: Submit the slide presentation you will use when leading discussion of your chosen
article. You should also sign up for a slot in which you will provide feedback to another
presentation. Submissions are due Friday April 24 at 5pm (1 submission per group).

Week 5 (April 30) – Pilot study workshop

Guest speaker Michael Perry will discuss his experience combining experiments and data
analytics at Google as Statistics Team Lead and Data Scientist - Advanced Measurement
Technologies Team.

Assignment:

Deliverable: Pilot study design and implementation (1 page): Submit your pilot study
design, implementation plan and summary of early implementation. You should include the
following: context/setting/population, sample size, treatment(s), randomization procedure and
details of implementation. Submissions are due Wednesday April 29 at 12pm (one submission
per group).

Week 6 (May 7) – Statistical analysis for experiments

We will discuss the statistical analysis of experiments with a focus on statistical power. Students
will participate in a workshop analyzing data and calculating statistical power.

Assignment:

Reading: Glennerster and Takavarasha (GT) Chapter 6, modules 6.1-6.5. This chapter
discusses statistical analysis and statistical power. Please read the chapter carefully as
students will be required to apply the concepts in their final experimental proposals.

Deliverable: Pilot study feedback (1 page): Submit feedback you provided to one other
group on their pilot study design. Submissions are due Wednesday May 6 at 12pm (one
submission per group).

Deliverable: Revised pilot study design (1 page): Submit your revised pilot design based
on feedback you received. Submissions are due Friday May 8 at 5pm (one submission per
group).

Week 7 (May 14) – Organizational constraints and ethics of experimentation

We will discuss organizational constraints on experimentation including ethical concerns.

Assignment:

Reading (3 articles): “OKCupid plays with love in users experiments”, “Were

OKCupid’s and Facebook’s experiments unethical?” and “Professors’ research project
stirs political outrage in Montana” The articles will be made available electronically.
 Students should read the articles carefully and participate actively in a discussion of ethics in
experimentation.

Deliverable: Progress report (1/2 page): Submit an update on the status of your pilot study
implementation. Submissions are due Wednesday May 13 at 12pm (one submission per
group).

Week 8 (May 21) – Perfecting your pitch

We will discuss how to build on the results of the pilot studies to create an experimental “pitch” to
key decision-makers in a company.

Assignment:

Deliverable: Progress report (1/2 page): Submit an update on the status of your pilot study
implementation. Submissions are due Wednesday May 20 at 12pm (one submission per
group).

Week 9 (May 28) – Preparation for final presentations

Students will work with the data from their pilot studies and prepare for their final presentation.

Assignment:

Deliverable: Pilot study results (2-3 tables and/or figures): Submit 2-3 tables or figures
that summarize your pilot study results. Submissions are due Wednesday, May 27 at 12pm
(one submission per group).

Week 10 (June 4) – Experimental proposals

Students will present their full experimental proposals including their experimental question,
experimental design and implementation strategy. Students will also provide feedback as the key
decision-maker in the company.

Assignment:

Deliverable: Final experimental proposal: You (or your group) will present your final
experimental proposal. You should prepare the presentation as if you were pitching the key
decision maker in the company where you propose to run your experiment. The “Final
Presentation Guide” outlines the topics you should cover. Submissions are due Wednesday
June 3 at 12pm.