You are on page 1of 7

Survey Course on Research Methods: Integrating Statistical

Analysis and Study Design

JoLaine R. Draugalis, Jean T. Carter and Marion K. Slack


College of Pharmacy, The University of Arizona, P.O. Box 210207, Tucson AZ 85721-0207

The purpose of the three-credit (45 hours) required course is to ensure PharmD student competence in
selecting, interpreting, and evaluating clinical drug literature. The course is a prerequisite for a three-credit
Drug Information/Drug Literature Evaluation course in the professional curriculum. Topics covered include
principles of research design, acquisition and interpretation of data, statistical analysis, and validity of
conclusions. The essential connection between research design and statistical analysis is emphasized
throughout the course in lectures, examples, assignments, and exams. The purpose of this article is to share
the methods used to convey statistical analysis and research design concepts as well as methods for
assessing the students. The techniques and approaches described are also applicable to other courses in
the pharmacy curriculum. Examples also show how instructors used evaluation and revision to cultivate
integration of concepts, encourage more active learning and class participation, and enhance assessment
procedures.

INTRODUCTION in moving the course from the final didactic semester (prior
In 1992, an article was published that provided a description to the final year of clerkship rotations) to first semester of
of a three credit (45 hours) required research evaluation the second professional year. The Drug Information/Litera-
course in an entry-level PharmD program(1). The article ture Evaluation (DI) course is now in the semester immedi-
concluded with a list of course improvement efforts includ- ately following the Methods course, rather than in the
ing content changes, additions of assignments, and examina- semester preceding it. The DI course was also increased by
tion improvement. In that more institutions will be offering one unit to three units. Some topics previously covered in
a course of this nature by virtue of increased numbers of the Methods course (e.g., Meta-Analysis, Proposal Writing,
entry-level PharmD programs, this update details our on- Evaluation Methods, and article critique) have been moved
going improvement efforts as well as other refinements of to the DI course, in order to dedicate more time to statistical
the past six years. The materials and methods of this course techniques/data analysis as an introductory biostatistics
address a number of specific educational outcomes and course is no longer a prerequisite. This decision was made
general ability-based outcomes (including thinking abili- because prerequisite course content and student retention
ties, communication abilities, responsible use of values and levels were so varied. However, as the prepharmacy curricu-
ethical principles, and self-learning abilities and habits) lum is evaluated over the next year, an entry level biostatis-
deemed requirements of a competent pharmacy practitio- tics course requirement will be revisited. The pharmaco-
ner(2). As well, the course meets Standards 12 (teaching and economic evaluation component was deleted several years
learning processes) and 13 (student evaluation) in the new ago as this topic is now covered in much more detail in the
ACPE accreditation manual(3). Particularly relevant are Professional Practice Management course.
Guidelines 12.2 and 12.3 which encourage teaching ap- The overall course purpose is to provide the student
proaches that require problem solving capabilities and pro- with the skills and principles of clinical research design and
mote life-long learning to in turn, ensure professional com- biostatistics needed for interpretation and evaluation of the
petence. Guideline 13.1 specifically argues for an evaluation medical literature and assessment of research reports and
system that examines higher levels of learning, such as proposals. The overview of statistics module begins with a
integration, application, and synthesis rather than simply quotation from H.G. Wells, “Statistical thinking will one
testing for rote memorization of specific facts. Finally, a day be as necessary for efficient citizenship as the ability to
philosophy of teaching as scholarship as espoused by Boyer read and write.” Our quest for conceptual understanding
guides instructors in that the goal is to transform and extend over mathematical “plug-and-chug” begins in earnest on
knowledge, rather than simply transmit information(4). day one and continues throughout the course. Teaching
Although this paper provides examples for a specific con- students to be healthy skeptics, arming them with tools to be
tent area, the methods embraced and the techniques and proficient professional decision makers, and challenging
approaches used most certainly can be applied to many them to accept responsibility for their actions are necessary.
other courses in the pharmacy curriculum. Encouraging open-mindedness and discouraging the urge
to over-generalize based on an N of 1 are crucial. The
COURSE DESCRIPTION
overriding objective is to ensure competence in selecting,
interpreting, and evaluating clinical drug literature, includ-
In order for students to use the skills gained in this course ing principles of research design, acquisition and interpreta-
throughout their study plan, a curriculum revision resulted tion of data, statistical analysis, and validity of conclusions.

American Journal of Pharmaceutical Education Vol. 62, Spring 1998 17


scientific prediction, describe an example of a cause-and-
effect relationship in health care, and identify the indepen-
dent (including levels) and dependent variables. Some re-
cent examples include: ill-fitting shoes cause blisters, sun
exposure causes skin cancer, drinking Tucson water causes
pregnancy, and completing lecture early results in class
dismissal. The class found out early-on that they would be
unable to test the last hypothesis. From day one, we begin
the discussion of plausible alternative hypotheses and the
role of research design in controlling these threats.
In Part 2, Statistical Techniques/Data Analysis, the
course notes provide definitions and compare and contrast
each of the statistical techniques. See Figure 2 for an in-class
exercise for the chi square test of distribution. The format
includes a very brief scenario, data display, null and alterna-
tive hypothesis, calculation example, and finally an exercise
where students use a table of critical values to ultimately
make a statistical decision.

RESEARCH DESIGN SECTION (Part 3 of Course)


The goal of the research design section is for students to
learn basic research design concepts. Specifically, students
should be able to distinguish among different types of
research design including experimental, quasi-experimen-
tal, pretest-posttest, and case studies, and associated threats
to internal validity when they complete the section. The
section is based on the presentation of research design and
internal validity by Campbell and Stanley(5). To learn
concepts, students must see examples of each type of re-
search design and learn to identify the critical characteris-
tics that differentiates one design from another(6). The
Fig. 1. Course outline. Each topic is covered in one 75-minute basic instructional strategy is to present a definition and
session unless otherwise noted. describe the critical characteristics of each research design
then to analyze a research report using that design. Follow-
ing the discussion of research designs, related issues such as
The specific objectives addressing this charge have not been types of randomization, how to handle compliance and
changed (other than deleting those related to deleted/reas- withdrawals in a clinical study, and how to interpret P-
signed material) since the original report was presented(1). values are presented in Part 4. Additional practice analyz-
The original report also details the specific learning objec- ing research reports is provided in the following semester in
tives. The course is now offered in two 75-minute class the drug information course.
periods per week (see Figure 1). Every effort is made to connect research design issues
to practice. To introduce the concept of research design, an
COURSE METHODS example study problem is presented. The current example
The first day of class we disseminate a 360-page course problem is a practitioner wanting to conduct a study to
notebook. This notebook contains the syllabus, lecture notes, determine if using community health workers will increase
reprints, objectives, and references to be used during the the number of people who complete prophylactic therapy
semester. Class materials, including in-class examples, are for tuberculosis. Students are presented with the question
chosen to make concepts more meaningful for students be it “How will we know if this program works?” and asked to
clinical or practice relevance. Articles and reports from the generate ideas about how they would answer the question.
medical literature as well as the lay press provide an endless Student responses are listed on the board and, as appropri-
source of new material. The course coordinator oversees all ate, their ideas are translated into research design language.
aspects of course preparation and planning and provides 80 For example, a student may express concern over the type
percent of the lectures. The other instructor provides 20 of education that the patients in the control group will
percent of the lectures. Occasionally, a guest lecturer may receive outside the study situation; this concern is trans-
provide one or two lectures (e.g., epidemiology topics). A lated into a concern with internal validity. Or a student
graduate teaching assistant is available to prepare and grade might suggest that a control group is needed which is
assignments. followed by a brief discussion of how to assign patients to
The course is still primarily lecture-based, however we groups.
have sought to provide increased opportunities for interac- To further increase relevance to a practice situation,
tion as well as in-class activities for certain topics. In fact, on external validity (generalizability) is presented as answer-
the first day of class, each of the instructors provides about ing the question “Do the results of this study apply to
25 minutes of lecture on terminology and the scientific patients that I see in my practice?” Students are directed
method and the remaining 25 minutes is spent having the toward the inclusion/exclusion criteria for the study and the
students (either in groups or the class as a whole) generate a description of the study sample in the results section. They

18 American Journal of Pharmaceutical Education Vol. 62, Spring 98


Fig. 2. Course notes example

are shown how to use the demographic and disease informa- onyms for a term (e.g., research design) to completing in-
tion to describe the sample and relate it to patients they class exercises such as using a random numbers table to
might see in their practice. assign patients to treatment groups or to completing a
The research design concepts are presented based on a worksheet identifying research design issues related to an
core knowledge approach. That is, certain concepts are example research problem.
considered critical to learning additional concepts and facts. We use an integrated approach to discussing issues
We believe that by learning to recognize basic research associated with research design such as internal validity
designs and the characteristics of those designs that students issues and statistical issues. For example, threats to internal
can refine their skills as they gain more experience reading validity and research design are integrated by asking stu-
research reports. For example, until students learn to recog- dents to know the primary threat or threats to internal
nize an independent versus a dependent variable or to validity associated with each type of research design. Spe-
differentiate an experimental from a pretest-posttest de- cifically, that experimental mortality (attrition) is the pri-
sign, they cannot hope to recognize threats to internal mary threat to internal validity for an experimental research
validity or identify whether the statistical tests reported are design and that differential assignment to groups (selection)
appropriate. Continued self-learning is very difficult if the is the primary threat for a quasi-experimental design. We
student cannot recognize the critical features of a research discuss how other threats to internal validity, for example
study such as the independent and the dependent variable. history or experimental mortality, may be issues with a
On the exam at the end of the section, students averaged 93 quasi-experimental design but students are tested only on
percent on the items testing recognition of research designs the primary threat. An average of 82 percent of the students
last semester. correctly identify the primary threat to internal validity for
Active participation in learning is included during class the four research designs presented. Presentation of rel-
time to the extent possible. Students are provided with a evant statistical tests proceeds in a similar manner. Statisti-
complete set of notes so that time is not required during class cal tests appropriate for use with an experimental design
to transmit information. Important points or suggestions include a Student’s t-test, an ANOVA, or a chi square,
about learning the material are provided during the opening depending on the type of data collected.
segment of the class followed by a class discussion or small To facilitate student learning of the critical characteris-
group activities. The activities range from generating syn tics of each research design, a cognitive map is provided. An

American Journal of Pharmaceutical Education Vol. 62, Spring 1998 19


example is shown in Figure 3. The cognitive map enables
students to focus on critical characteristics and to differen-
tiate relevant from irrelevant information. The students use
the cognitive map to analyze example research reports.
Students usually analyze six example reports. They are
provided with copies of the reports and the cognitive maps,
and asked to read the report and fill in the map before they
come to class. At the beginning of class, they turn in one copy
of the map (they keep a second copy for reference during
class) and then we discuss the characteristics of each study
and the issues associated with it. The discussion follows a
prescribed sequence in which the type of research is identi-
fied, the purpose of the study, the independent and depen-
dent variables, the statistical results, the primary threat to
internal validity and generalizability to patients seen in
practice. In general, the discussion is focused on research
design issues; strictly clinical issues may be mentioned but
are not discussed in depth.
Experience from teaching the course has shown us that
concept learning is not a linear, straight forward process.
Typically, students need to revisit basic concepts before they
can readily identify differences. For example, students can
have a great deal of difficulty learning to differentiate
random selection from random assignment. The topic is
introduced early in the course and revisited at every oppor-
tunity. In the 13th lecture (Topics in Sampling), the concepts
of random selection and random assignment are defined
and illustrated. Then as we analyze example research re-
ports, students are asked whether the researchers used
random selection or random assignment. We point out the
differences between the two techniques and the reasons
random selection is not used with clinical studies. When
students analyze example research reports, most students
can correctly differentiate random selection from random
Fig. 3. Cognitive map example. assignment. Last year, 52 of 53 students correctly identified
random assignment as the technique used to assign subjects
to treatment groups in an experimental study. Another

Fig. 4. Judgment steps for assessing a research report.

20 American Journal of Pharmaceutical Education Vol. 62, Spring 1998


difficult objective is learning to differentiate independent
from dependent variables; identification of variables is re-
visited during most research design lectures. On the relevant
exam questions, about 92 percent of the students answer the
questions correctly.
Experience also has shown us that students have a naive
misconception concerning the relative importance of inter-
nal validity versus external validity. Students typically judge
the quality of a research study by perceived external validity
and they tend to treat studies conducted in major research
and teaching hospitals as having little external validity. To
facilitate learning to assess internal validity first and exter-
nal validity only if the study is judged internally valid, a
sequential process is followed when discussing the report in
class. A supplemental handout (see Figure 4) also is used
that clearly delineates the sequence of steps in assessing a
research report: assessing the statistical conclusion, then
internal validity, and finally, external validity(7).
During an informal in-class evaluation of the research
design section, students were asked what they knew now
about research design that they didn’t know before. The
response of several students was “everything”, that they did
not know anything about research design before taking the
class. Several students commented that they liked the frame-
work (cognitive map) for analyzing the research reports and
one student stated they wished other professors would teach
this way. They also said the homework was very helpful and
they seemed to see value in the class. One student remarked
that the research design analysis was very basic, an observa-
tion that accurately reflects the level of the class.
Human subject considerations is the first topic in Part 4
(The Fundamentals of Clinical Trials). Real life examples of
unethical research and violation of informed consent are
used to underscore the need for institutional review boards’
critical analysis of research protocols. In addition, students
are shown official written informed consent and project
approval forms. Ensuing discussion invariably raises differ-
ing opinions leading to teachable moments about the role of
institutional review boards.
The final part of the class contains two sessions on each
of the topics, survey research and epidemiologic methods.
In survey research we concentrate on comparing and con- Fig. 5. Examples of assignment items over time.
trasting the four sources of error (coverage, sampling,
nonresponse, and measurement) as defined by Salant and ASSIGNMENTS
Dillman(8). Because surveys are so prevalent, it is a widely Assignments were only 10 percent of the semester grade
held belief that surveys are easy to conduct. Instructors when originally introduced to the course, now twenty-five
stress that survey research requires significant amounts of percent of the course grade is based on assignments. Five 20-
time, energy, money, and expertise; particularly when rela- point homework assignments are interspersed throughout
tionships are examined and concepts generated. Therefore, the semester serving as study guides for examinations. The
sampling, questionnaire construction, data analysis, and assignments are intended to provoke discussion and we
other technical aspects are covered. A journal reprint detail- encourage students to work together in completing course
ing the specifics of evaluating the results of mail survey objectives and homework.
research is provided to students(9). Homework assignments have evolved in several ways.
The epidemiology objectives were added to the course Items are based on scenarios from research reported in the
five years ago due to the increased numbers of literature literature. The format of items on the assignments was
reports using these methods. The primary objectives are: changed to include a mixture of multiple choice questions,
describe and evaluate the measures of disease in a popula- simple calculations, completion of tables, (e.g., source table
tion and compare and contrast the study designs used in for an analysis of variance or blocked random sampling
epidemiologic research with respect to their design and scheme). The question portion of the multiple choice items
analyses. Interpretation of relative risk ratios and odds- was converted from one-sentence statements about a con-
ratios and associated confidence intervals along with evalu- cept to simple scenarios that required the student to identify
ation criteria for cohort and case-control studies round out the concept based on the description. For example, rather
the materials. than ask the student which potential threat to internal
validity is most likely representing an experimental design,

American Journal of Pharmaceutical Education Vol. 62, Spring 1998 21


the student received a three or four sentence description of
a study with an experimental design (i.e., prospective, con-
current comparison groups with random assignment) and
was asked which potential threat to internal validity was
most likely with this study design. The identification re-
quirement was used to further discourage rote memoriza-
tion and provide experience in determining study designs
based on descriptions of research (see Figure 5).
The multiple choice items were further revised to in-
clude several questions with each scenario. This approach
has several advantages. Well-written scenarios and sets of
responses require a substantial amount of time and numer-
ous iterations. Using the same set of responses for a concept,
such as potential threats to internal validity for different
scenarios that may or may not have the same threats re-
quires students to compare and contrast the research de-
scribed. The assignment contains examples of experimental,
quasi-experimental, and crossover designs to provide the
student with convenient comparisons.
Another advantage of this assignment preparation strat-
egy is the ability to change scenarios from year to year
without having to completely re-write response items. Stu-
dents with access to homework assignments from previous
years should not have an advantage over students without
such access. The effect of access to the additional examples
may help the student learn the concepts.
The third advantage of this approach is the ability to use
reported research as the basis for the scenarios. On-line
searches provide quick and easy access to material for
scenarios that are based on actual research. The reports are
still cleaned up somewhat to help the student focus on the
statistical and design concerns, but some “red herrings” are
occasionally left in to maintain some sense of reality. The
use of reported studies begins to bring the students closer to
working with the literature. The scenarios may also include
tables of data or results, a correlation matrix, or a source Fig. 6. Assignment example of tests for categorical frequency data.
table to further simulate the literature. The final assignment
asks a series of questions about a recently published full
research article. change numbers in the small data set or use an entirely new
The response items used in the multiple choice items are scenario each year without changing the questions that
chosen carefully and the importance of knowing why each provide a model of an approach to statistical analysis. Figure
response is or is not correct is emphasized throughout the 6 shows the use of the template with the chi-square statistic.
semester. The information in the course builds on itself, so This commonly used statistic appears to be a source of
concepts covered early in the semester are woven into the confusion among practitioners and PharmD candidates who
assignments. For example, a scenario in the last assignment call seeking advice. It is hoped that this increased experience
may ask the student to determine the level of measurement with tests for categorical frequency data will improve under-
or identify the dependent variable. standing in the future.
After observing several episodes of a student copying Assignments have been based on practitioner inquiries
the answers to the multiple choice format, assignments were as well as the medical and pharmacy literature. Another
developed with approximately half multiple choice ques- successful approach has been to base an assignment on a full
tions and half simple calculation or other hands-on tasks. article (particularly appealing if research was conducted at
This was done to encourage students to spend more time the home institution). It has been very gratifying hearing
with the material and hopefully discourage last minute “hall” talk and seeing debates rage in the student lounge
copying. over these assignments. Students rated assignments very
The calculations are integrated into an overall schema favorably on course evaluations.
for conducting a statistical analysis. A template has been
developed that may be adapted to a variety of parametric EXAMINATIONS
and nonparametric procedures. A series of questions leads We assess application and other higher levels of learning, by
the student from the research question to the statistical constructing items using examples that require students to
hypothesis to be tested, selection of an appropriate test, apply concept skills rather than simply recognize or recall
calculation of the statistic, selection of a critical value, a information. The multiple choice format has been retained
judgment of whether the null hypothesis is retained or for several reasons: item analysis may be performed and,
rejected, and how this all relates to the original research tests may be machine graded, while retaining the ability of
question. This approach allows the course instructor to assessing higher levels of cognition (1,6). Examinations are

22 American Journal of Pharmaceutical Education Vol. 62, Spring 1998


evaluated and revised based on item analysis; such analysis Am. J. Pharm. Educ., 62, 17-23(1998);received 9/16/97, accepted 1/15/98.
differentiates a poorly written item missed by all levels of
References
performers from a well written item about a difficult concept (1) Draugalis, J.R. and Slack, M.K., “A course model for teaching re-
missed by all but the top performers. There are also “core” search evaluation in colleges of pharmacy,” Am. J. Pharm. Educ., 56,
or very basic items that everyone is expected to get right. 48-52(1992).
Each examination (three total) contains fifty multiple choice (2) Educational Outcomes, Center for the Advancement of Pharmaceu-
items and constitutes 25 percent of the course grade. tical Education Advisory Panel, American Association of Colleges of
Pharmacy, Alexandria VA (1994).
For the past five years, reliability coefficients have (3) Accreditation Standards and Guidelines for the Professional Pro-
typically ranged from 0.65 to 0.85 using the KR-20 estima- gram in Pharmacy Leading to the Doctor of Pharmacy Degree,
tion method. In that several assumptions (e.g., normal distri- Adopted June 14,1997, American Council on Pharmaceutical Educa-
bution of both item and test scores, some amount of varia- tion, Chicago IL (1997).
(4) Boyer, E.L., Scholarship Reconsidered - Priorities of the Professoriate.
tion in the item scores) are violated, this is a conservative The Carnegie Foundation for the Advancement of Teaching, Princeton
reliability estimate(10). However, reliability estimates for NJ (1990).
classroom examinations are typically much lower. A table of (5) Campbell, D.T. and Stanley, J.C., Experimental and Quasi-Experi-
specifications is constructed for each exam in order to assess mental Designs for Research. Houghton Mifflin Company, Boston
content validity(11). MA (1963).
(6) Merrill, M.D. and Tennyson, R.D., Teaching Concepts: An Instruc-
tional Design Guide. Educational Technology Publications, Englewood
FUTURE PLANS Cliffs NJ (1977).
We continue to evaluate and refine the course based on (7) Cook, T.D. and Campbell, D.T., Quasi-Experimentation-Design &
Analysis Issues for Field Settings, Houghton Mifflin Company, Boston
instructor, teaching assistant, and student input. Summative MA (1979), pp. 37-39.
instructor and course evaluations are conducted each time (8) Salant P. and Dillman D.A., How to Conduct Your Own Survey, John
the course is offered using standardized forms. Students Wiley and Sons, New York NY (1994).
report that they use the course notes as a reference through- (9) Harrison, D.L. and Draugalis, J.R., “Evaluating the results of mail
out the remainder of the program and in some cases, at their survey research,” J. Am. Pharm. Assoc., 37, 662-666(1997).
(10) Speedie, S.M., “Reliability: The accuracy of a test,” Am. J. Pharm.
residency or practice site. We continuously gather and inter- Educ., 49,76-79(1985).
pret quantitative data on student, instructor, and course (11) Kimberlin, C.L. “Characteristics desired in tests: Validity,” ibid., 49,
performance. As suggested by Miles and Huberman, we 73-76(1985).
find qualitative information serves as an additional rich (12) Miles, M.B. and Huberman, A.M., Qualitative Data Analysis: An
source to validate and reinterpret the quantitative find- Expanded Sourcebook, 2nd Ed., Sage Publications, Inc., Thousand
Oaks CA (1994) p.10.
ings (12). In the future we hope to have the students manipu-
late data sets using statistical software programs.

American Journal of Pharmaceutical Education Vol. 62, Spring 1998 23