You are on page 1of 5


Copyright O 1980 by The Orthopaedic and Sports Medicine Sections of the American Physical Therapy Association

Critically Reading a Research Article*


This paper presents a method for critically reading a research article. Emphasis is
placed on the specific format of the question being asked, the control of variables
to ensure internal and external validity, selected basic rules for applying statistics,
and the justification for specific conclusions drawn by the investigator. The
method is designed to be clinically relevant with the final evaluation stressing
improved patient care. This format has been effectively used by students of the
Department of Physical Therapy at the Medical College of Georgia for critiquing

The techniques of research are frequently for- The results section, which follows the meth-
eign to clinicians, and yet research .offers the odology, should concisely state the statistical
justification and validation for our livelihood. We outcome of the data that was collected. No opin-
should make every effort to investigate our prac- ion should be introduced at this point, only the
tice scientifically, validate the effective, eliminate pertinent data.
the ineffective, develop the undiscovered, and In the next section, the conclusion or discus-
critically analyze the published literature. To that sion, the author should relate the data to the
end, therefore, this article presents a method previously mentioned pertinent literature, and
which will provide clinicians with the basic steps clinically relevant conclusions should be offered.
necessary to analyze an article and determine If the author has asked a clinically important
whether its results are worthy of clinical appli- question, implemented an unbiased plan to an-
cation in their own practices. swer the question (internal validity), and statisti-
cally analyzed the data appropriately, then and
OVERVIEW only then can the reader begin to generalize the
information to patient populations outside the
The critical review of research must proceed subject sample of the investigation (external va-
along a specific path which may vary slightly lidity). If appropriate, the reader can then use
depending upon the type of research conducted. the information in the treatment of patients.
Most reports of inferential investigations should Let us offer a word of caution. Medicine is still
start with an introduction that includes, most an art, and therefore many gray areas exist. The
importantly, the statement of purpose or the intelligent clinician will take pieces of information
statistically testable research question which the from many articles and combine them in the
study was designed to answer. The clinical ap- treatment of patients. Your own particular clinical
propriateness of this question should be justified situation will influence how the information ob-
by a review of the literature, which is part of the tained in the literature will be used.
introduction. The methodology, the next section The various parts of a research article have
of the report, should explain the research design been quickly reviewed. A more detailed exami-
and the steps taken to answer the question. nation of what should be in those parts and how
Since the experiment should be reproducible, they can be read with a critical eye is now in
the report should be clearly and precisely stated. order.

* This paper was supported in part by Grant 5D12 AH 01 125 from THE QUESTION
the Department of Health, Education, and Welfare. Bureau of Health
t Assistant Professor. Division of Physical Therapy. West Virginia Every article published should have a stated
University Medical Center. Morgantown. WV 26506.
$ From the Department of Physical Therapy, School of Allied Health
purpose. The reader should not have to search
Sciences. Medical College of Georgia. Augusta. GA 30912. for the purpose but should find it stated explicitly
in the introduction of the paper. The purpose ysis will provide the reader with the type of
may take the form of a statistically testable re- relationship, direct or inverse. The typical cor-
search question, a research hypothesis, or a null relational question is stated: "Is there a relation-
hypothesis or it can be editorialized into the ship between X and Y?" For the purpose of
statement of purpose. If the statement is not drawing conclusions from any study, the corre-
specific, the investigator either was not clear lational question is stronger than the descriptive
about what he or she was asking or did not know but not as strong as the comparative.
how to ask it. If the question is not stated, you The comparative question is used to determine
cannot be sure that the study pursued a specific cause and effect relationships. It is based on the
question. statistical assumption that there is no difference
Once you know what is being asked, identify between any two treatments or between one
the question as descriptive, correlational, or treatment and no treatment at all (the null hy-
comparative. The descriptive question is the pothesis). The basic question asked then is: "Is
most basic question an investigator can ask. It there a difference between X and Y?" This ques-
seeks to describe or define something not yet tion can be answered by a yes or no and can be
studied or something that has been studied from analyzed statistically.
another point of view. The descriptive study In reviewing the question, make sure that it is
forms the base of the pyramid of research knowl- justified by appropriate literature review. Perti-
edge (Fig. 1). Each of the other questions is nent references should be cited to support the
based on a description of what is occurring. concepts being studied, the reasons for the
Descriptive questions frequently begin with the study, and the appropriateness of the study.
words "what," "how," or "how much." They After a question is well formulated, the author
usually do not lend themselves to anything other should present the plan for answering it. This is
than descriptive statistics. The descriptive ques- done in the methodology.
tion "What is the effect of heat on nerve excita-
bility?" cannot be answered by either a yes or
no and is not statistically testable. Be cautious METHODOLOGY
of the author wh'o reports statistical significance
for a descriptive question. If this occurs, the The methodology is a precise plan of action
author has either applied inappropriate statistics which the investigator follows to answer the re-
to the original question or tested a question that search question. It should not reflect any opin-
was not originally asked. Analysis of a question ions and should be written in language exacting
not proposed prior to the collection of data easily enough to allow precise duplication of the study.
lends itself to bias. If it is not reproducible, the method by which the
Correlational and comparative questions, the authors attempted to answer their question is not
next two levels of the pyramid of research, are totally known and is, therefore, difficult to as-
inferential and statistically testable. The corre- sess.
lational question is used to relate independent The methodology should state exactly how
characteristics in the same subject. The question specific variables were controlled. This control
of whether the characteristics are or are not is one of the most important aspects of the study.
related can be answered with a yes or no and, Good control of all the variables that can influ-
therefore, can be analyzed statistically. The anal- ence the outcome (high internal validity) results
in a meaningful study. If important variables are
not controlled (low internal validity), the findings
of the study are greatly compromised. Readers
must be extremely wary of poor control of vari-
ables for all studies.
The methodology should also state precisely
how the characteristics or variables were mea-
sured. Any methodology that does not provide
the reader with an exact description of what was
measured as well as how well it was measured
should be questioned. In a descriptive study, the
author must have precisely identified the variable
Fig. 1. The pyramid of research knowledge. or variables under examination.
74 HART ET AL JOSPT Vol. 2, No. 2

To gain a general idea of the characteristics A third question is: "Were extraneous vari-
or variables that should have been considered, ables controlled?" As discussed earlier, this is
the reader should look back to the research a critical concern in reviewing an article. Con-
question and ask: "What logically can affect the sider the reasonableness of comparing coordi-
outcome of this study?" Items such as age, sex, nation of a 20-year-old to a geriatric patient,
past medical history, and cultural background cardiac work capacity of men and women, or
may or may not affect various studies. The interactive styles of inner-city children and rural
reader should be aware of how the author has children. In each case, at least one obvious
controlled the impact that each trait has on the variable was not controlled, i.e., age, sex, and
characteristics being studied. One means of con- socioeconomic interaction, respectively. These
trolling variables is to select subjects with similar variables will bias the data.
characteristics (age, weight, sex). Anyone not The reader of investigative studies must differ-
fitting the predetermined criteria would not be entiate between characteristics that influence
included in the study, since they may adversely the collection of data and those that may not.
affect the measured characteristic. In general, Generally, if the subjects are homogeneous in
greater control of variables in a study leads to demographic data, then the study will be more
stronger conclusions that can be made from that specific and generalizable to a smaller, particular
study. population. If the subjects represent a more var-
To review a study for internal validity (the ied, heterogeneous population, the data should
control of variables), the reader should ask sev- be generalizable to a larger but still particular
eral questions. First, "How many subjects are population.
there?" The number of subjects should be suf- The next question a reader might ask in re-
ficiently large to represent the size of the popu- viewing an article could be: "Were the subjects
lation from which the sample was drawn. For placed into groups, and if so, how?" This ques-
example, if the population is extremely small and tion applies specifically to comparative studies.
homogeneous, the sample may be correspond- Usually, at least two groups receiving different
ingly small and still yield significant data. Con- treatments are compared. In some studies, an
versely, if the population is large and heteroge- extra group, a "control group," is used. The
neous, the sample must be correspondingly purpose of having groups is to enable the inves-
large to yield significant data. Generally, 30 or tigator to treat the groups exactly alike except
more subjects are considered appropriate for for the treatment variable. This increases internal
most unbiased statistical analyses. validity. If a difference in the outcomes between
Another question is: "How were the subjects the groups occurs, the conclusion could be
selected?" Subject selection is based on criteria drawn that the treatment probably caused the
predetermined by the author. If not well planned, difference. (One can never be absolutely sure,
the process of subject selection may introduce however, that the differences were not caused
bias into the study. Bias is evident in the following by chance alone.)
example. A researcher wants to determine The method by which subjects were selected
whether or not total knee replacement surgery for each group is an important consideration.
will reduce knee pain and selects only patients Was it a random selection, was selection based
with severe rheumatoid arthritis to participate. on the population being studied, or was bias
The data from the study will not demonstrate the introduced into the selection? If, for instance, all
effectiveness of the total knee replacement sur- the subjects older than 60 years of age were
gery in general but only for the patients with placed into group A and those younger than 40
severe rheumatoid arthritis. Based on this study, years of age were placed into group B, and age
a statement of the general effectiveness of total was not the variable being studied, the data
knee replacement surgery would be erroneous could be biased. When at all possible, selection
and biased. The study may have good internal should be made at random and based on the
validity (control of variables); however, the ex- population being studied to reduce bias.
ternal validity (ability to generalize) is poor, since A final question to ask is: "How were the
the subject sample was narrow. Hence, the cri- subjects treated?" The general rule is that each
teria for subject selection must be carefully con- subject or group of subjects should be handled
sidered, determined ahead of time, and related exactly the same except for the variable being
to the population from which the sample is to be tested. Many variations to this rule exist. Each
drawn and the question being asked. variation demands additional control to offset the
chance of introducing another confounding or sions you drew while reviewing the methodology,
uncontrolled variable. particularly the representativeness of the sample
The concerned investigator will state exactly and the appropriateness of the statistical analy-
how each subject or group was treated. While sis. Many times authors use parametric statistics
reviewing the study, recall the precise research when the sample selection, size, or demographic
question asked, the characteristics measured, charcteristics would have dictated nonparame-
the variables that were controlled, and those that tric statistics.
were not controlled. Then ask yourself if the
investigator really was measuring what he or she
thought he or she was measuring. For instance,
if the internal validity (control) was poor, the In this section, the author has the opportunity
possibility that the investigator did not measure and is expected to offer an educated, justified
the characteristic necessary to answer the orig- opinion about the results of the study. Many
inal question is fairly strong. authors overstep their responsibilities and offer
opinions that were not justified by their study. To
separate the opinions that were justified from the
ones that were not, review the research question
The results section of the article must report and relate it to the supporting literature review.
the data in an unbiased manner and without Then, relate both of those to the results of the
opinion. The data that supports or rejects the present study.
original research question should be presented The results should state whether or not the
along with the results of the statistical analyses. research question was answered. The type of
It should be presented in a clear, concise, and answer will be dependent on the type of question
usable form. The reader should look for data that asked. A descriptive study will present a descrip-
the author did not report but which he or she tion of the subject studied. A correlational study
said were collected. That data may influence the should state whether or not the characteristics
answer to the original question. being studied were correlated and, if they were,
Before explorirlg the results further, the statis- how they were. Similarly, a comparative study
tical methods should be examined. A few simple should state how the items being studied did or
rules should be kept in mind to guide your deci- did not compare.
sions concerning the appropriateness of the sta- Regardless of what the study revealed, the
tistical analyses. results should be related to the question, meth-
If the subjects cannot be assumed to represent odology, and results. If any conclusions are
the larger population, the study will require the drawn that were indirectly but not specifically
application of a nonparametric testing procedure related to the question, methodology, and re-
(X2,rank correlation, etc.) sults, the strength of those conclusions is low.
If however the sample does represent the pop- Finally, did the author overlook any important
ulation, statistics related to the normal distribu- analyses or conclusions? The reader's knowl-
tion or parametric tests can be used ( t test, edge of the subject may allow viewing the study
analysis of variance, correlation). in a different manner that would generate differ-
If the groups had fewer than 30 subjects, ent results and conclusions. That's good! Com-
parametric tests must be applied carefully since pare the two-yours and the author's. If there is
the small number of subjects may inadequately a large discrepancy, the reader may wish to
represent the total population. study the problem further.
In a well-designed and well-controlled study, Remember, the key to a sound clinically ap-
an isolated, aberrant value may occur. This may plicable study is the control of variables and
demand a change in the statistical analysis to justification of all conclusions. If these two qual-
control for the abnormal or skewed data. If this ifications are met, the value of the study in gen-
was done, the reasons for the change should be eral increases. When unsupported conclusions
clearly stated. are made, they must be examined for bias and
As you review the results of the statistical may be purely conjectural. Some conjectures
analysis, refer back to the research question. are educated assumptions and deserve consid-
The author should state the significant findings eration. They should be tested by an appropriate
of the study. Again, they should be related to the study. They should not be accepted as true
research question. Keep in mind any conclu- based on the reported study. Finally, research
76 HART ET AL JOSPT Vol. 2, No. 2

does not prove anything. It merely generates when reviewing a specific patient problem, but
data that either supports or does not support the also keep in mind the limitations of the study.
null hypothesis. Never consider any results as proof of positive,
definitive answers for any clinical situation. Pa-
CONCLUSIONS tients are people and are always unique; they
will always break the rules.
The final evaluation of any medically oriented
study lies in its contribution to the advancement SUMMARY
of patient care. If the study is not clinically ap-
plicable, its value is greatly reduced. Using the method presented in this paper to
To get an impression of the overall evaluation review research articles will allow students and
of a study, the following questions should be clinicians to make better judgments about the
answered: volumes of research that are being published.
I ) Did the author answer the question that was This article has been directed toward research
asked? applied to clinical situations, review of which
2) Was the methodology appropriate and un- should improve patient care.
biased with strong internal validity?
3) Were the results unbiased? SELECTED BIBLIOGRAPHY
4 ) Was the review of literature related to the 1. Currier DP: Elements of Research and Physical Therapy. Balti-
results of the study and was it used to draw more: The Williams 8 Wilkins Co. 1979
2. Ferguson GA: Statistical Analyses in Psychology and Education.
sound, logical conclusions?
Ed 4. New York: McGraw-Hill Book Co. 1976
5) Are the results applicable to the clinical 3. Leedy PD: Practical Research, Planning and Design. New York:
situation? McMillan Publishing Co. Inc. 1974
If the questions can all be answered in the 4. Levin J: Elementary Statistics in Social Research, Ed 2. New
York: Harper 8 Row, Publishers, 1977
affirmative, the study is valid. However, no study 5. Michaels E: Introduction to Clinical Research. Philadelphia: Uni-
can be applied to all clinical problems. Reflect versity of Pennsylvania Press. 1977
6. Payton OD: Research: A Validation of Clinical Practice. Phila-
upon the study when confronted with a similar
delphia: FA Davis Co, 1979
patient pathology. Use the conclusion,s as a base 7. Winer BJ: Statistical Principles in Experimental Design. Ed 2.
of knowledge for a clinical frame of reference New York: McGraw-Hill Book Co. 1974