Evaluating Research Articles

Approach With Skepticism

Rebecca L. Fiedler
January 16, 2002
 Preview of Article

• Does the abstract intrigue me to read this article?

• Will it be useful to me?
• Who are the researchers?
• Are they credible and unbiased?
• What sources did the researchers use?
 Major Components

• Title • Methodology
• Abstract • Summary of Results
• Introduction • Discussion & Conclusion
• Literature Review • References
• Research Question
• Hypothesis
Qualitative Research

• The research is conducted in

the natural setting
• Is descriptive
• Researcher observes a specific
situation
• Sometimes variables are
manipulated, but not always
• Goal is to gain insight or
identify key variables or ask
new questions for further
research
Quantitative Research

• Attempts to quantify key variables and relate

them
• Variables are manipulated in some way
(Called a treatment)
• Results are measured and analyzed
statistically
• Goal is to identify cause-and-effect
relationships
Quantitative Research

Hypothesis testing
 Title

• Is it specific? • Are the results accurately

• Is the nature of the research indicated?
clear? • Are the main variables
• Does it reflect the content of clear?
the article? • Is the population clear?
 Abstract

• Was the purpose clear?

• Was the methodology indicated?
• Were the populations and
samples clearly identified?
• Did the the abstract highlight the
findings?
 Introduction

• Is the purpose made clear?

• Did the authors explain the
significance of the study?
(Do you agree?)
• Clearly written and
well-organized?
 Literature Review

• Does the review establish • Are the reviewed articles

significance of the study?
• Does the review address the
problem area?
• Is the review easy to read and
understand?
• Is it balanced?
relevant and current?
• Is the review comprehensive?
• How credible are the cited
sources? Most should be
primary sources.
Types of Sources

• General references
• Primary sources
• Secondary sources
 Research Question/Hypothesis

• Is it clearly stated?
• Is there a hypothesis?
• How well is it related to the other
components?
• Is it ethical to ask?
Methodology

• Experimental
• Survey
• Correlation
• Factorial study
• Causal-Comparative
• Regression analysis
• Ethnographic study
 Methodology - Research Design

• Do the authors justify the design decisions?

• Did they discuss the limitations?
• Are variables identified?
Dependent & independent.
• Are any external variables identified?
• Are those external variables controlled?
• Was the design appropriate?
 Methodology - Samples

• Is the population identified?

• Are the samples representative of that population?
• How were the samples selected and will the techniques
compromise the results?
• Can the information be generalized to the proposed
population?
 Methodology - Instruments

• What instruments were used to

collect the data?
• Was the choice of instrument
justified?
• Is evidence of reliability and
validity provided?
• Were any limitations
addressed?
 Common Threats to Validity

• Subject Characteristics
• Mortality
• Location
• Instrument
• Maturation
• Regression
• Hawthorne Effect
• History
• Implementation
 Methodology - Procedures

• Is the description of procedures robust?

• Are there any threats to validity due to the procedures
discussed?
• Are you able to identify additional threats to validity?
• Are there any ethical issues in the procedures?
 Summary of Results

• Are the results reported without any interpretation first?

• Are the results directly tied to the question, hypothesis or
problem?
• Did the author provide enough detail for you to independently
check the results?
• Is there enough description for you to interpret the results in
context?
 Discussion & Conclusion

• What are are the findings? Are they clearly stated?

• Are the findings related to the results of the study and the
literature review?
• Any weaknesses or limitations?
• Did the authors make any statements about generalizability?
• Recommendations for future study?
 References

• Are most references primary

sources?
• How many citations are
offered?
• Are the cited references
recent?
• Based on the given info, can
you find them for your own
review?
Qualitative

Very descriptive
 Similarities to Quantitative

• Title
• Abstract
• Introduction
• Review of Literature
 Research Problem

• Did the focus of the research shift?

(That’s not bad)
• Was the shift justified?
• Did the researchers avoid a hypothesis at the start?
• What (if any) hypotheses were formed based on the data?
 Samples

• Often purposive. The purpose should be identified.

• Accessibility is sometimes an important consideration. It
should be named if it is/was a factor.
• Is there a detailed description of the sample?
 Setting

• Is there a full rich description

of the setting?
 Researchers

• Are there biases that may

interfere with the study?
• Was there interaction between
the researcher and the
participants? Or was the
researcher only an observer?
• How were observers trained?
 Data Collection

• Did the researchers use more

than one way to observe the
same phenomenon?
• Was there evidence of validity
by triangulation?
• Was there any quantitative
data? Frequency counts are
common.
 Procedures

• Should be the strongest

section.
• Are procedures fully described?
• Are they appropriate?
• Are there any ethical concerns?
• Are there any threats to
validity?
 Data Analysis

• Is analysis in descriptive form?

• If so, is the description
supported by the evidence?
• Is there any quantitative data?
• Is data provided for reader to
review?
Results

• What were the reported

results?
• Did the researcher form a
hypothesis?
Discussion and Conclusions

• What conclusions did the researchers

reach?
• What implications can be drawn from the
research?
• Are there suggestions for further
research?
• What limitations were mentioned?
Now what?
 You might want to….

• Use one of their suggestions for further research for your own
topic
• Replicate a research study
• Cite the article in your own research
• Use the research design in your own work
• Pick up other ideas for your own research
• Read the cited articles
 My Bibliography

–Girden, E. (2001). Evaluating Research Articles: From Start to Finish

(2nd edition ed.). Thousand Oaks, London, New Delhi: Sage
Publications.
–Goubil-Gambrell, P. (1992). A Practitioner's Guide to Research
Methods. Technical Communication: Journal of the Society for
Technical Communication, 39(4), pp. 582-591.
–GraphicsLand. (2002). Squares PowerPoint template [template file].
GraphicsLand. Retrieved 1/12/02, 2002, from the World Wide Web:
www.graphicsland.com
–Lunsford MSE CO, T. R., & Lunsford MS MAPT, B. R. (1996). How to
Critically Read a Journal Research Article. Journal of Prosthetics and
Orthotics, 8(1), pp. 24-31.
 My Bibliography (continued)

–Spyridakis, J. (1992). Conducting Research in Technical

Communication: The Application of True Experimental Designs.
Technical Communication: Journal of the Society for Technical
Communication, 39(4), pp. 607-624.
–Sullivan, P., & Spilka, R. (1992). Qualitative Research in Technical
Communication: Issues of Value, Identity, and Use. Technical
Communication: Journal of the Society for Technical Communication,
39(4), pp. 592-606.
–Yaw, M. (2001). Notes from Fundamentals of Graduate Research in
Education.
 Class Exercise

• Divide into two groups

• Prepare 10-minute presentation (45 minutes allowed)
–Presentation should critique one of the following articles
• Illustrations in User Manuals
• Learning How to Use a Cellular Phone
• Group presentations
 Bibliography Management

• File cabinet
• Sticky notes
• Index card file
• Database
• Bibliographic software
Free downloads on web
–EndNote
–Reference Manager
–ProCite