Can Meta-Analysis Resolve the ESP Controversy?
CHARLES AKERS:
Metaanatysis is a quantitative, analytical approach (0 the review of scientific
literature (Green and Hall 1984). The initial applications to psychology were in
major areas of controversy, such as employment test validity (Schmidt and
Hunter 1977), experimenter expectancy effects (Rosenthal and Rubin 1978), sex
differences in conformity (Cooper 1979), and effects of psychotherapy
(Smith and Glass 1977), Meta-analysis has become a popular technique for the
exploration of many areas where experimental outcomes are in conflict, includ
ing parapsychology (Honorton 1985; Hyman 1985).
Meta-analytic methods do have some potential for resolving parapsycto-
logical controversies. However, I will argue that there are serious limitations to
what such methods can accomplish with old data—data on which skeptics and
believers disagree vehemently. It is apparent that skeptics and believers do not
agree on the quality of parapsychological studies (and hence on how much
‘weight each study should receive in the analysis). If this is true, then meta-
analysis may be premature. It may be preferable to collect new data, under
conditions prespecified by both parties to the debate, With research standards
prespecified, the problem inherent to meta-analysis, of subjective bias in the
coding of research quality, could be avoided.
Before exploring the issue of quality coding, and possible sesolution of the
issue, a brief account of meta-analytic methods is in order. The initial step in
‘meta-analysis (after delineation of the problem area) is a definition of the data
base. This involves setting explicit eriteria for which studies should be included
in the literature review and which should be excluded. The reviewer then defines
the nature of the independent and dependent variables he wishes to consider.
Methodological or design quality features can and should be included. All such
study characteristics are then coded and assigned numerical values (if only for
Prepared withthe support ofthe Hodgson Fund of Harvard Universiy. 1 thank George Hansen,
Soh Palmer, Robert Rosenthal, and EH. Walker for thet eral comments on cari drafts
onthe purpose of nominal classification). Once the quantification has been com-
pleted, the data can be statistically analyzed, either by conventional techniques
or by methods specific to meta-analysis. The usual goal is to determine how (or
whether) study outcomes are correlated with or influenced by certain indepen-
dent variables (though conclusions about causality may not be warranted).
“Study outcome" can be defined in terms of some quantitative measure of effect
size, such as Cohen's (197) d index.
“The emphasis on quantification is one feature that distinguishes meta-analy-
sis from a traditional, qualitative review. Another distinguishing feature of meta-
analysis isthe attempt to make every step in the inferential process explicit. In @
qualitative review, the steps in the process are more likely to be implicit, re-
Quiring the reader to make inferences. An advantage of meta-analysis is that,
with all steps inthe inferential process made explicit, it should be easier than in
4 traditional review to identify sources of error or bias. If two meta-anaiysts
disagree, it should be easier to identify the reasons for their disagreement.
Because all variables must be quantified and all inferences made explicit,
meta-analysis involves difficult or even arbitrary judgments. In a review of the
various pitfalls inthe approach, M. J. Strube and D. P. Hartmana (1983) con-
,
ST:131-446,Charles Akers 625
——. 1984. The Integrative Research Review: A Systematic Approach. Beverly Hills,
Cali: Sage
Davis, J, W.,and Akers, C. 1974, Randomization and tests for randomness, Journal of
Parapsychology, 38:393-407.
Dixon, N. F. 1981. Preconscious Processing. Chichester, England: Wiley.
Feldman, S. E, 1980, Solomon E. Feldman comments on Edward W. Karnes et al, re
remote viewing. Zeteic Scholar, No. 7:131
Fiske, D. W. 1983. The meta-analytic revolution in outcome research. Journal of Con-
sulting and Clinical Psychology, $1:65-10,
Girden, E. 1978. Parapsychology. In Handbook of Perception, vol. 10, edited by E. C
Carterette and M. P. Friedman. New York: Academie Press
Glass, G. V., B. McGaw, and M. L. Smith. 1981. Mera-analysis in Social Research,
Beverly Hills, Calif: Sage
Glucksberg, S. 1982. Not seeing is believing [review of Norman F. Dixon's Preconscious
Processing). Contemporary Psychology, 27:856-858.
Green, B. F.,and J. A. Hall. 1984, Quantitative methods for literature reviews. Annual
Review of Psychology, 35:37-53.
Hansel, C. E. M. 1980. ESP and Parapsychology. A Critical Re-Evaluation, Buttalo:
Prometheus Books.
Hoebens, P. H. 1980. Piet Hein Hoebens comments on Edward V. Karnes et al. re
remote viewing. Zeteic Scholar, No. 7131-132
Honorton, C. 1977. Psi and internal attention states. In Handbook of Parapsychology,
edited by B. B. Woman, 435-472, New York: Van Nostrand Reinhold.
——. 1979, Methodological issues in free-response psi experiments. Journal of the
American SPR, 73:381-394.
——. 1985. Meta-analysis of psi ganzfeld research. Journal of Parapsychology.
4951-91
Hyman, R. 1981, Further comments on Schmidt's PK experiments: Alternative explana-
tions are abundant. Skeptical Inquirer, 5(3}:34-40.
1985, The ganzfeld/psi experiment: A critical appraisal. Journal of Paraps)~
chology, 49:3-49,
Johnson, M. 1975. Models of control and control of bias. European Journal of Para-
psychology. 1, (1):36-44.
Karnes, F. W. 1980. Edward W. Karnes replies to Solomon E. Feldman, Piet Hein
Hoebens, and Evan Harris Walker. Zevetic Scholar, No. 7137-138.
——. 1981, Edward W. Karnes replies to Evan Harris Walker's above comments.
Zetetic Scholar, No. 8128-130
Karnes, E. W., E. Susman, P. Klusman, and L. Turcotter. 1980. Failures to replicate
remote viewing using psychie subjects. Zevetic Scholar, No. 6:66-16.
Lord, C., L. Ross, and M. Lepper. 1979. Biased assimilation and attitude polarization:
‘The effect of prior theories on subsequently considered evidence. Journal of Per-
sonality and Social Psychology, 37:2098-2108,
Mahoney, M. J. 1977. Publication prejudices: An experimental study of confirmatory
bias in the peer review system. Cognitive Therapy and Research, :161-17S625 Can Meta-Analysis Resolve the ESP Controversy?
Marks, D. and R. Kammann. 1980. Comments by David Marks and Richard Kammann
{commentary on the paper by Professor Karnes etal]. Zeteric Scholar, No. 6:83-84,
McClenon, J. 1982. A survey of elite scientists. Their attitudes toward ESP and paré-
psychology. Journal of Parapsychology, 46:127-182
Merikle, P. M. 1982. Unconscious perception revisited, Perception and Psychophysics,
31:298201
Mintz, J. 1983, Inepsating research evidence: A commentary on meta-analysis, Jounal
of Consulting and Clinical Pxychology. 5171-15
Morris, R. L. 1982. Review of Parapsychology: Science or Magic? A Psychological
Perspective, by J. E. Alcock. Journal of the American SPR, 76:177-186.
Moss, S., and D. C. Butler. 1978. The sciemtifie credibility of ESP. Perceptual and
‘Motor Skills, 46:1063-1078.
Nicol, J. ¥. 1986. Some difficulties in the way of scientific recognition of extrasensory
perception. in Exirasensory Perception, edited by G. E. W. Wolstenholme and
E.C. P. Millar. New York: Citadel Press.
Palmer. J. 1977. Altitudes and personality traits in experimental ESP research. In
Handbook of Parapsychology, edited ¥y 8. B, Wolman, 175-201. New York: Van
Nostrand Reinhold
1981. Methodological objections to the case for psi: Are formal controt con-
divions necessary for the demonstration of psi? Paper presented at the convention
of the American Psychological Association, Los Angeles, Calif, August
1983, Sensory contamination of free-response ESP targets: The greasy fingers
hypothesis. Journal of the American SPR, 77:101-113.
(in pres). Sensory identification af contaminaied free-rsponse ESP targets:
Return of the greasy finger. Journal ofthe American SPR.
Pratt, J. G., J. B. Rhine, B. Mott-Smith, C. E, Stuart, and J. A. Greenwood. 1966,
Exura-Sensory Perception after Sixty Years. Boston: Bruce Humphries, (Originally
publishes by Holt in 1940.)
Price, G. R. 1955. Science and the supernatural. Science, 122:389-367,
1972, Apology to Rhine snd Soal, Letter to Science, 175:389.
Puthoff, H. E.. and R. Targ. 1976, A pereeptual channel for information transfer over
kilometer distances: Historical perspectives and recent research. Proceedings of the
IEEE, 329-384,
Rand Corporation, 1955. A Millon Random Digits. Glencoe, th: Free Press.
Randi, J. 880a, Flim-Flam? The Trush about Unicorns, Parapsychologs. and other
Delusions. New York: Lippincott & Crowell. [Revised edition published by Pro-
metheus Books, Buffalo, N.Y, 1982.)
19800. Comments by James Randi. [commentary: on the paper by Professor
Karnes etal. Zetetic Scholar, No. 684.
Riess, BF. 1937. A case of high scores in card-guessing at a distance. Journal of
Parapsychology, 1260-263.
1939, Further dats from a case of high scores in card-guessing, Journal of
Parapsychology. 3:79-84
Rosenthal. R. 1986. Mesa-Analytc Procedures for Social Research. Beverly Hills, Cali
Sage.Charles Akers 627
Rosenthal, R., and D. B. Rubin, 1978. Interpersonal expectancy effects: The first 345
‘studies. Behavioral and Brain Sciences, 3:377-386
Sargent, C. L. 1980a. Comments on “Effects of associations and feedback on psi in the
‘anafeld.” Letter to the editor. Joural of the American SPR, 74:265-267,
19806. Exploring Psi in the Gansfeld. Parapsychological Monographs, No. 17,
[New York: Parapsychology Foundation,
Schmidt, F. L., and J. E, Hunter. 1977. Development of a general solution to the
problem of validity generalization. Journal of Applied Psychology, 62:329-S40.
‘Schmidt, H, 1980, A program for channeling psi data into the laboracory and onto the
critic's desk. In Research in Parapsychology 1979, edited by W. G. Roll. Metuchen,
Jz Searecrow Press
‘Smith, M. L.. and G. V. Glass. 1977. Meta-analysis of psychotherapy outcome studies.
American Psychologist, 32:752-700.
Soal, S. G., and F, Bateman, 1954. Modern Experiments in Telepathy. London: Faber
and Faber
‘Strube, M.J., and D. P. Hartmann, 1983, Meta-analysis: Techniques, applications, and
functions. Journal of Consulting and Clinical Psychology, $1:14-27
Tart, C.T. 1980. Comments by Charles T. Tart [commentary on the paper by Professor
Kames et all. Zeretic Schalar, No. 685-6.
‘Terry, 4, and C. Honorton. 1976. Psi information retrieval in the ganzfeld: Two con-
firmatory studies. Journal of the American SPR, 10:207-217
Walker, E, H. 1980, Evan Harris Walker comments on Edward W. Karnes etal. re
remote viewing. Zetevie Scholar, No, 7:132-137,
—. 1981. Evan Harris Walker replies. Zereule Scholar, 8:124-127,
Wilson. R. 1966. Deviations from randomness in ESP experiments. International
Journal of Parapsychology, 8387-295