Professional Documents
Culture Documents
INVOLVED IN GRADUATESTUDY:
Donald E. Powers
Mary K. Enright
Report
Report
December 1986
Analytical
Donald E. Powers
Mary K. Enright
Report
December 1986
Copyright,
0c
1986 by Educational
Testing
Service.
All
rights
reserved.
Acknowledgments
The authors wish to thank the membersof the GREResearch Committee
for helpful suggestions; Laurie Barnett fr>r programmingthe analyses;
Neal Kingston, William Ward, and Cheryl Wild for helpful reviews of the
report.
Abstract
This study was intended to provide information on the role of
analytical
abilities
in graduate study. Graduate faculty in six
fields of study (chemistry, computer science, education, English,
engineering, and psychology) were asked to judge:
(a)
(b)
(c)
of
reasoning errors
had
-2-
-3(b)
(c)
-4-
The Sample
Six academic fields (English, education, psychology, chemistry,
computer science, and engineering) were included in the final survey.
These fields were thought to represent the variety of fields of
graduate study and the variation in the kinds of reasoning abilities
involved in graduate education. Using the data tapes of the Higher
Education General Information Survey (HEGIS), nonoverlapping samples
of 64 graduate institutions with doctoral programswere drawn for each
of the six graduate fields.
A randomsampling procedure was used such
that eight institutions from each of the eight HEGISgeographic
regions were selected for each field.
This sampling was greatly
The admission requirements
facilitated by the work of Oltman (1982).
of these institutions were determined from the Directory of Graduate
Programs (GRE/CGS,1983), and only those that either required or
recommended
GREGeneral Test scores were included in the sample. In
this manner, 40 institutions were selected for the final sample for
each field.
In addition, one institution with a relatively large
proportion of Black students and one with a relatively large
percentage of Hispanic students were included in the samples for each
field, thus raising the total numberof institutions to 42 per field.
Letters were then sent to departmental chairpersons, who were asked to
nominate two faculty memberswho would be willing to complete the
questionnaire. Respondentswere paid $25 for their participation.
Data Analysis
Means and standard deviations were calculated for each question
by academic field of study, and analyses of variance were run for each
question to assess differences amongthe six fields.
The various
ratings were correlated within questionnaire sections. For example,
within the section on reasoning-skills, the ratings of frequency-and
importance were correlated; within the section on reasoning errors,
the ratings of frequency and seriousness were correlated. Finally, within each section (and for each kind of rating), the
data were factor analyzed to effect somereduction in the large number
of questions. A principal axis factoring, with squared multiple
correlations as the initial estimates of communalities, was used to
determine the numberof factors to be retained for each section,
according to both the magnitude of the eigenvalues and the breaks in
their size.
(Our inclination was to err on the side of retaining too
many factors at this exploratory stage.) Various numbersof factors
were then rotated according to the varimax criterion.
Although other
oblique rotations could have been used also, it was felt that-the
detection of uncorrelated factors would best serve the objectives of
further test development.
-5Results
The Sample
A total of 165 chairpersons (65% of those contacted) nominated a
total of 297 faculty members, of whom255 (86%) returned usable
The response rates across fields were generally
questionnaires.
comparable.
Full professors constituted a slight majority of the responding
sample (51%) ; associate professors made up the next largest proportion
(34%). About 13% were assistant professors, and the remaining small
proportion were deans, associate deans, or lecturers.
Item-level
Results
-6particular
choice of questions, from differences
disciplines,
or from some other factor(s) .
in standards amonq
fallacies
and logical
contradictions
in arguments, deducing new information from a set of
relationships, and recognizing structural
similarities
between one
type of problem or theory and another were the next most highly rated
skills.
These were followed closely by taking well-known principles
and ideas from one area and applying them to a different
specialty,
monitoring ones own progress in solving problems, and deriving
from the study of single cases structural
features or functional
principles
that can be applied to other cases.
Table 6 lists the reasoning errors and critical
incidents that
were judged overall to be the most serious or to have the most effect
on the estimation of students abilities.
Three errors/incidents
were
judged to be most serious or critical:
accepting
the central
assumptions in an argument without questioning them, being unable to
integrate and synthesize ideas from various sources, and being
unable to generate hypotheses independently.
It should be noted that there are many other decision rules,
based on average ratings and differences among disciplines,
that could
have been used here to form a commoncore of skills or errors/
incidents.
Tables l-3 could be consulted to apply alternative
rules.
Factor Analytic
factor
Results
-7-
-8-
-9-
Critical
thinking skills involved in developing or otherwise
dealing with conclusions (Scale 2) were viewed as very important
(means greater than 4.0) in all disciplines
except computer science.
Abilities
involved
were rated as extremely
engineering (m = 4.00),
especially English (m =
-lO-
2.
4.
6.
Limitations
Any study of this nature is necessarily limited in several
respects.
First of all, the survey approach used here is but one of
several that can be used to inform decisions about extending the
measurement of analytical
abilities.
Tuckers (1985) results provide
useful information from different
perspectives--those
of cognitive
psychologists and philosophers.
Other approaches that might also be
informative include the methods of cognitive psychology, which could
be used not only to supplement but also to extend the survey results
These methods would seem especially appropriate
reported here.
because they relate more directly
to actual skills and abilities
than
to perceptions.
Second, the diversity
that characterizes graduate education
Some clues have been
renders the results of this study incomplete.
gained as to similarities
and differences among a limited sample of
However, the substantial differences found among
graduate fields.
fields are a source of concern, since we cannot be certain whether or
not some other sample of fields might exhibit even greater variation.
Finally,
as several survey respondents pointed out, many of the
reasoning skills about which we asked are expected to, and do, improve
In some sense these skills may
as the result of graduate study.
represent competencies that differ from, say, the verbal skills
measured by the GENE
General Test in the respect that these analytical
skills may develop much more rapidly.
A question of interest,
then,
is how to accommodate the measurement of these skills in the context
of graduate admissions testing, which currently focuses on the
predictive
effectiveness
of abilities
that are presumed to develop
slowly over a significant
period of time.
-14-
Future Directions
The study suggested several possible future directions.
Because
of the substantial variation among fields,
one possibility
would
involve extending the survey to include additional
fields of graduate
study.
Some refinements could now be made on the basis of past
experience.
For example, ratings of the frequency with which skills
are used, as well as the frequencies of errors and critical
incidents,
could probably be omitted without much loss of information.
OII the
other hand, it would seem desirable to add categories allowing ratings
of the differential
importance of various reasoning skills at
different
stages of graduate education, ranging from entry level to
dissertation
writing.
Finally,
based on the reasoning skills identified
as most
important, criterion
tasks might be developed against which the
validity
of the current GRE analytical
measure could be gauged. This
strategy would make especially good sense for those important skills
that may not be measurable in an operational test like the GRE General
Test, but which might correlate highly with the abilities
now measured
One specific possibility
would be the development of
by the test.
rating forms, which could be used by faculty to rate the analytical
abilities
of their students.
These ratings could then be used as a
criterion
against which GRE analytical
scores could be judged.
-15-
References
Altman, R. A., Carlson, A. B., & Donlon, T. F. (1975). Developing a
plan for the
renewal
of the GREnorming test: A report on
I
,
investigations znto the possible shortening of the GRE-Vand
GRE-Qand the creation of a modular aptitude test (GREReport
GREBNo. 74-3). Princeton, NJ: Educational Testing Service.
Arons, A. B. (1979). Somethoughts on reasoning capacities implicitly
expected of college students. In J. Lochheadand J. Clement
Philadelphia, PA: The
(Eds.), Cognitive process instruction.
Franklin Institute Press.
Campbell, S. K. (1974). Flaws and fallacies in statistical
EnglewoodCliffs, NJ: Prentice-Hall, Inc.
thinking.
-16-
Kahane, H. (1984).
Logic and contemporary rhetoric: The use of reason
Belmont, CA: WadsworthPublishing
in everyday lite (4th ed.).
Company,Inc.
Kaiser, H. F. (1958). The varimax criterion for analytic rotation in
factor analysis. Psychometrika, -23, 187-200.
Miller, R., & wild, C. (Eds.). (1979). Restructuring the Graduate
Record Examinations Aptitude Test (GREBoard Technical Report).
Princeton, NJ: Educational Testing Service.
Nosich, G. M. (1982). Reasonsand arguments. Belmont, CA: Wadsworth
Publishing Company,Inc.
01tman, P. K. (1982). Content representativeness of the Graduate
Record Examinations advancedtests in chemistry, computer
science, and education (GREBoard Professional Report GREBNo.
81-12~). Princeton, NJ: Educational Testing Service.
Powers, D. E., & Swinton, S. S. (1981). Extending the measurementof
graduate admission abilities beyond the verbal and quantitative
domains. Applied Psychological Measurement,5, 141-158.
Powers, D. E., & Swinton, S. S. (1984). Effects of self-study for
coachable item types. Journal of Educational Psychology, 76,
266-278.
thinking.
thinking.
New York:
Table
Mean
Ratings
of
Importance
of
1
Reasoning
Skills
by Disciplines
Computer
Tota
Variable
(N=255)
Reasoning
principles
from
Reasoning
when facts
Reasoning
when all
Reasoning
when
rules
Applying
Analyzing
formal
to
progress
Deriving
the
Understand
i ng and
a given
present
specialty
position
previous
data
that
probabilistic
eva
information
what
hwotheses
,
I
known
into
can
research
a framework
problems
be applied
SC ience
Education
Engineering
(N=37)
(N=43)
(N=42)
(N=43)
-.21
-.04
-.09
-.23
Engi ish
(N=44)
Psychology
(N=46)
of
4.03
-.I1
.07
-.06
.44
-.59
-.15
-. 10
.29
-.06
-.59
-.03
.55
1.03
-.21
.lO
-.31
3.75
-.05
-.08
-.13
.23
-.ll
3.52
-.23
-.62
.47
.05
4.33
.13
-.66
3.37
.38
-.40
3.71
0.20
.I5
3.68
0.11
.16
3.24
-.40
-.16
.29
events
-.06
-.63
.31
.oo
-.56
.09
from
3.86
.03
-.16
.02
4.19
.32
-.75
.14
thesis
will
support
.22
-.46
4.05
relationships
-.Ol
2.76
evidence
or
nature
4.24
.31
.23
.35
3.80**
-.34
.02
-.09
.I2
-.65
10.81***
.13
1.60
.34
5.92***
-.15
9.62
.14
1.32
.47
.23
7.74***
.21
.25
7.63***
.04
.23
3.19
-. 12
1.71
-.18
0.79
-.24
to
cases
Recognizing
Knowing
are
in solving
principles
other
not
a different
isolated
Monitoring
is
3.63
logic
evaluating
Incorporating
4.02
known explicitly
information
refuting
and
or cases
inconsistencies
of
or
facts
are
principles
Supporting
Deducing
Chem i stry
Skills--General
Deriving
Knowing
.09
.44
-.38
.16
.ll
-.40
.70
-.07
.70
9.49
.14
10.51***
.Ol
0.60
.22
10.35
a
-.26
.31
Table
1 (Continued)
Discipline
Computer
Tota
Variable
(N=255)
Specific
Skills--Problem
Identifying
central
Recognizing
sides
Generating
issues
of
down complex
Identifying
variables
Identifying
approaches
Recognizing
similarities
Developing
Setting
Recognizing
Specific
an
the
or
hypotheses
issue
hypotheses
Ed ucat
(N=43)
4.34
-.13
-.34
.16
3.58
-.15
-.75
.39
.16
3.87
.05
-. 50
problems
4.06
.05
.57
in a problem
3.88
.36
.02
to
solving
between
operational
up formal
SC I ence
(N=37)
i on
(N=42)
Engineering
(N=43)
Engl ish
(N=44)
Psychology
(N=46)
Definition
alternative
Breaking
Chem 1stry
problems
problems
definitions
models
historical
context
Skills--Constructing
Theses
-.05
-.13
.Ol
.09
.22
3.20
-,38
.62
.26
10.59
-.24
.13
.39
4.80***
.54
-.85
-. 17
14.34***
.16
.51
-.59
-.47
7.65**
.43
.07
-.20
.oo
3.03
-.06
.13
1.16
.46
5.49***
3.98
-.25
3.83
-.25
.03
-.02
.17
3.66
-.66
.36
-.07
-. 11
.02
3.36
-. 12
.59
-.22
.73
-. 93
2.97
-.70
-.74
.41
-.55
1.36
.22
26.65***
3.92
-.08
-.60
.13
-.22
.74
.04
11.22**t
3.57
-.17
-.04
13.49
or
Arguments
Supporting
Making
assertions
explicit
reason
with
components
details
In a chain
i ng
Distinguishing
information
relevant
and
of
.06
-. 10
.Ol
.29
.09
-. 10
.30
-.09
1.10
irrelevant
4.20
.05
-.45
.11
3.78**
L
Lo
I
Table
1 (Continued)
Discipline
Computer
Tota
Variable
(N=255)
Perceiving
relationships
Drawing
Using
among observations
distinctions
analogies
Elaboratlng
Drawing
between
similar
ideas
appropriately
an argument
sound
Producing
inferences
a consistent
argument
4.10
Chemistry
Science
Education
Engineering
(N=37)
(N=43)
(N=42)
(N=43)
.15
-.70
.03
3.67
-.48
-.30
.19
3.36
-.38
-.26
.12
3.80
-.37
-. 57
.30
4.25
.32
-.74
.25
4.12
.15
-.54
.lO
.04
-.13
.18
-.43
l 03
-.05
Engl ish
(N=44)
Psychology
(N=46)
.15
.33
7.74***
.56
.16
6,67***
.53
-.18
3.93**
1.02
.04
18.59
-.04
.18
8.84
.35
-.Ol
4.67
I
Synthesizing
different
Translating
Specific
Recogn
graphs
positions
or
symbolic
Skills--Analyzing
argument
assumptions
Testing
the
Finding
alternative
Recognizing
Detecting
statements
validity
supporting
fallacies
Distinguishlng
major
-.34
3.55
.40
4.04
.12
-.26
.45
.33
.02
-.lO
.50
.22
.16
2. go*
-1.44
.07
18.17
.oo
7.13***
Arguments
I zi ng the central
Identifying
3.10
of
an argument
explanations
points
and contradictions
points
in an argument
3.98
-.17
3.57
-.09
-.57
-.02
-. 11
.57
-.12
-.03
.06
.47
-.23
.02
-.09
.13
1.19
-.27
.55
7.89***
.27
-.20
3.14***
3.82
.16
-.56
.28
-. 16
3.64
.04
-. 52
.Ol
-.20
.61
.06
6.84
3.92
.08
-.36
.06
-.22
.22
.21
2.18
-.52
-. 14
.63
.11
5.84
3.28
-.12
.02
1 (Continued)
Table
Discipline
Computer
Tota
Variable
Being
(N=255)
sensitive
Recognizing
Judging
to
shifts
whether
Determining
a thesis
of
Drawing
(N=43)
Education
Engineering
(N=42)
(N=43)
Engl ish
(N=44)
Psychology
(N=46)
-. 96
.71
-.42
.05
.95
18.86***
an argument
2.90
-.90
-.69
.52
-.55
1.48
.14
33.50***
supported
3.86
-.08
-.81
.24
-. 32
.73
.25
12.76***
-. 79
.27
.05
.11
.27
8.65***
.38
3.31**
are
supported
among factors
Skills--
(N=37)
-.34
has been
conclusions
Sc i ence
3.31
evidence
in meaning
whether
Distinguishing
Specific
different
Chem i stry
4.25
.lO
3.45
-. 20
-.35
.lO
.31
-.24
Conclusions
Generating
vat id explanations
4.16
.18
-.76
.27
.07
.lO
.14
8.26
Supporting
conclusions
4.24
.15
-.77
.31
.02
.22
.07
8.92***
-.72
.26
-. 28
.59
.23
14.82***
-.15
.40
.29
8.90
-.61
.38
10.08
-. 14
.41
6.33***
-.09
.05
3.25**
.07
.18
6.38
Standard
deviations
Recognizing
implications
of
Qualifying
conclusions
Generating
new questions
Considering
Comparing
Revising
alternative
views
Note.
Entries
range
< .Ol
p -
interpretation
4.07
experiments
conclusions
conclusions
totals
*yp
or
an
< .05
< .OOl
for
new observations
under
from
.78
individual
to
1.31,
discipl
ines
and 33 of
are
56 are
-.07
3.99
.06
-.73
.13
3.88
.53
-.58
.07
.21
3.85
.24
-. 59
.25
-. 16
3.91
.28
-.47
.04
.18
3.84
.30
-. 77
.14
.07
deviations
between
from
.9
and
the
1.1.
grand
means given
under
ltTotaI.ll
for
Table
Mean
Ratings
of
Seriousness
of
2
Reasoning
Errors
by Disciplines
Disci pl ine
Computer
Tota
Variable
(N=i55)
Chemistry
SC ience
Educat i on
(N=42)
Eng i neer
i ng
Engl ish
Psychology
(N=37)
(N=43)
2.92
-.59
-.55
.64
.41
-.82
.91
16.90***
2.64
-.18
-.62
.36
.14
-.03
1.34
24.59***
3.05
-.46
-.66
.58
-.53
.41
.66
8.27***
3.34
-.42
-.45
.34
-.24
.45
.32
6.31**
3.60
-.47
-.44
.49
-.22
.83
9.51***
3.02
-.66
-.62
.18
-.38
.54
15.13
3.20
-.20
-.07
.lO
-.21
.65
3.96**
3.96
-.23
-.22
.09
-.12
.24
.24
2.14
3.74
-.23
-.63
.35
-.09
.37
.23
5.55
3.48
-.40
-.53
.27
-.18
.61
.23
7.49***
3.86
-.24
-.42
.31
-.14
.30
.20
3.97**
3.49
-.52
-.61
.39
-.28
.62
.40
11.22***
(N=43)
(N=44)
(N=46)
Error
Faillng
to
recognize
sample
Failing
and
to
Offering
account
for
irrelevant
Failing
to
to
accept
Accepting
with
rate
conclusion
causation
term
an unproven
assumptions
Fai I ing to
base
a concealed
coincidence
an ambiguous
Arguing
between
statements
recognize
Confusing
Using
differences
population
evaluate
with
the
conclusion
questioning
credibil
ity
of
source
Offering
irrelevant
Making
general
ev i d ence
inappropriately
evidence
izations
from
interpreting
-.20
.93
-.26
insufficient
text
Table
2 (Continued)
Discipline
Computer
Tota
Variable
Allowing
Basing
Fail
ing
anecdotal
conclusions
to
information
on partial
recognize
data
simi larities
Note.
Entries
totals
range
*p
- < .Ol
**p < .05
< .OOl
under
from
.98
individual
to
1.48,
Chemistry
Sc fence
(N=37)
(N=43)
Education
Engineering
(N=42)
(N=43)
3.23
-. 64
-. 33
.72
.Ol
3.57
-.08
-.36
.09
.oo
3.06
-.33
-.42
.28
.03
Engl ish
(N=44)
Psychology
_F_
(N=46)
.74
10.32***
.16
.19
1.50
.17
.28
3.46
Standard
deviations
-.51
in
analogies
***p-
(N=255)
disciplines
and 8 of
are
15 are
deviations
between
from
1.0
and
the
1.2.
grand
means given
under
Total.~
for
Table
Mean
Ratings
of
Effects
of
Critical
Incidents
by Dlscipl
lnes
Discipline
Computer
Chem i stry
Science
(N=37)
(N=43)
3.23
-.23
-.46
.26
-.33
criticism
2.79
-.22
-.40
.35
-.23
paper
3.67
Tota
Variable
(N=255)
Ed ucat i on
(N=42)
Engineering
(N=43)
Incidents
Making
irrel
Offering
evant
remarks
nonconstructive
Submitting
a nonresponsive
Accepting
an argument
Accepting
on emot iona
with
uncritically
hard
I appeal
data
conclusions
unable
Failing
to
Argulng
Unable
Falling
Making
integrate
recognize
support
Using
to
for
to
ideas
evidence
the
statements
draw
implausible
analogy
conclusions
breakdown
about
assumptions
-.23
3.34
-.23
-.77
.49
-.46
3.24
-.24
-.45
.28
-. 14
3.46
-.ll
-.42
.15
-.23
3.96
-.15
-.19
3.39
-.12
-.39
3.33
obvious
perceive
-.05
-.05
-.lO
can
conclusions
inconsistent
to
that
-.32
of
authorities
Being
.17
a pattern
.12
.11
-.18
-.38
-.17
-.oo
-.22
-. 14
2.76
-.09
-.16
2.98
-.03
-.35
3.41
.18
-.18
3.44
.23
-.42
.05
-.29
.Ol
.12
.14
.09
Engl
(N=44)
Table
3 (Continued)
Discipline
computer
Tota
Variable
(N=255)
Applying
rules
Relying
solely
Preferring
Unable
without
complex
to
justification
3.68
to
simple
explanations
see a pattern
to
Writing
a one-sided
qual ify
Searching
for
Ignoring
detai
Unwilling
to
conclusions
essay
Is
respond
in an unknown
to
generate
Unable
to
suggest
a I ternat
Unable
to
suggest
tests
Note.
Entries
totals
range
p < .Ol
< .05
< ,001
solution
hypotheses
under
from
situation
.93
of
i ves
individual
to
1.27,
ines
15 of
are
25 are
(N=43)
.11
.70
-.12
Ed ucat ion
Engineering
(N=42)
-.33
-.ll
.I3
-.25
-.06
-.23
.13
-.32
.02
-.05
3.44
.07
-.56
.39
-.17
-.60
.47
-.28
3.28
.02
3.74
.20
3.28
.56
3.94
.14
deviations
between
.16
-.37
.06
-.ll
.31
from
1 .O and
the
1.2.
(N=44)
-1.06
.72
3.76
-.30
Engl ish
(N=43)
-.ll
3.52
discipl
and
Science
(N=37)
.07
3.43
hypotheses
Chemistry
3.23
3.01
a complicated
Unable
**,,
3.38
on narrative
Fai I ing
**p
-.ia
.50
-.12
.23
-.Ol
.47
-.19
-.04
-.28
.26
a.35
.20
-F
(N=46)
-.14
16.24
-.16
2.99
.ia
.I6
.23
Psychology
.a1
-.oo
1.61
.2a
4.99
25
7.46
-.04
1.11
.082
-.19
.23
3.76
-.27
.oa
-.06
-.12
.23
1.54
-.15
.30
-.04
-.oo
.oo
.90
-.28
.I7
-.19
-.64
.63
8.15
grand
means given
under
Total
.I
Standard
deviations
for
Table
Reasoning Skills,
Most Important
Discipline
Chemistry
Errors,
and Incidents
Rated as
or Critical
by Disciplines
Skills
Errors/Incidents
from
Critically
analyzing
and
evaluating
previous research
reports
in a field
(4.46)
Applying a formula,
algorithm,
other rule without sufficient
justification
(4.08)
Being unable to generate
hypotheses independently
or
or
(4.08)
Generating
new questions or
experiments
to extend or support
the interpretation
of data (4.42)
Computer
Science
Education
Supporting
conclusions
with
data or information
(4.55)
Determining
whether
drawn are logically
and adequately
data (4.52)
in
are
sufficient
the conclusions
consistent
with,
supported
by,
Making generalizations
basis of insufficient
(4.17)
Confusing
correlation
on the
evidence
coincidence
and/or
with causation
(4.10)
the
Failing
to evaluate
the credibility
or reliability
of a source or
text (4.10)
Table
Discipline
4 (Continued)
Errors/Incidents
Skills
Clearly
identifying
central
issues
and problems to be investigated
or
hypotheses to be tested (4.50)
Drawing sound inferences
observations
(4.50)
Engineering
from
Applying a formula,
algorithm,
other rule without sufficient
justification
(4.09)
or
English
involved
Elaborating
an argument and developing
its implications
(4.82)
Understanding,
analyzing,
and
evaluating
arguments (4.75)
Supporting general
details
(4.66)
assertions
Recognizing
the central
thesis in a work (4.61)
with
argument
or
Table
Discipline
Psychology
4 (Continued)
Skills
Errors/Incidents
Critically
analyzing
and evaluating
previous research or reports
in a
field
(4.58)
Confusing
correlation
coincidence
and/or
with causation
(4.43)
Clearly
identifying
central
issues
and problems to be investigated
or
hypotheses to be tested (4.57)
Determining
whether the conclusions
drawn are logically
consistent
with,
and adequately
supported by,
the data (4.52)
Note.
Numbers in parentheses
each discipline.
are
the average
ratings
for
each skill,
error,
or incident
for
-29-
Table 5
Reasoning Skills Rated Consistently
Moderately Important
As At Least
Mean
Rating
Skill
Reasoning or solving problems in situations in
which all the needed information is not known
4.24
Detecting fallacies
in arguments
3.92
and logical
contradictions
from a set of
3.86
Recognizing structural
similarities
between one
type of problem or theory and another
3.83
3.75
Monitoring
3.71
ones
3.68
Making explicit
all relevant
of logical reasoning
3.57
components in a chain
of an argument by searching
3.57
an average
Note.
c Moderately important is defined as having There
were
rating over all disciplines
of 3.5 or greater.
no significant
differences among disciplines
with respect
to the average importance of these skills.
-3o-
Table 6
Errors or Incidents Rated Consistently as at Least
Moderately Serious or Critical
Mean
Rating
Error/Incident
Accepting the central assumptions in an
argument without questioning them
3.96
3.96
3.94
3.76
Ignoring details
desired result
in results
that contradict
or to
an expected or
3.74
failed
to address the
3.67
of only part of
3.57
-31Table 7
Summaryof Variables Defining Factors Underlying
Ratings of Importance of Reasoning Skills
Variables
Factor
I
(73)
(72)
(70)
(67)
Understanding,
analyzing,
Generating valid
observations
context of a problem
and evaluating
(64)
explanations
in a work
(63)
to account for
(72)
data or
(6%
from observations
(69)
w3)
(67)
Considering alternative
III
arguments
known
(5%
conclusions
(69)
into
(59)
(57)
Identifying
(56)
all
the variables
involved in a problem
(53)
-32-
Table 7 (Continued)
Factor
IV
Variables
Recognizing structural
similarities
between one
type of problem or theory and another
(61)
Drawing distinctions
identical
ideas
(53)
Synthesizing
position
between similar
two different
positions
Finding alternative
Generating alternative
into a third
(44)
but not
explanations
hypotheses
cases structural
that can be
(41)
for observations
(53)
(46)
-33Table
Factor
I
Variables
Offering
irrelevant
Making generalizations
evidence
Failing to evaluate
of a source or text
II
(75)
the credibility
or reliability
(71)
(70)
(76)
(68)
between a sample
Offering irrelevant
statements about a persons
character or circumstances to oppose his/her
conclusion
63)
(61)
with
Note.
Only the variables that best character ize the factors are
given.
Loadings are given in parentheses for the factor on which
the variables
loading was most predominant.
Table 9
Summaryof Variables Defining Factors Underlying
Ratings of the Effects of Critical Incidents
Factor
I
II
III
(72)
(65)
(65)
(62)
(71)
(59)
(58)
(56)
(55)
(67)
(58)
(54)
Note. only the variables that best characterize the factors are
given. Loadings are given in parentheses for the factor on which
the variable's loading was most predominant.
-35Table 10
Means of Scales Based on Items that Best Defined
Factors by Disciplines
Discipline
Chemistry
3.34
4.29
Computer Science
2.97
Education
Scale
3
3.57
3.38
3.43
3.37
4.05
3.58
2.90
3.83
4.29
3.44
3.73
3.93
Engineering
3.25
4.11
4.00
3.61
3.44
English
4.53
4.10
2.70
3.87
3.46
Psychology
3.73
4.26
3.37
3.78
4.21
Total
3.61
4.07
3".52
3.66
3.56
Key.
Note.
Appendix A
Questionnaire
REASONING SKILLS
I.
The followlng
are checkllsts
In your graduate
of sune general
Please
program.
( 1) frequency-how
often
(2)
success-to
Importance
for
rate
do students
and specific
these skills
1n your graduate
what extent
reasonlng
wfth respect
does thfs
skills
that
may be Important
to success
to:
dffferentfate
ski I I?
between marglnal
and successful
students
in your progran?
For ftlmportance
skill
success,t8 please
for
1.
lhls
2.
There
Is little
3.
There
Is a moderately
4.
There
Is a very
5.
There
1s a crItIcally
refer
to these descrlptlons:
to my ffeld
Js not relevant
or no difference
important
Important
of teaching
between marglnal
difference
difference
Important
and successful
between margfnal
between marglnal
difference
students
and successful
and successful
betneen margfnal
with
students
and successful
respect
students
with
students
to this
wfth respect
respect
to this
to thls
wlth respect
ImDortance for
Frequency
skill
skill
skrll
to thfs
Success/d
sklll
f f f erence
rbt
Never/
Hardly
Derlvlng
general
dlsparate
facts
or abstract
or cases.
Reasoning or problan
which all
facts
principles
solvlng
fn sltuatfons
In sltuatlons
fnformatlon
of formal
logic
. .
. .
. .
. .
. .
. .
. .
. .
problems .
Jn
In
in
. . . . . .
them to a different
analyzing
or reports
lnccrporatlng
preexlstlng
Mftorlng
specialty
a given
. . . . . . . . . . . . . . . . . . .
CYltfcally
research
is not known
are present
posftlon.
. . . . . . . . . . . . . . .
Being able
. . . . . . . . . . .
. .
a problem solution
and evaluating
In a field.
Isolated
prevlous
. . . . . . . .
Instances
or data
fr wk.............
ones
Q-ltlcal
Important
Llttle
frun
. . . . . . . . . .
underlylng
Vet-Y
Frequent I y
Ever
Wet-ate
Relevant
own progress
Jn solving
Into a
-2-
Freauencv
Importance
for
Success/d
1f f erence
r&t
Navel-/
Hardlv
Derlvlng
features
or functional
prrnclples
cases.
. . .
that
WeI evant
VSY
Ever
Freauentlv
Wderate
LIttIe
O-Hlcal
Important
can be
. . . . . . . . . . .
. . . .
Generatjng
ReccgnlzIng
wents.
nature
of most
. . . . . . . . . . .
Understanding,
analyzing,
and evaluating
arguments.
frcm a set of
. . . . . . . .
WI I I suppoct or
refute
. . . . . . . . . . .
Other
a hypothesis
or thesis
(specIfyI
B. Speclflc
Clearly
Skills-+roblem
ldentlfylng
be lnvestlgated
Break!ng
slmplm
DefInftlon
central
or hypotheses to be tested.
alternatlve
hypotheses
. . . . . . . . .
Identlfylng
all
Into
. . . . . . . . . . . . . . . . . . .
the variables
involved
In a
Problem . . . . . . . . . . . . . . . . . . . . . .
ldentffylng
Problem . . . . . . . . . . . . . . . . . . . . . .
f?ecognIzlng structural
sfmflarltles
operatlonal
very preclse)
RecognIzlng
. . . . . . .
deflnStIons
. . . . . . . . . . . . . . . . . . . .
co&deratJon
Other
(cr
between one
and another
problems under
. . . . . . . . . . . . . . . .
the hlstorlcal
(specify)
context
. . .
of a problem
. .
Freauencv
Importance for
Success/d
I f ference
r+kYt
New-/
I-Wdly Ever
c.
Speclflc
Sklsts--&struct~ng
Met-ate
Relevant
VWY
Lfttle
Frequently
CrItIcal
Important
Theses
or Argunents
Supporting
bklng
of
. . . .
. . . . . . . . . . . . . . . . . .
appropriately
. .
. . . . . . . . . . . . . . . . . .
general
expllcft
logIca
assertlons
all
relevant
reasoning.
D!stfngufshlng
. . . . . . . . . . . . . . .
and Irrelevant
. . . . . . . . . . . . . . . . . . . .
Percelvlng
relatfonships
Drawing dlstlnctlons
ldentlcal
Ideas
Using analogles
Elaborating
among observations
betwean slmllar
. . . . . . . . . . .
frcm observations.
an argument that
Syntheslzlng
posItJon.
Its
. . . . . . . . . . . . . . . . . . .
but not
ImpI IcatIons.
two dlfferent
1s Internally
posStfons
Into
. . . .
conslstent.
a thlrd
. . . . . . . . . . . . . . . . . . . . .
Translating
graphs or symbolic
versa.
statements
Into
. . . . . . . . . . . . . . .
(specIfyI
D. Speclffc
Skills-Analyzing
Recognlzlng
the central
Arguments
argument or thesis
awork......................
Identlfylng
Testing
for
. . . .
ccmponents In a chain
between relevant
lnfmt!on
Other
with details.
Flndlng
the va1ldH-y
of an argunent
explanations
suppcrtlng
fal lacfes
In arguments.
and unstated
assumptions.
by seat-chlng
18s . . . . . . . . . . . . . . .
alternatlve
Recognlzlng
Detecting
both stated
counterexamp
In
po!nts
and logical
for observations
In an argument.
contradlctlcns
Importance
Freauencv
for
Success/d 1f f erence
Not
#3ver/
Hardly
Dlst1ngulshlng
argument.
. . . . . . . . . . . . . . . .
. . . . . . .
. . . . . . . .
to the strength
Recognlzlng
shifts
of djfferent
causal,
types
testlmony).
. .
course of an argument . . . . . . . . . . . . . . .
Judging whether a thesls
. . . . . . . . . . . . . . . . . . . . .
whether the conclusions
loglcally
consistent
supposed
by,
the
Dlstlngulshlng
sufflclent
wlth,
data.
factors.
drawn are
and adequately
.
among contt-lbutlng,
necessary
and
Other (specIfyI
E. Speclflc
Generatlng
Skills-Bawlng
valld
observations.
Supportlng
Conclusions
explanations
. . .
to account for
. . . . . . . . . . . . . . . .
conclusions
with sufflclent
data or
InformatIon
. . . . . . . . . . . . . . .
F&cognlzlng
the
lnterpt-etatlon.
@al lfylng
lmpl lcatlons
. . . . .
or consequences of an
. . . . . . . . . . . . . . . . . .
conclusions
appropriately
GeneratIng
new questions
CN- support
the
Oonslderlng
and recognlzlng
could be challenged
or experiments
lnterpretatlon
alternative
of data
concIuslons
with
Revising
a previously
Other
(specify)
. . .
to extend
Cornpat-lng concIuslons
observations.
Important
(correlational,
Determlnlng
cf-ltlcal
Wderate
Little
of evidence
supported
Ever
Fzle
I evant
In an
. . . . . . . . . . . . . . . . . . . . .
Eielng sensltlve
very
Frequent I y
what is already
known. .
new
. . . . . . . . . . . . . . . . . . .
-5-
REASONINGEMms
I I.
Please
indicate
frequency
with
kinds of reasoning
errors
in your graduate
students
and the
!Seriousness
Frequency
Never/
V'3t-Y
Hard I y Ever
Error
Failing
to recognize
differences
between a smple
Frequent
IW
very
SeriOUS
!ht-iOUS
and a
1
premise . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Confusing coincidence
. .
. . . . . . . . . . . . . . . . . . . . . . . .
population
Failing
of
interest.
to take
in a population
Offering
statements
about a persons
to oppose his/her
conclusion.
when a conclusion
and/or
Arguing that
Accepting
a conclusion
them.
with causation
. . . . .
should be accepted
assunptions
(proved).
. .
(rejected)
. . . . . ,.....
in an argunent
without
the credibility
or reliability
of a source
. . . . . . . . . . . . . . . . . . . . . . . . . . . .
irrelevant
evidence
into a text
interpreting
text
data.
to support
on the basis of
ones
a point
evidence.
information
Basing ccnclusions
. . . . . . . . .
insufficient
Al lowing anecdotal
statistical
within
. . . . . . . . . . . . . . . . . . . . . . .
Reading
or
the central
questioning
character
. . . . . . . . . . .
is concealed
correlation
because it
for a phenanenon
. . . . . . . . . . . . . . . . . . . . . . . . .
irrelevant
circumstances
. . . . . . . . . . . . . . . . . . . . .
on analysis
to override
of only
own experience.
more extensive
part of a text
or data
set...............................
Failing
to recognize
relevant
similarities
and dissimilarities
to argue . . . . . . . . . . . . . . . . . .
Other
(specify)
Other
(specify)
CRITICAL
III.
l-bw frequently
klnds of
of the analytlcal
ablllty
Incidents
IICIDENTS
or tendencies
In your students?
New-/
Hardly
Incidents
Repeatedly
Offerlng
maklng Irrelevant
crltlclsm
of other
not constructive
Submlttlng
falled
or supportlng
. . . . . . . . . . . . . . . .
. .
. . . . .
. . .
. . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
. . . . . . . . . .
presentations
that
was
lssues
. . . . . . .
Accepting
of recognized
wlthout
evaluating
Being unable to
them.
Integrate
. . .
authorltles
. . . . . . . . . . . . . . . .
and synthesize
sources.............................
Falling
to recognize
ccKIcluslon.
statements
Reoognlzlng
a pattern
or attemptIng
a formula,
justlflcatlon
Relylng
solely
breaks down. . . . . . .
assumptions.
. . . . . . .
algorlthm,
or other
on narratlve
rule
or descrlptlon
1s appropriate.
wlthout
about
sufflclent
. . . . . . . . . . . . . . . . . .
ccmplex or far-fetched
appropriate
. . . . . . . . . . . . . . . . . . . . . . . . . .
when analysis
Referrlng
when an analogy
to explain
MakIng lmplauslble
Applylng
can suppcrt
to support
lt
evidence
. . . . . . . . . . . . . . . . . . . . . . . . . . .
Uslng Inconslstent
Arguing
that
explanations
In results
over simpler
or to generalize
ones.
when
. . . . . . . . . . . . . . . . . . . . . . . . . . .
Falllng
to recognize
reached
(or
falllng
that
a deflnlte
to qualify
Ef feet
. . . . . . . . . . . . . . . .
the concIuslons
Ef feet
crltlcally
Frequent
an argunent
dlscusslons
Q-eat
bb
VWY
Ever
studentst
class
have
. .
or wel I founded.
a paper that
Accepting
remarks during
what effect
of your students?
conclusion
a concIuslon
cannot be
approprlately).
. . .
-7-
Never/
Hardlv
Searching
=lsts.
Ignoring
details
that
I Imlts
of a students
hypotheses
Effect
an expected cr desired
contradld
Q-eat
..*......*.....
result.
Being unwllllng
the
solution
. . . . . . . . . . . . .
lb
Ef feet
VV
Freauent
Ever
Independently
when
. . . . . .
. . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . .
alternatIves.
tests
of hypotheses
. . . . .
Other
W=lfy)
Iv.
2.
Acadmlc
rank
(e.g.,
3.
Instltutlon
4.
Department
5.
Years of teaching
6.
Your current
Associate
experience
teaching
BACGROUM INFOlMATION
Professor)
at the undergraduate/graduate
levels
7.
Altogether,
students
levels
Msters
Doctoral
youd
llke
Thanks for
your contrlbutlon
analytlcal
to our study.
abllltles?
past three
years?
Appendix B
Factor Analyses
Appendix B.l
Varimax Factor Loadings for Ratings of
Importance of Reasoning Skills
Appendix
Varimax
Factor
Loadings
for
Ratings
B.l
of Importance
of Reasoning
Skills
Factor
Variable
Reasoning
11
III
IV
Skills--General
Deriving
general
from disparate
or abstract
principles
facts or cases
of formal
are present
27
48
28
36
38
35
31
32
38
41
in
34
logic
38
53
Taking well-known
principles
and ideas from
one area and applying them to a different
specialty
Being able
position
Communality
to both
support
and refute
16
a given
51
Critically
analyzing
and evaluating
research or reports
in a field
Incorporating
isolated
instances
a preexisting
framework
Monitoring
one's
problems
own progress
Understanding,
arguments
40
30
46
previous
40
or data
45
into
23
in solving
36
Deriving
from the study of single cases
structural
features
or functional
principles
that can be applied to other cases
Recognizing
events
31
the probabilistic
nature
20
21
41
of most
32
analyzing
25
and evaluating
49
64
from a set
of
33
53
43
33
32
52
Appendix
B.l
(Continued)
Factor
Variable
11
Clearly
identifying
central
issues and
problems to be investigated
or hypotheses
to be tested
32
30
Recognizing
58
36
54
32
46
39
Snecific
Skills--Problem
Generating
alternative
of an issue
hypotheses
the variables
structural
of problem
Recognizing
problem
involved
Supporting
very
44
32
56
43
31
36
39
in a
to solving
61
problems
context
48
33
69
53
under
of a
67
--Constructing
general
assertions
with
details
57
Distinguishing
irrelevant
36
relationships
Drawing distinctions
identical
ideas
Using analogies
40
52
40
42
and
among observations
between similar
appropriately
55
Theses
40
between relevant
information
47
precise)
Making explicit
all relevant
components
in a chain of logical
reasoning
Perceiving
59
30
models for
the historical
Specific
Skills
or Arguments
Communalitv
24
similarities
between
or theory and another
Developing operational
(or
definitions
of concepts
Setting
up formal
consideration
or situations
Identifying
more than one approach
a problem
Recognizing
one type
IV
Definition
Identifying
problem
III
51
46
37
52
46
53
54
35
39
29
58
but not
Appendix
B.l
(Continued)
Factor
Variable
Elaborating
an argument
implications
Drawing
and developing
that
is
positions
central
Finding alternative
observations
43
49
a
44
major
34
57
argument
36
or thesis
34
53
and unstated
52
47
by
44
36
points
and logical
33
46
53
53
for
38
supporting
Distinguishing
argument
56
Arguments
explanations
Detecting
fallacies
in arguments
i
69
statements
Recognizing
into
63
Identifying
both stated
assumptions
in an argument
70
33
64
58
32
54
contradictions
in an
53
Being sensitive
to the strength
of different
types of evidence (correlational,
causal,
testimony)
41
Recognizing
shifts
in the meaning of a word
in the course of an argument
73
Judging whether
supported
64
a thesis
Communalitv
64
32
Translating
graphs or symbolic
into words and vice versa
Recognizing
the
in a work
internally
51
Skills--Analyzing
IV
its
from observations
Synthesizing
two different
third
position
Specific
III
72
sound inferences
Producing an argument
consistent
11
43
35
42
52
65
61
Appendix B.l
(Continued)
Factor
Variable
II
36
68
Distinguishing
among contributing,
and sufficient
factors
Specific
Skills--Drawing
Communality
64
necessary,
43
35
49
34
to account
72
59
31
69
58
47
45
54
48
56
57
data
or consequences
IV
Conclusions
Considering
III
to
conclusions
57
37
52
59
37
60
67
56
58
49
CommonVariance
Note.
8.41
7.37
as have all
3.84
3.21
2.51
decimal points.
25.34
Appendix B. 2
Varimax Factor Loadings for Ratings of Frequency and
Seriousness of Reasoning Errors
Appendix
Varimax Factor
Frequency
B.2
Loadings for
of Reasoning
Ratings
Errors
of
Factor
Error
IT.
Communality
Failing
to recognize
differences
between a
sample and a population
of interest
74
56
Failing
for
75
58
48
40
39
51
40
43
30
53
38
Offering
irrelevant
statements
about a
person's
character
or circumstances
to oppose his/her
conclusion
Failing
to recognize
when a conclusion
concealed within
a premise
Confusing coincidence
with causation
and/or
correlation
is
senses
65
a conclusion
should be accepted
because it has not been
(proved)
37
47
34
25
63
40
Failing
to evaluate
the credibility
reliability
of a source or text
63
44
67
47
63
46
70
55
Allowing
anecdotal
information
more extensive
statistical
42
53
46
58
30
42
50
40
42
Offering
irrelevant
a point
evidence
to support
Making generalizations
on the basis
insufficient
evidence
Basing conclusions
on analysis
of a text or data set
Failing
to recognize
and dissimilarities
to argue
CommonVariance
Note.
Loadings
less
or
of
to override
data
of only
part
relevant
similarities
when using analogies
than
4.06
2.60
as have all
decimal
6.66
points.
Appendix
B.2
of
Factor
II
Communality
Failing
to recognize
differences
between a
sample and a population
of interest
68
48
Failing
for
76
59
30
63
48
40
56
47
38
61
51
50
48
48
35
60
48
Error
Offering
irrelevant
statements
about a
person's
character
or circumstances
to oppose his/her
conclusion
Failing
to recognize when a conclusion
concealed within
a premise
Confusing coincidence
with causation
and/or
correlation
is
senses
a conclusion
should be accepted
because it has not been
(proved)
Accepting
the central
assumptions in an
argument without questioning
them
70
53
Failing
to evaluate
the credibility
reliability
of a source or text
71
56
75
61
72
56
Offering
irrelevant
a point
evidence
or
to support
Making generalizations
on the basis
insufficient
evidence
of
65
33
53
Allowing
anecdotal
information
more extensive
statistical
43
56
50
Basing conclusions
on analysis
of a text or data set
Failing
to recognize
and dissimilarities
to argue
to override
data
of only
part
39
55
relevant
similarities
when using analogies
46
44
40
Appendix
B.3
Appendix
Varimax
Factor
Loadings
B. 3
of Effects
of Critical
Incidents
Factor
I
Incident
Repeatedly
making irrelevant
discussions
Offering
that
remarks
criticism
of other
was not constructive
Submitting
a paper
assigned issues
that
students'
or well
failed
during
anecdote
and/or
opinion
with
and synthesize
Failing
to recognize
that
than one conclusion
evidence
Using inconsistent
position
statements
Arguing
of the obvious
in support
Being unable
down
data"
36
65
44
62
42
72
60
65
52
authorities
ideas
can support
to support
60
more on
"hard
implausible
Applying a formula,
without
sufficient
56
58
40
55
40
from
more
the same
38
when an analogy
40
37
39
49
assumptions
34
48
algorithm,
or other
justification
51
breaks
Recognizing
a pattern
but failing
to draw any
conclusions
about it or attempting
to
explain
it
Making
71
56
to perceive
Communality
the
Accepting or supporting
an argument based
emotional
appeal than on evidence
Confusing
III
class
presentations
founded
to address
I.1
38
47
48
40
53
53
67
45
rule
39
Preferring
complex or far-fetched
over simpler ones
32
37
41
explanations
40
33
Appendix
B.3
(Continued)
Factor
Incident
Being unable
generalize
to see a pattern
when appropriate
in results
11
III
Communality
40
42
40
32
51
or to
Failing
to recognize
that a definite
conclusion
cannot be reached (or failing
to qualify
a
conclusion
appropriately)
42
48
Writing
a one-sided
essay when a more balanced
treatment
is appropriate
48
41
contradict
44
when an
an expected
34
to generate
hypotheses
but unable
appropriate
independently
40
Loadings
56
42
54
39
35
44
59
tests
than
40
of
41
4.15
less
39
to suggest
CommonVariance
Note.
40
or
Being unwilling
to respond in a new or unknown
situation
when the limits
of a student's
knowledge have been reached
Being unable
58
.30
as have all
3.66
37
3.05
decimal
31
10.86
points.
Appendix
Varimax
Factor
Loadings
B.3
of Frequency
of Critical
Incidents
Factor
Incident
remarks
criticism
of other
was not constructive
Submitting
a paper
assigned issues
that
students'
or well
failed
during
Confusing
anecdote
and/or
opinion
with
and synthesize
Failing
than
evidence
to recognize
that
one conclusion
statements
Arguing
of the obvious
Being unable
down
46
22
to perceive
data"
53
59
39
authorities
41
ideas
can support
to support
when an analogy
implausible
Applying a formula,
without sufficient
44
29
36
35
the same
53
53
58
47
32
35
30
42
breaks
33
28
42
assumptions
complex or far-fetched
ones
39
more
32
50
42
54
34
rule
Relying solely
on narrative
or description
in
papers and reports when analysis
is appropriate
Preferring
simpler
47
from
47
algorithm,
or other
justification
27
71
Recognizing
a pattern
but failing
to draw any
conclusions
about it or attempting
to
explain
it
Making
36
more on
40
Using inconsistent
position
in support
31
the
"hard
Communality
52
37
Accepting
or supporting
an argument based
emotional
appeal than on evidence
III
class
presentations
founded
to address
II
explanations
28
39
over
49
30
Appendix
B.3
(Continued)
Factor
I
Incident
Being unable
generalize
to see a pattern
when appropriate
in results
III
Communality
47
45
45
or to
Failing
to recognize
that a definite
conclusion
cannot be reached (or failing
to qualify
a
conclusion
appropriately)
41
Writing
a one-sided
essay when a more balanced
treatment
is appropriate
65
II
42
43
48
when an
64
43
32
41
42
Being unwilling
to respond in a new or unknown
situation
when the limits
of a student's
knowledge have been reached
48
36
36
Being unable
72
54
52
39
68
52
Ignoring
details
desired result
that
contradict
an expected
or
39
to generate
hypotheses
but unable
appropriate
independently
to suggest
35
tests
of
CommonVariance
Note.
Loadings
3.75
less
than
.30
as have all
3.27
9.75
2.72
decimal
points.