ANALYTICAL REASONING SKILLS INVOLVED IN GRADUATESTUDY

:

PERCEPTIONS OF FACULTY IN SIX FIELDS

Donald E. Powers Mary K. Enright

GRE Board Professional ETS Research

Report Report

GREB No. 83-23P 86-43

December 1986

This report presents the findings of a research project funded by and carried out under the auspices of the Graduate Record Examinations Board.

Analytical

Reasoning Skills Involved in Graduate Study: Perceptions of Faculty in Six Fields

Donald E. Powers Mary K. Enright

GRE Board Professional

Report

GREB NO. 83-23P

December 1986

Copyright,

c 0

1986 by Educational

Testing

Service.

All

rights

reserved.

Acknowledgments The authors wish to thank the members the GREResearch Committee of for helpful suggestions; Laurie Barnett fr>r programming the analyses; Neal Kingston, William Ward, and Cheryl Wild for helpful reviews of the report.

Graduate faculty in six fields of study (chemistry. Fairness should be aided by including for measurement those skills Finally. extremely important in all disciplines. and incidents. errors.Abstract This study was intended to provide information on the role of analytical abilities in graduate study. and impact of these skills. (c) the definition and analysis of problems. Faculty members were generally able to discriminate among the various skills. and (e) the generating of alternative explanations or These dimensions were judged to be differentially hypotheses. These dimensions or factors involved (a) the analysis and evaluation of arguments. however. be much more important in some fields than in others would imply the need to carefully balance the measurement of various reasoning skills to ensure fairness to test takers from different fields of study. the rated as consistently important across all disciplines. English. errors. important for success in the six disciplines included in the study. engineering. Several relatively interpretable dimensions were found to underlie particular clusters of reasoning skills. The study results would appear to have definite implications for developing future editions of the GRE analytical ability measure. Some skills were viewed as More typically. and psychology) were asked to judge: (a) (b) (c) the importance for academic success of a wide variety analytical skills the seriousness of various reasoning errors had of the degree to which a variety of “ critical incidents” affected their estimations of students’ analytical abilities. (d) the ability to reason inductively. some reasoning abilities important are well represented in the current version of the That some abilities appear to analytical measure but others are not. (b) the drawing of inferences and the development of conclusions. several dimensions underlying various clusters of reasoning skills should be applicable to extending the test specifications for the current version of the analytical measure. . for example. computer science. were rated to be at although the vast majority of skills. seriousness. For that were perceived as very instance. education. disciplines varied quite markedly with respect to faculty perceptions of the importance. least moderately important on average. and incidents they were asked to consider.

1984) has given us somereason to question both the convergent and the discriminant validity of the two remaining item types. which constituted a very small part of the original analytical measure. the administration of the new measure to large numbersof examineesunder operational conditions enabled the further collection of information about the new measure. standardized admissions tests have tended to focus almost exclusively on the measurement broadly applicable verbal and quantitatice aptitudes. 1979). further research suggested serious problems with the two item types (analysis of explanations and logical diagrams) that comprised the bulk of the analytical section. of several possible new areas of measurement(e. 1983).g. 1983. after careful psychometric study of several alternative analytical item types. and study skills). Powers & Swinton. and students were most receptive to assessing analytical or abstract reasoning skills (Miller & wild. and that they do not therefore adequately reflect test takers' differential development in other important cognitive areas. Carlson. Developmental activities then followed and. administrators. four distinct kinds of items were selected for the new analytical section of the GREAptitude Test. scientific thinking. which was introduced operationally in the 1977-78 testing year. 1981). 1984) and to within-test practice (Swinton. Wilson. The most recent research on the General Test (Stricker & Rock. wild. 1982). 1975). 1985. Unfortunately. Performance on these item types was shownto be extremely susceptible to special test preparation (Swinton & Powers. the two currently used analytical item types correlate more highly with other verbal items or with other quantitati ve items than they do with each other. of One criticism of such omnibusverbal and quantitative ability measures is that they provide only limited descriptions of students' academic strengths and weaknesses. Consequently. Someresearch strongly suggested the promise of the analytical section: it appeared to measurean ability that was distinguishable from the verbal and quantitative abilities measuredby the test (Powers & Swinton. were inserted. Moreover. however. after reviewing the psycholog cal and educational research literature on i ran. A survey of constituents revealed that. In 1974 the GREBoard approved a plan to restructure the GRE Aptitude Test in order to allow examinees to demonstrate a broader range of academic skills (Altman. graduate faculty. Graduate institutions were cautioned against using the scores from the new analytical section until further evidence could be generated on the validity of the new measure. & Donlon. Specifically. PowerI? and Swinton (in press) concluded that the S reasoning. & Wallmark.. and the score derived from it was related to successful performance in graduate school (Wilson. abstract reasoning. Du . Subsequently. in 1981 the two problematic item types were deleted from the test.Analytical Reasoning Skills Involved in Graduate Study: Perceptions of Faculty in Six Fields Despite the complexity of humancognitive abilities. and additional numbersof analytical reasoning and logical reasoning items.

It was thought that this information might be especially useful for developing additional analytical item types. At this preliminary stage.. 30 department chairs (in English. The objective of the study reported here was to generate information that might guide the developmentof future versions of the GREanalytical measure. to develop hypotheses. it was felt that these departments might be more interested than nonrequiring departments in efforts to improve the GREGeneral Test. identifying assumptions on which an argument is based). As mentioned earlier. and asked to identify three faculty members their departments who would be willing to provide in their insights into the analytical or reasoning skills that are most critical for successful performance in graduate school. and to determine the relative importance of these skills or abilities both within and across academic disciplines. i.-2- two currently used GREanalytical item types reflect only a limited portion of the reasoning skills that are required of graduate students. The most notable omission is the assessmentof inductive reasoning skills.. the intention was to gain a better understanding of what reasoning (or analytical) skills are involved in successful academic performance at the graduate level. reasoning. particularly as these skills differentiate successful from marginal students . This survey.e. More specifically. Method Questionnaire Development Initially. or thinking skills they perceived as most important for successful graduate study in their fields (e. was not designed to provide any detailed information on the importance of specific analytical skills. examples of: (a) the analytical. education. however. or psychology) were contacted in 30 graduate institutions. or to integrate previously learned materials into a more useful and comprehensivebody of information. where the purpose is to learn new subject matter. faculty members were informed of the purpose of the project and asked to give. computer science.g. These 30 institutions were chosen from the GREDirectory of Graduate Programs in such a way as to ensure somedegree of geographical representation. reasoning from incomplete knowledge. as was the intention here. chemistry. in an open-endedfashion. the initial version of the GREanalytical ability measurewas developed after a survey had suggested the importance of abstract reasoning to success in graduate education. Thus. the analytical ability measureof the GPEGeneral Test might be improved through further effort. engineering. it seemed. All of these departments require or recommend that applicants submit GPEGeneral Test scores.

in press) and on a numberof additional books or texts on reasoning (e.who suggested a total of 138 important reasoning or thinking skills. (c) Useable responseswere obtained from 33 faculty members. Questions were grouped under several headings.-3(b) specific critical incidents related to thinking or reasoning that caused them to either raise or lower their estimation of a student's analytical ability (e.g. Kahane. a seminal work by Ennis (1962) and a list of skills by Arons (1979).g. mainly to give respondents somesense of their progress in responding to the rather lengthy questionnaire. proved especially useful." Some these responses of were duplicates. Rieke. and Weddle.g. Powers.g. 1984. "the ability to break complexproblems into simpler components"). Various issues of CT News. Previous work on critical incidents in graduate student performance (Reilly. and cognitive psychologists. Other responseswere revised or eliminated because they were too general (e. 1983. failing to qualify a conclusion as appropriate) particular reasoning or thinking "flaws" they have observed in their students (e. 1982. Several other respondents did not specify discrete skills or errors but chose rather to send helpful discursive replies to our inquiry.g. Several other articles. & Swinton.. 86 critical incidents. "the ability to resolve into enthymemicform any argumentative work" or "the ability to take ecological validity into account").. The structured questionnaire was constructed on the basis of this preliminary survey. and others because they were too specific or applied only to a particular field (e.. who gathered the impressions of ETS test development staff. Wason& Johnson-Laird (1972).. . Fischer. All responseswere condensedand edited. Nosich. "the ability to think independently"). 1974b) was also consulted. Some responses constituted usable questionnaire items essentially as stated by respondents (e. Campbell. and of specific incidents that may have led faculty to adjust their estimates of students' analytical abilities. Finally. and generally evaluated with respect to whether they should be included in the larger. the list generated by Tucker (1985). 1978). 1976.g. Johnson& Blair.. were also perused. of commonly observed errors in reasoning. 1970. The final questionnaire (see Appendix A) was structured to include questions about the importance and frequency of various reasoning skills. more structured questionnaire that was planned. published by the Critical Thinking Project at California State University at Sacramento. and several of the incidents related to critical facility were included in the present study.. 1984. Salmon.g. Striven. 1974. 1974a. 1984.. e. on a review of relevant literature (Duran. Toulmin. also proved to be a valuable resource. using the conclusion as the premise of an argument). & Janik. philosophers. and 75 reasoning "flaws.

according to both the magnitude of the eigenvalues and the breaks in their size. . (Our inclination was to err on the side of retaining too many factors at this exploratory stage. 40 institutions were selected for the final sample for each field. within the section on reasoning-skills.-4- The Sample Six academic fields (English. In GRE this manner. and analyses of variance were run for each question to assess differences amongthe six fields. education. psychology. These fields were thought to represent the variety of fields of graduate study and the variation in the kinds of reasoning abilities involved in graduate education. A randomsampling procedure was used such that eight institutions from each of the eight HEGISgeographic regions were selected for each field. it was felt that-the detection of uncorrelated factors would best serve the objectives of further test development. Finally. For example. the ratings of frequency-and importance were correlated. This sampling was greatly The admission requirements facilitated by the work of Oltman (1982). the data were factor analyzed to effect somereduction in the large number of questions. nonoverlapping samples of 64 graduate institutions with doctoral programswere drawn for each of the six graduate fields. one institution with a relatively large proportion of Black students and one with a relatively large percentage of Hispanic students were included in the samples for each field. was used to determine the numberof factors to be retained for each section.) Various numbersof factors were then rotated according to the varimax criterion. of these institutions were determined from the Directory of Graduate Programs (GRE/CGS. who were asked to nominate two faculty members who would be willing to complete the questionnaire. Letters were then sent to departmental chairpersons. thus raising the total numberof institutions to 42 per field. with squared multiple correlations as the initial estimates of communalities. chemistry. The various ratings were correlated within questionnaire sections. and engineering) were included in the final survey. Although other oblique rotations could have been used also. A principal axis factoring. and only those that either required or recommended General Test scores were included in the sample. computer science. In addition. within each section (and for each kind of rating).1983). Respondents were paid $25 for their participation. within the section on reasoning errors. Data Analysis Means and standard deviations were calculated for each question by academic field of study. Using the data tapes of the Higher Education General Information Survey (HEGIS). the ratings of frequency and seriousness were correlated.

Engl i sh faculty. for example. Full professors constituted a slight majority of the responding sample (51%) . saw greater importance in skills involving argumentation-.0 or higher varied markedly by discipline as follows: 23 for chemistry. 29 for English. Tables 1-3 show a substantial number of significant differences among disciplines with respect to the importance placed on various reasoning skills (Table l). Table 4. gives a flavor of the differences among these six disciplines. or experiments. analyze. and to analyze and evaluate previous research.94.99. for “ frequency of use” and “ importance for success” correlated . the seriousness with which they regard particular kinds of reasoning errors (Table 2). or lecturers. to draw sound inferences from observations. on the other hand. since their correlation with frequency ratings was . The numbers of reasoning skills that received average ratings of 4. questions. showing only the very highest rated skills and most critical errors and incidents. Numbers under each discipline represent for each item the deviations from these means. 5 for computer science. comparable. evaluate. For example. Item-level Results Tables 1-3 show the mean ratings by discipline for each question included in the survey instrument. 22 for engineering. and 26 for psychology. and the impact that various critical incidents have on the estimation of students’ analytical abilities (Table 3). over all respondents. for critical incidents. Faculty in the six disciplines also appeared to have quite different views as to the numbers of skills that were important in their respective disciplines. recognize.98. The numbers in the total column are the grand means for all disciplines.-5Results The Sample A total of 165 chairpersons (65% of those contacted) nominated a total of 297 faculty members. only the importance ratings are presented for reasoning skills. chemistry faculty placed a high premium on being able to generate hypotheses. Likewise. only the “ seriousness” ratings are presented for reasoning errors. and. Because the average ratings. and support aspects of an argument. The F tests in the right-most column indicate whether the means are significantly different among the six disciplines.being able to understand. About 13% were assistant professors. 27 for education. These differences may have arisen. only the average “ effect” ratings are presented. and the remaining small proportion were deans. from our . associate professors made up the next largest proportion (34%). associate deans. of whom255 (86%) returned usable The response rates across fields were generally questionnaires. elaborate. since their correlation with frequency ratings was .

and for which average ratings did not differ significantly across disciplines. and adequately supported by. . however.” “ being unable to integrate and synthesize ideas from various sources.” and “ being unable to generate hypotheses independently. Three errors/incidents were judged to be most serious or critical: accepting ‘ the central assumptions in an argument without questioning them. from differences disciplines. breaking ‘ down complex problems into simpler ones” was rated as the single most important skill (of the 56 skills listed) in both computer science and engineering. “ drawing sound inferences from observations” was the highest rated skill in chemistry and nearly the highest in education.” Table 6 lists the reasoning errors and critical incidents that were judged overall to be the most serious or to have the most effect on the estimation of students’ abilities. For example. and incidents that were viewed as relatively important. ” and “ recognizing structural similarities between one type of problem or theory and another” were the next most highly rated skills.” It should be noted that there are many other decision rules. errors. These were followed closely by “ taking well-known principles and ideas from one area and applying them to a different specialty.” “ deducing new information from a set of relationships. The extent to which faculty in different disciplines agreed on the importance of various skills.” and “ s deriving from the study of single cases structural features or functional principles that can be applied to other cases. analyses were computed for each section of the questionnaire. or incidents can be examined in a slightly different manner.-6particular choice of questions.5 over all six disciplines combined. even from Table 4. that some skills were viewed as very important by several disciplines. errors.’ “ monitoring one’ own progress in solving problems. Such skills as detecting ‘ fallacies and logical contradictions in arguments. “ Determining whether conclusions are logically consistent with. based on average ratings and differences among disciplines. or from some other factor(s) . Table 5 was prepared. that could have been used here to form a “ commoncore” of skills or errors/ incidents. and for which analyses of variance did not detect any significant differences among disciplines. To get some idea of the skills. Factor Analytic factor Results To condense the many questions into a more manageable form. the data” was rated as one of the three most important skills by both education and psychology faculty. “ Reasoning or problem solving in situations in which all the needed information is not known” was the skill rated as most important overall. Tables l-3 could be consulted to apply alternative rules. in standards amonq It can be seen. This table shows only those skills that received average ratings of importance of more than 3.

g. and drawing sound inferences from observations. Factor I. not the degree to which these dimensions represent “ factors of the mind. generally.15%) than did Factors I and II. from skills involving arguments.-7- For the section on reasoning skills. Table 7 summarizes the variables that were most instrumental in defining each factor.” Thus. 1. 2. as is the case for Factor I. indicate logical necessity. and was defined primarily by variables related to the drawing of conclusions. the results presented below are intended to provide a parsimonious representation of faculty perceptions rather than a basis for postulating distinct analytical abilities. The reader should bear in mind that the factors resulting from this analysis should not be construed as representing dimensions of analytical ability.4. 1966) suggested the appropriateness of a five-factor solution. These dimensions merely reflect the extent to which graduate faculty tended to rate certain skills as about equally important (or equally unimportant ) . but not enough to Factor V is somewhat difficult to define.1. but.4. which was then rotated The five-factor according to the varimax criterion (Kaiser. loadings and communalities are given in Appendix B. I generating valid explanations. and the application of a scree test (Cattell. best defined by skills related to defining and setting up problems or analyzing their components as a prelude to solving them. which accounted for about a third of the common variance. Reasoning skills. only the importance ratings were analyzed because they were so highly correlated with frequency Because frequency ratings were slightly less correlated with ratings. supporting conclusions with e. Thus. i. and 1. the drawing of conclusions that have some evidential support..9. was characterized by highest loadings. conclusion-oriented skills that define this second critical thinking factor would seem to be of a more active or productive nature. . Factors III-V each accounted for a somewhat smaller proportion of Factor III is common variance (10% .e. Factor I seems to involve a kind of critical thinking related to argumentation. 3. by virtue of its two highest loadings. For the ratings of importance of reasoning skills. involving the construction of inferences or conclusions. Factor II accounted for about 29% of the common variance. 1958). Factor IV is best characterized by inductive reasoning skills. they too were analyzed for the questionnaire sections on reasoning errors and critical incidents. ratings of seriousness and criticality in the other two sections. but rather only as reflecting the dimensions that underlie faculty perceptions of analytical abilities. the largest eigenvalues were 16. it seems to reflect an ability to generate alternatives.6. The sufficient data. rather than evaluating the soundness of arguments or inferences. The factor varimax rotation accounted for 80% of the common variance.

failing to recognize differences between populations and samples. seemed to involve more formal logical errors. and this three-factor of the common variance. particularly as related to reasoning with more statistically oriented material--for example. being able to criticize but unable to suggest better alternatives. also factor analyzed and are presented in Appendix B. accounting for about 34% of the common variance. and confusing correlation with causation.g. and somewhat less important in the other three disciplines. failing to take account of a base rate. e. is difficult to interpret. and the complete set of loadings is given in Appendix B. Factor III. This factor appears to involve critical facility. 1. defined by such incidents as applying a formula or rule without sufficient justification.97). Table 10 gives the average scores for each discipline on scales composed of the questionnaire items that best defined the reasoning skill factors discussed earlier. they are not discussed here.53).) As shown in Table 8.6. there are substantial differences Skills involved in analyzing/ among disciplines on these scales. Factor II. being unable to integrate ideas from various sources. Factor I.5 and 1. was characterized by loadings from errors involved in the evaluation of evidence. and the two factors (Frequency ratings were accounted for 96% of the common variance.” presentations. offering nonconstructive or unsound criticism of other students’ and confusing anecdote and/or opinion with “ hard data. the largest eigenvalues were 6. . solution accounted for 89% the largest factors. errors. evaluating arguments (Scale 1) were rated as extremely important in English (m = 4. particularly computer science (m = 2.. explaining about 38% of the conanon variance. the effects of critical incidents. Factor I. was best defined by highest loadings from such incidents as accepting/ supporting arguments based more on emotional appeal than evidence. Because the results were so similar to the analysis of seriousness ratings. As for reasoning errors.2. other hand. and 0. on the offering irrelevant evidence to support a point.9 for analyzed. and being unable to generate hypotheses.-8- For the ratings of seriousness of reasoning Reasoning errors.83) and psychology (m = 3. and searching for a complicated solution when a simpler one is obvious.1. only ratings of Critical incidents. not their frequencies. appears to involve the ability to consider or to generate alternatives. I’ Scales based on reasoning skill factors. Table 9 summarizes the results. were factor This analysis yielded eigenvalues of 8. Factor II. quite important in education (m = 3. which explained about 52% of the common variance. being defined primarily by high loadings from such incidents as accepting conclusions without critically evaluating them.73). As is clear. One possible characterization might be a kind of rationality or critical facility that is sometimes referred to as practical judgment or perhaps “ commonsense.

by one respondent. in which they had greater interest. . The skills composing Scale 5. lamented that the questions were “ too fine. therefore. especially English (m = in analyzing and defining problems (Scale 3) important in computer science (m = 4.” and “ your lists are right questions. characterized the questions as oriented more toward argument than problem solving. particularly computer science (m = 2. questionnaire was pitched at about the right level and included appropriate kinds of reasoning skills. ” “ topics covered are critical. Moreover. than to theirs. most of the comments about questions were positive: “ Items seem especially well chosen. were rated very high in psychology (m = 4. ” but also. ” “ suggested that the on target. and a computer science faculty member observed that questions on abilities involved in formulating proofs. and as somewhat less important in other disciplines.0) in all disciplines except computer science. ” The majority of comments. a better job of assessing skills associated with hypothesis-testing than with other research skills.” “ questions are questions were quite thorough.05) and but less important in other disciplines. ” Finally. a number of other respondents also believed that the questions were more relevant to other disciplines Several computer science professors. and many pertained more specifically to the kinds of questions that were asked. as a “ complete waste of time.” Even with these difficulties. or sciences. ” Most of the commentswere positive. An engineering professor said that some of the questions were more pertinent to educational research than to scientific or technical research. Abilities involved were rated as extremely engineering (m = 4.93). were only partially An education faculty member noted that the survey did questionnaire. 2.70). however.21) and in education (m = 3. The inductive reasoning skills reflected on Scale 4 were rated as moderately important on each of the six disciplines. for example.” “ good set of a appropriate. which are vital covered in the to success in computer science.-9- Critical thinking skills involved in developing or otherwise dealing with conclusions (Scale 2) were viewed as very important (means greater than 4. and one English faculty found that questions seemed “ geared to the hard some noted ambiguities or redundancies.90). however. Along these same lines. faculty in the several disciplines sometimes had different ideas as to what kinds of questions would have been appropriate. one English faculty member noted the lack of questions on the use of language in critical writing. Other Commentsfrom Respondents A number of general commentswere made about the study--some positive and some negative. The study was described alternately as “ very well done” and “ interesting. For example. The consensus seemed to be that the questionnaire was not easy to complete. generating alternatives/hypotheses.00).

analytical ability without regard to the amount of knowledge available. they leave the t I’ program. such general skills as reasoning ‘ or problem solving in situations in which all the needed information is not known” were viewed as extremely important in all disciplines. If they can’ learn. with respect to both the seriousne ss of Facul ty also made distinctions different kinds of reasoning errors and-the effects of various critical incidents. that successful problem solving is predicated on having specific knowledge in a field..D. the essence of graduate training is analytical skills. engineering. ” Many noted that by the end of their programs. (b) judged the seriousness of various reasoning errors. although the vast majority of skill s were seen.g. skills would be expected to increase and various reasoning errors could be expected “ Entering students are more likely to make to occur less frequently: these errors and graduates to make far fewer.” and “ being unable to generate hypotheses independently” were judged to be very serious in all disciplines. e.’ Discussion Summary Some 255 graduate faculty in six fields of study (chemistry. and psychology) (a) rated the importance for success in their programs of a wide variety of reasoning skills. computer science. Several faculty commentedon the development of analytical skills in graduate school and on the differential importance of these skills one respondent said. “m more concerned about the presence of these behaviors after my course than before it. education. and (c) indicated the degree to which a variety of “ critical incidents” affected their estimations of students’ analytical abilities. Individually. rated entering behavior or behavior across the entire program If I were to rate the dissertation). internships. “ In some sense. but one could be very critical of a student about to finish a Ph...” Another said. on average. the ratings would have been much higher.. dissertation experience alone. as at least moderately important.“ being -“ unable to integrate and synthesize ideas from various sources.-lO- A number of commentswere made about the relationship between subject matter and analytical skills. (courses. thesis. English. Faculty members were able to di scriminate among the various skill s they were asked to consider.’ “ These Flhen they enter they make most of are skills which students acquire. ” Another said. .” and another noted that the measurement of analytical abilities is quite Another commentedon the difficulty of measuring discipline specific. Sucfieasoning errors and critical incidents as “ accepting the central assumptions in an argument without questioning them. “ I at various stages of graduate education. One respondent believed that the questionnaire downplayed the importance of “ context effects” in favor of “ strict reasoning ability. the mistakes you mentioned. One simply does not harshly judge a beginning student who makes an error.

errors. Converting this information to operational test items will represent a significant step. and drawing inferences/developing conclusions was rated as relatively important in all disciplines except computer science.-llMore typically. Inductive reasoning skills were judged to be about equally important in all disciplines. which were perceived as differentially important for success in each of the six disciplines included in the study. seriousness." that seemwell represented in the two item types (analytical reasoning and logical reasoning) currently included in the analytical section of the General Test. and generating alternatives most important in psychology and education. . the findings do seemto contain several useful bits of information: 1. Nonetheless. errors. however. As migh be . Implications In providing someinformation on faculty perceptions of the involvement of various reasoning skills in their disciplines. and (e) the generating of alternative explanations or hypotheses formed five distinct dimensions. Skills involved in (a) the analysis and evaluation of arguments. or incidents. implications for developing future versions of the GREanalytical ability measure. (d) the ability to reason inductively. In this regard. "Reasoningor problem solving in situations in which all the needed information is not known"was amongthe skills rated as most important in each discipline. however. (c) the definition and analysis of problems. and it is not crystal clear at this stage exactly how helpful these results may be eventually. disc iplines varied quite markedly wi th respect to faculty perceptions of the importance. the previous 2.t expected. Someskills that are not measuredby the current version of the analytical measurewere rated as very important. This suggests that these item types should continue to play a role in future editions of the GREGeneral Test. e. Among the specific reasoning skills perceived as the most important were several. however.g. whereas "knowingthe rules of formal logic" WEIS rated as one of the most important skills in computer science. defining and analyzing problems most important in computer science and engineering. we hope. thi S skill was rated as quite unimportant in all other disciplines. Several relatively interpretable dimensionswere found to underlie faculty perceptions of the substantial numberof reasoning skills. "deducing new information from a set of relationships" and "understanding.. (b) the drawing of inferences and the developmentof conclusions. evaluating. and incidents that were rated in the study. and analyzing arguments. but currently unmeasured. for example. Analysis and evaluation of argumentswas judged to be most important in English. by the analytical measure.. the study has. or impact of various reasoning skills .at least in any explicit manner.

not merely choose. and Woisetschlager These investigators studied (1983) is noteworthy. With respect to their perceived importance. and noted the resemblance of these problems to one variant of the logical reasoning item type used in the analytical measure. 1977. 1978. however. including one especially promising Formulating Hypotheses. The results of the faculty survey reported here would appear to support this renewal. some skills seen as highly important in some disciplines but not in others may not be as well represented currently. current study suggest that the inclusion of this item type would probably meet with faculty approval in most fields of study. Ward & Frederiksen. Although the research suggested that this item type complemented the GRE verbal and quantitative measures in predicting success in graduate school. and the inability independently was one of the incidents rated consistently as having a substantial effect on faculty perceptions of students’ analytical abilities. “ breaking down complex problems into simpler ones” was perceived as . the work was discontinued. A number of years ago the GRE Board sponsored a series of studies (Frederiksen & Ward. “ ill-structured” problems. Ward. analyzing. However. Frederiksen & Ward. a correct response. and evaluating arguments” was seen as more important in English than in computer science. For example. i . 3. 1975) that explored the development and validation of tests of scientific thinking.” which required item type called “ examinees to generate hypotheses. . “ understanding. For example. skills involving the generation of hypotheses/alternatives/explanations tended to generate hypotheses to cluster together. They did note. 1978. & Carlson. that the “ ill-structured” item type could be used to increase the The findings of the variety of items types in the test. Carlson. They concluded that there was no indication that “ illstructured” problems measure different aspects of analytical ability than do “ well-structured” problems. Frederiksen. problems that do not provide all the information necessary to solve the problem. and therefore that “ ill-structured” problems could not be expected to extend the range of cognitive skills already measured by the GRE General Test. 4. Carlson and Ward (1986) have proposed to renew work on the “ Formulating Hypotheses” item type in light of recent advances in evaluating questions that involve constructed responses. Some of the highly important skills that are currently well represented in the analytical measure are viewed as more important for success in some disciplines than in others. largely because of problems in scoring items that require examinees to construct.-12GRE-sponsored work of Ward. e.

6. First of all. the verbal skills measured by the GENE General Test in the respect that these analytical skills may develop much more rapidly. is how to accommodate the measurement of these skills in the context of graduate admissions testing. then. This would suggest. . perhaps. so that skills thought to be important (or unimportant) in particular disciplines are neither over. A question of interest. Other skills judged to be very important in only a few disciplines might best be considered for extending the measurement of reasoning skills in the GRE Subject Tests. Other approaches that might also be informative include the methods of cognitive psychology. 5. The reasoning skills that were rated as very important. and consistently so. which could be used not only to supplement but also to extend the survey results These methods would seem especially appropriate reported here. Tucker’ s (1985) results provide useful information from different perspectives--those of cognitive psychologists and philosophers.nor underrepresented. the substantial differences found among graduate fields. especially if new item types are developed to represent some of these dimensions. represent competencies that differ from. Second. Finally. since we cannot be certain whether or not some other sample of fields might exhibit even greater variation. Limitations Any study of this nature is necessarily limited in several respects. the need to balance the inclusion of items reflecting particular skills. as several survey respondents pointed out. and do. fields are a source of concern. which currently focuses on the predictive effectiveness of abilities that are presumed to develop slowly over a significant period of time. across disciplines point to a potential common core of skills that could be appropriately included in an “ all-purpose” measure like the GRE General Test. the diversity that characterizes graduate education Some clues have been renders the results of this study incomplete. because they relate more directly to actual skills and abilities than to perceptions. say. improve In some sense these skills may as the result of graduate study. gained as to similarities and differences among a limited sample of However. The several dimensions that appear to underlie clusters of reasoning skills may provide an appropriate way to extend the current test specifications for the analytical measure. Faculty comments about the difficulty in separating reasoning from subject matter knowledge would seem to support this strategy.-13extremely important in computer science and engineering but not at all important in English. many of the reasoning skills about which we asked are expected to. the survey approach used here is but one of several that can be used to inform decisions about extending the measurement of analytical abilities.

OII the other hand. could probably be omitted without much loss of information. This strategy would make especially good sense for those important skills that may not be measurable in an operational test like the GRE General Test. . but which might correlate highly with the abilities now measured One specific possibility would be the development of by the test. ratings of the frequency with which skills are used. These ratings could then be used as a criterion against which GRE analytical scores could be judged. Some refinements could now be made on the basis of past experience. based on the reasoning skills identified as most important. ranging from entry level to dissertation writing. For example.-14- Future Directions The study suggested several possible future directions. one possibility would involve extending the survey to include additional fields of graduate study. which could be used by faculty to rate the analytical abilities of their students. as well as the frequencies of errors and critical incidents. it would seem desirable to add categories allowing ratings of the differential importance of various reasoning skills at different stages of graduate education. criterion tasks might be developed against which the validity of the current GRE analytical measure could be gauged. Because of the substantial variation among fields. Finally. rating forms.

Directory of graduate programs: 1984 and 1985 (4 volumes). (1970). B. A concept of critical thinking: A proposed basis for research in the teaching and evaluation of critical thinking Harvard Educational Review. W. (1978). H. Cattell. (1974). R. Carlson. & Ward. Princeton. Princeton. 75-18). Princeton. (1986).. Flaws and fallacies in statistical Englewood Cliffs. Franklin Institute Press. . P. Frederiksen.. NJ: Educational Testing Service. J. 81-111. (in press). 2. T. A. Fischer. W.. PA: The (Eds.. Princeton.-15- References Altman. Developmentof measuresfor the study of creativity (GREBoard Professional Report GREB No. (1975). B. Somethoughts on reasoning capacities implicitly expected of college students. 72-2P. R. Applied Psychological Measurement. S. Multivariate Behavioral Research. S. Duran. & Blair. A. Carlson. 74-3). N.. C. Johnson. D. Canada: McGraw-Hill Ryerson Limited. (1979). S. ability. N. R. (1962). NJ: Educational Testing Service. C. R. A. Publishers. Developing a plan for the renewal of the GREnorming test: A report on I . Campbell. E.). & Ward. K. Measures for the study of creativity in scientific problem-solving. (1975). A new look at formulating hypotheses items (proposal to the GREResearch Committee). January 1984). (1983). NJ: Educational Testing Service. Historians' fallacies: Toward a logic of historical thought.. EMiS. C. Lochheadand J. Arons. & Ward. (1966). investigations znto the possible shortening of the GRE-Vand GRE-Qand the creation of a modular aptitude test (GREReport GREB No. W.. B. H. Inc. & Donlon. 1. D.. NJ: Educational Testing Service. Cognitive process instruction. & Swinton.). Graduate Record Examinations Board and Council of Graduate Schools in the United States. S. NewYork: Harper & Row. Princeton. . NJ: Prentice-Hall. Clement Philadelphia. Construct validity of the GREanalytical test (Draft final report to the GREResearch Committee. H. 32.. Frederiksen. B. F. NJ: Educational Testing Service.l-24. thinking. R. A. In J. Logical self defense (2nd ed. Powers. (1983). The scree test for the numberof factors.245-276. also ETS RB No.

Restructuring the Graduate Record Examinations Aptitude Test (GREBoard Technical Report).. Journal of Educational Psychology. Miller.Inc. (1982). D. Investigation of practice effects on item types in the Graduate Record Examinations Aptitude Test (GREBoard Professional Report GREB No. (1982). L. (1976). J. 83-10~~). The varimax criterion for analytic rotation in factor analysis.. M. R. K. (1985). Belmont. R. Powers. (1983). C. 266-278. E. Princeton.. (1984).). NJ: Educational Testing Service. wild. H.. 75. (1974a). & Swinton. R. Testing Service. . S. -23. Reilly. H. (1984). & Rock. CA: WadsworthPublishing in everyday lite (4th ed.. Princeton. & Swinton. computer science. (1984). Content representativeness of the Graduate Record Examinations advancedtests in chemistry. & Wallmark. G.-16- Kahane. thinking.. (Eds. L. (1981). R. M. NewYork: McGraw-Hill Book Company. Company. 104-115. Nosich. 76. A. E. Princeton. Logic and contemporary rhetoric: The use of reason Belmont. & wild. (1958). CA: Wadsworth Publishing Company. Princeton. NJ: Educational Testing Service. Salmon. S.Inc. S. The factor structure of the GREGeneral Test for older examinees: Implications for construct (GREBoard Research Report GREB No. S. (1983).). Reilly. & Powers. Publishers. Extending the measurement graduate admission abilities beyond the verbal and quantitative domains. Princeton. C. S. E. Reasonsand arguments. P. D. 187-200. S. S. 70-5). Psychometrika. 71-2P. Reasoning. H. 01tman. M. F. D. NJ: Educational Testing Service. (1974b). Journal of Educational Psychology. Critical incidents of graduate student performance (GREB Board Research Report GREB No. 141-158. Striven. NJ: Educational Testing Service. 80-1cP). D. NJ: Educational Testing Service. Effects of self-study for coachable item types. Princeton. R. Introduction to logic and critical New York: Harcourt Brace Jovanovich. Factors in graduate student performance (GRE Board Professional Report GREB No. also RB 74-2). Swinton. of Powers. and education (GREBoard Professional Report GREB No. M. Kaiser. Applied Psychological Measurement. (1979). M. 5.5. Swinton. Stricker. A study of the effects of special preparation on GREanalytical scores and item types. 81-12~).

Ward. A. Frederiksen. Weddle. Delineation of reasoning processes important to the construct validity of the analytical test (GREReport submitted to the GREResearch Committee). (1984). & Woisetschlaeger. W... New York: Wils[on. S.-17Toulmin. Princeton. Ward. MA: Harvard Universtiy Press. Illstructured problems as multiple-choice items (GREBoard Professional Report GREB No. M. (1972). Cambridge. Psychologyof reasoning: Structure and content.. The relationship of GREGeneral Test item-type part scores to undergraduate grades (GREBoard Professional Report GREB No. 81-18P).. NJ: Educational Testing Service. NJ: Educational Testing study of the predictive thinking (GREBoard also ETS RB No. A study of the validity of the restructured GRE Aptitude Test for predicting first-year performance in graduate (GREBoard Research Report GREB No. NJ: Educational Testing Service. 78-15). K. NewYork: Macmillan Publishing Company. Tuck er. NJ: Testing Service. N. Inc.). (1978). thinking... C. Princeton. 81-22~). 77-6). An introduction to reasoning (2nd ed. Princeton. Service. W. (1978). 74-8P. 74-6P. C. N. Ward. (1983). W. & Frederiksen. NJ: Educational Testing Service. M. S. (1982). & Janik. & Carlson. Rieke. also ETS RB No. P. Carlson. Argument: A guide to critical McGraw-Hill Book Company. (1984). K. B. C. A validity of the tests of scientific Professional Report GREB No. Princeton.. (1985). R.. C. 78-6R). Wason. C. N. Princeton. E. Construct validity of free-response and machine-storable tests of scientific thinking (GREBoard Professional Report GREB No. Wilson. & Johnson-Laird. NJ: Educational Testing Service. P. P. S. (1977). Princeton. .

16 -.23 7.70 .03 -.16 -.09 -.21 -.05 .19” ” 1.31 -.03 -.13 .34 .04 -.Table Mean Ratings of Importance of 1 Reasoning Skills by Disciplines Computer Tota Variable I Chem i stry (N=37) SC ience (N=43) Education (N=42) Engineering (N=43) Engi ish (N=44) Psychology (N=46) F - (N=255) Skills--General Reasoning Deriving Reasoning Reasoning principles when facts when all from are facts or cases 4.59 1.25 7.65 3.52 -.06 -.66 .13 .81*** known explicitly is not information 4. 10 -.37 3.15 -.76 .86 .23 -.03 2.03 -.21 .02 3.I2 3.63*** Incorporating isolated data into a framework 3.Ol -.08 -.70 9.35”” “ .06 .14 -.20 -.71 Monitoring Deriving other progress principles cases in solving that can problems be applied to .ll .63 .80** 10.18 0.09 -.16 .07 .60 evidence or .47 .46 .34 -.22 10.92*** Reasoning Knowing when rules inconsistencies of formal logic are present 4.16 .62 .31 .19 .24 -.47 -.38 0.04 .31 -.68 0.33 .63 known -.oo -.05 -.60 5.23 . hwotheses .26 .44 .24 -.74*** Analyzing and evaluating previous research 4.51*** Deducing Knowing I information what - 3.59 .75 .35 -.75 -.02 .I1 -.11 .40 . thesis 4.44 .22 -.71 .I5 .23 -.15 .09 .14 1.79 Recognizing the probabilistic nature of events 3. 12 3.32 -.13 -.23 9.lO .29 -.02 .62”” “ 1.31 -.38 .55 -.Ol 0.05 Supporting or refuting a given position 3.24 .56 .40 -.ll -.29 .09 .14 10.21 .07 -.49”” “ Understand i ng and eva I uat ing arguments from will relationships support a 4.32 Applying principles to a different specialty 3.23 -.40 -.06 .

02 .38 .17 -.oo .88 problems problems . 93 1.06 .05 -.22**t” explicit components reason i ng relevant and irrelevant 3. 10 .04 .30 .98 3.13 .45 .07 -.39 -.03 .83 3.13 -.Table 1 (Continued) Discipline Computer Tota Variable I Chem 1stry (N=37) SC I ence (N=43) Ed ucat (N=42) i on Engineering (N=43) Engl ish (N=44) Psychology (N=46) + (N=255) Skills--Problem Definition Specific Identifying Recognizing central sides of issues an or hypotheses 4.70 .09 -.17 .51 .34*** .29 -.74 -.02 -.74 .85 .97 -.05 .02 -.65*** Recognizing Specific Skills--Constructing Theses or Arguments Supporting Making assertions with details In a chain of 3.49*** definitions Setting up formal the models historical context 3.80*** 14.36 2.16 .59”” “ issue Generating Breaking alternative down complex hypotheses problems 3.34 -.78** .65**” 3.09 1.54 Identifying variables in a problem 3.20” ” 10.11 3.Ol .36 -.59 -.15 -.66 .57 -.46 1.24 .06 -.13 -.20 -.49”” “ 26.09 .04 11. 50 .16 . 17 4.57 .16 -.07 . 10 . 11 -.36 .05 -.59 -.62 .41 .47 7.87 4.75 .66 .36 -.10 Distinguishing information 4.92 -.06 .25 -.20 .22 .08 -.22 13.03” L Lo I Identifying Recognizing Developing approaches similarities operational to solving between 3.43 -.22 .39 .34 3.13 -.73 -.13 -.05 .16 5.60 .55 -.25 -.58 -.26 3.22 . 12 -.Ol -.

22 -1.57 -.22 .19 7.43 .12 -.27 .74*** 6.18 -.20 7.44 .57 .10 3.05 -.67”” “ I a consistent different graphs .18 Distinguishlng major points in an argument 3. 52 -.25 .03 .33 .67*** appropriately 3.06 .09 -.67 .16 .lO .27 -.47 -.12 -.40 .89*** alternative explanations .lO .48 -.84”” “ 2.11 5.14*** Identifying Testing Finding the validity of an argument 3.56 -.22 .18 7.13*** 3.04 .15 .02 3.26 .23 .09 .08 -.12 .55 1. 11 .82 -. 14 .19 .64 3.Ol .25 4.63 .59”” “ Elaboratlng Drawing Producing Synthesizing Translating an argument inferences argument positions or symbolic statements 3.55 -.02 .28 -.Ol 8.34 l 03 -.02 -.06 -.80 4.18 -.93** 18.12 -.36 .52 -.37 -.70 -.12 -.10 3.32 .03 -.74 -.07 2.84”” “ .26 .30 .54 .21 6.17 -.17”” “ 7 .57 3.04 sound .61 .20 -.16 -.16 .13 .13 .04 -.38 -.92 .36 -.56 .53 1.35 3.84”” “ 4.02 -.15 -.98 .28 Recognizing Detecting supporting fallacies points and contradictions 3. 16 .50 Specific Recogn Skills--Analyzing Arguments argument I zi ng the central assumptions 4.Table 1 (Continued) Discipline Computer Tota Variable I Chemistry (N=37) Science (N=43) Education (N=42) Engineering (N=43) Engl ish (N=44) Psychology (N=46) E (N=255) relationships distinctions analogies among observations between similar ideas Perceiving Drawing Using 4.45 .33 .06 .30 .02 -.04 .oo -. 57 -. go* 18.15 -.04 3.

Table 1 (Continued) Discipline Computer Tota Variable Being sensitive to different in meaning evidence of an argument supported supported I Chem i stry (N=37) -.71 .25 .77 .31** Specific Generating Supporting Skills-- vat id explanations conclusions 4.07 8.27 .90”” “ Generating new questions 3.05 Engl ish (N=44) .1.07 new observations 3.86*** 33.18 .81 -.52 .90 3.72 .24 of an interpretation .14 .08 Sc i ence (N=43) Education (N=42) . 59 -.18 -.40 -.85 3. totals “ p *yp ““ “p Entries range under from .14 .91 .07 .07 . 14 -.08”” “ 6.13 -.27 F 18.lO -.15 -.26 . grand means given under ltTotaI.47 -.84 .27 .24 .16 .15 .28 .65*** (N=255) 3. 28 .95 .05 1.82*** 4. 20 Distinguishing among factors Drawing Conclusions 3.45 -.Ol < .9 from and the 1.09 8.lO Engineering (N=43) -.76*** 8.23 .42 -.55 -.22 Recognizing implications 4. 77 .73 .58 -. 96 -.25 -.61 -.26”” “ 8.35 . 32 .88 3.99 or experiments .18 .07 Note.92*** 14.86 4.31 -.31 2.33*** 3.lO .25 .11 Psychology (N=46) .06 -.78 individual to 1.31.29 Qualifying conclusions 3.38 3. 16 .50*** 12.30 -.34 -.38”” “ Considering Comparing Revising alternative conclusions views for conclusions .14 .73 .38 10.07 . 79 Recognizing Judging shifts whether a thesis has been Determining whether conclusions are . discipl ines are deviations between .24 .02 -.04 .48 .76 -.ll Standard deviations for and 33 of 56 are < .90 -.24 .05 < .21 -.OOl .59 .25** 6.31 .41 .69 -.07 -.05 .53 .

40 .53 -.55 -.13”” “ Using Arguing an ambiguous to accept term an unproven conclusion 3.49 -.35 -.31 -.90*** 24.92 base rate 2.63 .61 .64 -.49*** Offering irrelevant evidence 3.66 -.02 -.74 -.41 .24 .55”” “ 7.36 .86 -.34 -.20 -.44 .23 Making general ev i d ence izations from insufficient 3.62 .62 .23 -.22*** inappropriately interpreting text 3.14 Accepting Fai I ing to source assumptions evaluate with the questioning ity of a credibil 3.03 .82 -.37 .64 .09 -.18 -.48 -.47 .Table Mean Ratings of Seriousness of 2 Reasoning Errors by Disciplines Disci pl ine Computer Tota Variable I Chemistry (N=37) SC ience (N=43) Educat i on (N=42) Eng i neer (N=43) i ng Engl ish (N=44) Psychology (N=46) f (N=i55) Error Faillng to recognize and differences between 2.18 .52 -.65 .24 3.18 -.42 -.12 .46 -.61 .09 .32 8.40 -.38 .97** 11.27 -.28 .66 .07 -.45 -.34 16.39 -.21 -.93 -.41 .14 .54 9.24 -.60 -.66 -.05 conclusion 3.23 -.27*** 6.62 3.34 .53 .96** 2.31**” Failing to recognize a concealed Confusing coincidence with causation 3.23 5.lO .59*** sample population Failing Offering to account for irrelevant statements 3.83 .58 .30 .14 -.49 .20 3.45 -.42 .20 3.26 .59 -.20 .91 1.96 -.24 -.22 .51*** 15.22 -.

23 3.oo Engl ish (N=44) -.32*** 1. disciplines and 8 of are 15 are deviations between 1.05 ***p< .~’ Standard deviations for *p < . grand means given under “ Total.Table 2 (Continued) Discipline Computer Tota Variable Allowing Basing Fail ing anecdotal conclusions to recognize information on partial data in 3. totals Entries range under from .28 .50 .03 .72 .06 -.19 _F_ (N=255) 3.57 10. 33 -.2.48. 64 -.74 .33 -.Ol **p < .42 .08 Sc fence (N=43) -.46” ” I Chemistry (N=37) -.36 Education (N=42) .98 individual to 1.Ol .51 Psychology (N=46) .OOl .16 simi larities analogies Note.17 .09 Engineering (N=43) .28 3.0 from and the 1.

03 .23 -.09 Making implausible assumptions .77 -.17 -.26 .98 3.42 .44 -.23 nonconstructive a nonresponsive Submitting .23 -.24 Accepting an argument with on emot iona “ hard I appeal 3.33 2.19 -.Table Mean Ratings of Effects of Critical 3 Incidents by Dlscipl lnes Discipline Computer Tota Variable Incidents 1 Chem i stry (N=37) Science (N=43) Ed ucat i on (N=42) Engineering (N=43) Engl (N=44) (N=255) Making Offering irrel evant remarks criticism paper 3.15 -.09 -.23 -.15 -.39 -.46 -.18 -.05 -.Ol .38 -.76 -.42 .23 -.ll -.67 -.11 -.22 -.40 -.33 -.oo Using Argulng inconsistent for the statements obvious . 14 Unable Falling to to perceive draw analogy conclusions breakdown about a pattern 2.23 2.18 .46 -.18 -.96 -.49 .35 -.17 -.12 .35 -.16 .39 3.46 -.41 3.12 -.34 3.28 -.79 3.05 -.24 -.32 .14 .23 -.12 -.22 -.45 .lO authorities Being Failing unable to to integrate that ideas evidence can 3.05 recognize conclusions support 3. 14 Conf usi ng anecdote data” Accepting uncritically conclusions of 3.29 .

93 individual to 1.56 .06 -.56 .23 Ignoring Unwilling Unable Unable to to detai to 3.37 .28 -.43 3.32 .04 -.I3 .44 .02 Fai I ing to qual ify conclusions 3.70 .04 .23 -.76” ” 1.76 .28 -.16 -F (N=255) 3.28 3..I6 .27.23 .16 -.01 3.ll .05 < .25 16.oo .a1 1.12 .94 .46”” “ Writing Searching a one-sided for essay solution 3.31 Note.11 -.19 25 -.12 -.07 -.17 -.001 .20 -.ia -. totals **p **.06 -.27 .06 Psychology (N=46) -.oo -.I7 .39 -.O and the 1.47 -.07 -.64 .30 -.15”” “ Unable to .14 -.38 .12 .47 -.23 -.‘ I Standard deviations for 15 of 25 are “ < .26 a.23 .05 .33 Engineering (N=43) Engl ish (N=44) -1.Ol p < .90 8.68 -.35 -.28 .Ol 4.02 .Table 3 (Continued) Discipline computer Tota Variable Applying rules without justification I Chemistry (N=37) Science (N=43) Ed ucat ion (N=42) -.19 a complicated Is respond .13 -.63 .19 -.28 -. discipl and ines are deviations between from 1 .ll -. Entries range under from .74 in an unknown hypotheses situation 3.11 .60 .30 .52 -.2.oa .ll .ia -.50 Preferring complex to simple explanations 3.99” ” 7.oo .99” Relying solely on narrative 3.04 1. grand means given under “ Total .72 -.54 generate .082 -.2a .06 3.15 -.23 suggest suggest a I ternat tests of i ves hypotheses 3.24”” “ 2.20 .14 -.61 Unable to see a pattern 3.

57) from Applying a formula.52) the conclusions consistent with.10) supported Failing to evaluate the credibility or reliability of a source or text (4.55) sufficient Making generalizations basis of insufficient (4. by. algorithm. the coincidence and/or with causation (4.17) Confusing correlation on the evidence Determining whether drawn are logically and adequately data (4.08) Generating new questions or experiments to extend or support the interpretation of data (4. other rule without sufficient justification (4.19) in are Being unable to integrate and synthesize ideas from various sources (3.77) Education Supporting conclusions with data or information (4.62) Reasoning or solving problems situations in which all facts underlying a problem situation known (4.10) . and Incidents Rated as or Critical by Disciplines Discipline Skills Errors/Incidents Chemistry Drawing sound inferences observations (4. Most Important 4 Errors.42) Computer Science Breaking down complex problems or situations into simpler ones (4.08) Being unable to generate hypotheses independently or Critically analyzing and evaluating previous research reports in a field (4.Table Reasoning Skills.46) or (4.

algorithm. and evaluating arguments (4.50) Drawing sound inferences observations (4.18) Recognizing the central thesis in a work (4.53) Identifying all the variables in a problem (4.50) Engineering from Breaking down complex problems or situations into simpler ones (4.82) Understanding.66) assertions with Relying solely on narrative or description in papers and reports when analysis is appropriate (4. other rule without sufficient justification (4.40) involved Applying a formula. analyzing.20) Supporting general details (4.75) Being unable to integrate and synthesize ideas from various sources (4.Table 4 (Continued) Discipline Skills Errors/Incidents Clearly identifying central issues and problems to be investigated or hypotheses to be tested (4.27) Accepting the central assumptions in an argument without questioning them (4.60) Reasoning or solving problems in situations in which all the needed information is not known (4.61) argument or .09) or English Elaborating an argument and developing its implications (4.

Numbers in parentheses each discipline.52) Confusing correlation coincidence and/or with causation (4. error.58) Clearly identifying central issues and problems to be investigated or hypotheses to be tested (4.Table 4 (Continued) Discipline Skills Errors/Incidents Psychology Critically analyzing and evaluating previous research or reports in a field (4.20) Note.57) Determining whether the conclusions drawn are logically consistent with. are the average ratings for each skill. or incident for .43) Accepting the central assumptions in an argument without questioning them (4. and adequately supported by. the data (4.

71 Deducing new information relationships Recognizing structural similarities between one type of problem or theory and another Taking well-known principles and ideas from one area and applying them to a different specialty Monitoring one’ s own progress in solving problems Deriving from the study of single cases structural features or functional principles that can be applied to other cases Making explicit all relevant of logical reasoning Testing the validity for counterexamples components in a chain 3. no significant differences among disciplines with respect to the average importance of these skills.92 from a set of 3.57 of an argument by searching Note.-29- Table 5 Reasoning Skills Rated Consistently Moderately Important As At Least Skill Reasoning or solving problems in situations in which all the needed information is not known Detecting fallacies in arguments and logical contradictions Mean Rating 4. Moderately important is defined as having an average c There were rating over all disciplines of 3.57 3.24 3.68 3.83 3. .75 3.5 or greater.86 3.

-3o- Table 6 Errors or Incidents Rated Consistently as at Least Moderately Serious or Critical Error/Incident Accepting the central assumptions in an argument without questioning them Being unable to integrate and synthesize ideas from various sources Being unable to generate hypotheses independently Being unable to see a pattern generalize when appropriate Ignoring details desired result in results or to Mean Rating 3. Moderately critical is defined as having an average c rating over all disciplines of 3.5 or greater.74 Submitting a paper that assigned issues to address the 3.57 Note. .76 that contradict failed an expected or 3.96 3.96 3.94 3.67 Basing conclusions on analysis a text or data set of only part of 3. There were no significant differences among disciplines with respect to the average ratings of seriousness or criticality.

and adequately supported by.-31Table 7 Summaryof Variables Defining Factors Underlying Ratings of Importance of Reasoning Skills Factor I Variables Loading Highest on Factors (73) (72) (70) (67) (64) (64) argument or thesis in a work (63) Recognizing shifts in the meaning of a word in the course of an argument Elaborating an argument and developing its implications Recognizing supporting points in an argument Recognizing the historical Understanding. the data Comparing conclusions with what is already Considering alternative III conclusions known w3) (67) (5% Setting up formal models for problems under consideration Breaking down complex problems or situations simpler ones into (69) (59) (57) (56) (53) Translating graphs or symbolic statements into words and vice versa Identifying all the variables involved in a problem Knowing the rules of formal logic . context of a problem and evaluating arguments Judging whether a thesis supported Recognizing the central II Generating valid observations has been adequately explanations to account for (72) data or (6% (69) Supporting conclusions with sufficient information Drawing sound inferences from observations Determining whether the conclusions drawn are logically consistent with. analyzing.

-32- Table 7 (Continued) Factor IV Variables Loadincr Hichest on Factors (61) (53) Recognizing structural similarities between one type of problem or theory and another Drawing distinctions identical ideas Synthesizing position between similar positions but not into a third two different (44) Deriving from the study of single features or functional principles applied to other cases V Finding alternative Generating alternative explanations hypotheses cases structural that can be (41) for observations (53) (46) Only the variables that best characterize the factors are Note. . Loadings are given in parentheses for the factor on which given. the variable’ s loading was most predominant.

-33Table 8 Summaryof Variables Defining Factors Underlying Ratings of Seriousness of Reasoning Errors Factor I Offering Variables irrelevant Loadina Hiuhest on Factors (75) evidence to support a point on the basis of insufficent Making generalizations evidence Failing to evaluate of a source or text (72) the credibility or reliability (71) Accepting the central assumptions in an argument without questioning them II Failing to take into account the base rate for a phenomenonin a population Failing to recognize differences and a population of interest between a sample (70) (76) (68) Offering irrelevant statements about a person’ s character or circumstances to oppose his/her conclusion Confusing coincidence and/or correlation causation with 63) (61) Note. Only the variables that best character ize the factors are given. . Loadings are given in parentheses for the factor on which the variable’ s loading was most predominant.

Loadings are given in parentheses for the factor on which the variable's loading was most predominant. algorithm.Table 9 Summary Variables Defining Factors Underlying of Ratings of the Effects of Critical Incidents Factor I Variables Loadinq Hiqhest on Factors Accepting or supporting an argument based more on emotional appeal than on evidence Offering criticism of other students' presentations that was not constructive or well founded Confusing anecdote and/or opinion with "hard data" Submitting a paper that failed to address the assigned issues (72) (65) (65) (62) II Accepting the conclusions of recognized authorities without critically evaluating them Being able to criticize better alternatives but unable to suggest (71) (59) (58) (56) (55) Being unable to integrate and synthesize ideas from various sources Being unable to generate hypotheses independently Failing to recognize that evidence can support more than one conclusion III Applying a formula. or other rule without sufficient justification Searching for a complicated solution when an obvious simple one exists Being unwilling to respond in a new or unknown situation when the limits of a student's knowledge has been reached (67) (58) (54) Note. only the variables that best characterize the factors are given. .

-35Table 10 Means of Scales Based on Items that Best Defined Factors by Disciplines

Discipline Chemistry Computer Science Education Engineering English Psychology Total
Key.

1 3.34 2.97 3.83 3.25 4.53 3.73 3.61

2 4.29 3.37 4.29 4.11 4.10 4.26 4.07

Scale 3 3.57 4.05 3.44 4.00 2.70 3.37 3".52

4 3.38 3.58 3.73 3.61 3.87 3.78 3.66

5 3.43 2.90 3.93 3.44 3.46 4.21 3.56

Scales are unweighted composites of the following variables: Scale 1 = Recognizing shifts in the meaning of a word in the course of an argument Elaborating an argument and developing its implications Recognizing supporting points in an argument Recognizing the historical context of a problem Judging whether a thesis has been adequately supported Recognizing the central argument or thesis in a work Understanding, analyzing, and evaluating arguments Scale 2 = Generating valid explanations to account for observations Drawing sound inferences from observations Supporting conclusions with sufficient data or information Determining whether the conclusions drawn are logically consistent with, and adequately supported by, the data Comparing conclusions with what is already known Considering alternative conclusions Revising a previously held view to account for new observations Scale 3 = Setting up formal models for problems under consideration Breaking down complex problems or situations into simpler ones Translating graphs or symbolic statements into words and vice versa Identifying all the variables involved in a problem Knowing the rules of formal logic Scale 4 = Recognizing structural similarities between one type of problem or theory and another Drawing distinctions between similar but not identical ideas Deriving from the study of single cases structural features or functional principles that can be applied to other cases Synthesizing two different positions into a third position Deriving general or abstract principles from disparate facts or cases Scale 5 = Finding alternative explanations for observations Being sensitive to the strength of different types of evidence (correlational, causal, testimony) Generating alternative hypotheses Recognizing the probabilistic nature of most events Note. Means have been divided by the number of items on each scale.

Appendix A Questionnaire

I.
The followlng are checkllsts program. often of sune general Please rate and specific

REASONING SKILLS that may be Important to success

reasonlng to:

skills

In your graduate

these skills

wfth respect

( 1) frequency-how
(2) Importance for

do students

1n your graduate does thfs

program need to use this skfll dffferentfate

ski I I? and successful students

success-to

what extent

between marglnal

in your progran? success,t8 please refer to these descrlptlons:

For ftlmportance

for

1. 2. 3. 4. 5.

lhls There There There There

skill

Js not relevant

to my ffeld

of teaching and successful students with respect to this skill to this skrll to thfs sklll skill

Is little

or no difference important Important

between marglnal difference difference

Is a moderately Is a very 1s a crItIcally

between margfnal betneen margfnal

and successful and successful

students with students

wfth respect respect wlth respect

Important

difference

between marglnal

and successful

students

to thls

Frequency

ImDortance for

Success/d

f f f erence

rbt Never/
A. Reasonfng Sk1 I Is--General Hardly Ever

Vet-Y
Frequent I y

Relevant Llttle

Wet-ate

Q-ltlcal Important

Derlvlng dlsparate

general facts

or abstract or cases.

principles

frun

. . . . . . . . . . fn sltuatfons Jn

. .

1

2

3

4

5

1

2

3

4

5

Reasoning or problan which all facts

solvlng

underlylng .

a problem solution

are known expllcltly.

. . . . . . . . . . .
In sltuatlons In

. .

1

2

3

4

5

1

2

3

4

5

Reasoning or problem solvfng which all

the needed InformatIon

is not known are present in

. .

1

2

3

4

5

1

2

3

4

5

Reasonlng when lnconsistencfes the fnformatlon

. . . . . . . . . . . . . . . of formal logic . . . . . .

. . . .

1

2 2

3 3

4 4

5 5

1 1

2 2

3 3

4 4

5 5

Knowfng the rules

1

Taklng wel I known prlncfples area and applying Being able posftlon. CYltfcally research

and Ideas fran one specialty a given

them to a different

. .

1

2

3

4

5

1

2

3

4

5

to both support and refute

. . . . . . . . . . . . . . . . . . . analyzing or reports and evaluating In a field. Instances prevlous

. .

1

2

3

4

5

1

2

3

4

5

. . . . . . . . or data Into a

. .

1

2

3

4

5

1

2

3

4

5

lnccrporatlng preexlstlng Mftorlng

Isolated fr wk.............

. .
Jn solving problems .

1 1

2 2

3 3

4 4

5 5

1 1

2 2

3 3

4 4

5 5

one’ s

own progress

. . . . . . Generatjng alternatlve hypotheses . . . that can be 1 2 3 4 5 1 2 3 4 5 Ever VSY Freauentlv WeI evant LIttIe Wderate O-Hlcal Important appl led to other ReccgnlzIng wents. . . . . 1 1 2 2 3 4 4 5 5 1 2 2 3 4 4 5 Other (specIfyI Skills-+roblem central DefInftlon Issues and problems to . . . . . . . . . . . . . . . . . . . . . . . . 2 1 5 1 2 3 4 5 1 2 3 4 5 Break!ng slmplm down ccmplex problems or sftuatlons ones. . . . . f?ecognIzlng structural sfmflarltles between one . . deflnStIons 1 2 3 4 5 1 2 3 4 5 type of problem or theory Developing of concepts Sel-tlng operatlonal (cr and another 1 2 3 4 5 1 2 3 4 5 very preclse) . . Speclflc Clearly ldentlfylng be lnvestlgated or hypotheses to be tested. . . . . . . . . . . . nature of most the probabl I IstIc . . . . . . . . . analyzing. . . Deducing new InformatIon relatIonshIps . . . .-2- Freauencv Importance for Success/d 1f f erence r&t Navel-/ Hardlv Derlvlng features frcm the study of sSngle cases structural or functional cases. . . . . . . . . . . ldentffylng more than one approach to solving a 1 2 3 4 5 1 2 3 4 5 Problem . . . . . . . . . . . . the hlstorlcal context . . . . . . . . . . . . . . . . . . . . . . . . all Into 1 2 3 4 5 1 2 3 4 5 . . 3 1 3 5 B. 1 1 2 2 3 3 4 4 5 5 1 1 2 2 3 3 4 4 5 5 Understanding. . 1 2 3 4 4 4 5 1 2 3 4 5 of a problem 1 1 2 2 3 3 5 5 1 1 2 2 3 3 4 4 5 5 (specify) . . . . 1 1 2 3 3 4 4 5 5 1 2 2 3 3 4 4 5 F@cognlzlng two or more sides of an Sssue . . . . . . . . . . . . 1 2 3 4 5 1 2 3 4 5 Knowing what klnd of evidence refute a hypothesis or thesis WI I I suppoct or . . . . . . . . . . . . problems under 1 2 3 4 5 1 2 3 4 5 up formal models for co&deratJon RecognIzlng Other . . . . . . . . . . . . . . prrnclples . . . . . . . and evaluating frcm a set of arguments. . . . the variables involved In a Identlfylng Problem . .

. . . .. . frcm observations. . . . . Identlfylng both stated and unstated . . . . . . . ... . Speclflc Sklsts--&struct~ng Theses or Argunents Supporting bklng of general all assertlons relevant with details.. . . . . assumptions. .. . . . . .Freauencv Importance for Success/d I f ference r+kYt New-/ I-Wdly Ever VWY Frequently Relevant Lfttle Met-ate CrItIcal Important c. . . 1s Internally . . . . . .. .. . . Its 1 2 3 4 5 1 2 3 4 5 an argument and developfng . .. . . . . . . . .. statements Into words and vice Other (specIfyI . conslstent. . . . relatfonships among observations . . appropriately . . . .. . . . . . . . .. Speclffc Skills-Analyzing Recognlzlng the central argument or thesis In 1 1 2 2 3 4 4 5 1 2 2 3 3 4 5 5 awork. . . .. . . . . . . . . . . . . . . . . .. 1 2 3 4 5 1 2 3 4 5 . Flndlng alternatlve 1 1 3 3 5 5 2 2 4 4 5 5 Recognlzlng Detecting suppcrtlng fal lacfes and logical In arguments. . . . . . 1 2 2 3 3 4 4 5 5 1 1 2 3 3 4 5 5 1 Arguments 2 4 D.. . . . 1 1 2 2 3 3 3 4 4 5 5 1 1 2 2 3 3 4 4 5 5 D-awing sound Inferences Producing an argument that 1 2 4 5 1 2 3 4 5 Syntheslzlng posItJon.. . . . . . and Irrelevant D!stfngufshlng lnfmt!on Percelvlng between relevant . . . . . .. . . . . . . . . . . . . . . .. . . . .. 3 5 1 4 Testing for the va1ldH-y of an argunent by seat-chlng 1 2 2 2 3 4 4 4 5 1 1 1 2 3 3 3 4 5 counterexamp 18s . . . explanations po!nts for observations In an argument. . . 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 Drawing dlstlnctlons ldentlcal Ideas betwean slmllar but not 1 2 3 4 5 1 2 3 4 5 . . . . Using analogles Elaborating ImpI IcatIons.. . .. Translating two dlfferent posStfons Into a thlrd 1 2 3 4 5 1 2 3 4 5 . . 1 2 3 4 5 1 2 3 4 5 expllcft ccmponents In a chain 1 2 3 4 5 1 2 3 4 5 logIca reasoning. . . . . . . . . . . . . . . . . . . . . graphs or symbolic versa. . . .. . contradlctlcns . . . . .

. . . . . . . . . . . necessary and 1 2 2 3 4 5 5 1 1 2 2 3 4 4 5 factors. . . . . . . whether the conclusions consistent by. . . . . . . . . Judging whether a thesls supported Determlnlng loglcally supposed has been adequate1 y 1 2 3 4 5 1 2 3 4 5 . . . . . . . . Supportlng InformatIon F&cognlzlng lnterpt-etatlon. . 1 2 3 4 5 1 2 3 4 5 Dlstlngulshlng sufflclent among contt-lbutlng. . . . . . . . . . . with sufflclent data or 1 2 3 4 5 1 2 3 4 5 conclusions . and adequately . . . known. . . . . . . . . . . . . . . . . . . . . . . . . . . . shifts causal. 1 2 2 3 4 4 5 5 1 1 2 2 3 4 4 5 5 1 3 3 . . . . . . . . . . . . types . . .Freauencv Importance Not for Success/d 1f f erence #3ver/ Hardly Ever very Frequent I y Fzleevant I Little Wderate cf-ltlcal Important Dlst1ngulshlng argument. . Other (specIfyI E. . . . . . . . . . . . . . . . 1 2 3 4 5 1 2 3 4 5 ways in which conclusions could be challenged GeneratIng CN. . . Other (specify) . . . . . . . . . new Cornpat-lng concIuslons Revising a previously what is already 1 held view to account for observations. . . . @al lfylng conclusions appropriately and recognlzlng .support Oonslderlng new questions the or experiments of data to extend . . . . . . 1 2 3 4 5 1 2 3 4 5 In the meaning of a word ln the 1 2 3 4 5 1 2 3 4 5 course of an argument . . . . 1 2 3 4 5 1 2 3 4 5 or consequences of an 1 2 3 4 5 1 2 3 4 5 . . . . . . . . major and minor points In an 1 2 3 4 5 1 2 3 4 5 . . . to the strength of djfferent testlmony). . . . . . . . . 1 1 1 2 2 2 3 3 3 4 4 4 5 5 5 1 1 2 2 2 lnterpretatlon alternative concIuslons with . . . . . . . . . . Speclflc Generatlng observations. . . . . Eielng sensltlve of evidence Recognlzlng (correlational. . . . . . . data. . . . . . . Skills-Bawlng valld explanations Conclusions to account for 1 3 4 3 5 . . . . . . . the drawn are wlth. . the lmpl lcatlons .

. Offering . assunptions in an argunent . . . . . evidence. .. . . . . . differences between a smple and a 1 2 3 4 5 population Failing . . . . . . . . . . . . . . ... . . without 1 2 3 4 5 1 2 3 4 5 has not been disproved Accepting questioning the central them. . . (rejected) . .. .. . . . . . . . . . . . . . .. 1 2 3 3 3 4 5 1 1 2 2 3 3 4 4 5 5 1 1 2 2 4 4 5 5 Making general izations on the basis of insufficient 1 2 3 4 5 Reading into a text text one’ s own views or inappropriately own experience. . . . . . .. . . . . . . . . . . Confusing coincidence and/or correlation with causation .. the credibility or reliability of a source Fail ing to evaluate or text. . .. . . .. . . . . . . . . . . . . . . . . . . . . . . 1 2 4 5 Using an ambiguous term to shift senses in the sane argument. . .. . .-5- I I. . . . . .. . . . .... . . . . Frequency !Seriousness Never/ Error Hard I y Ever V'3t-Y Frequent IW SeriOUS very !ht-iOUS Failing to recognize of interest. .. for a phenanenon to take into account the base rate in a population Offering . . .. .. . . . . . . . . 1 2 3 4 5 Fail ing to recognize when a conclusion is concealed within a 1 2 3 3 4 5 premise .. ... . . . . . 2 3 4 5 1 2 3 4 5 Other Other (specify) (specify) 1 1 2 2 3 4 5 5 1 2 3 4 5 3 4 1 2 3 4 5 . . . . . . . . Failing to recognize 1 relevant similarities and dissimilarities 1 2 3 4 5 1 2 3 4 5 when using analogies to argue . . statements about a person’ s conclusion.... . . . . . . . . . . . . .. . . . . . on analysis of only part of a text or data Basing ccnclusions set. . . .. character or 1 2 3 4 5 irrelevant circumstances to oppose his/her . information to override more extensive 1 2 3 4 5 1 2 3 4 5 . . . . . . 1 2 3 4 5 1 2 3 4 5 interpreting or data on the basis of one’ s Al lowing anecdotal statistical data. ... . .. .. . ... . .. . . 1 2 3 4 5 Arguing that because it a conclusion should be accepted (proved). . . . irrelevant evidence to support a point . . . . . .. . . . . . . . Please indicate with which of the following kinds of reasoning REASONINGEMms errors you have observed in your graduate students and the frequency which you have noted these errors. . . . . .. . . . . . . .

. .. . .. . 1 2 3 4 5 1 2 3 4 5 to recognize (or falllng to qualify a concIuslon approprlately). . . . .. . . . . . . . . . . l-bw frequently have you observed the followfng of the analytlcal klnds of ablllty CRITICAL Incidents IICIDENTS In your students? what effect have or tendencies these had on your estlmatlon of your students? New-/ Incidents Repeatedly Offerlng maklng Irrelevant crltlclsm of other remarks during studentst class dlscusslons that was 1 1 2 3 4 4 . Falling to recognize that evidence can suppcrt more than one 1 2 3 4 5 1 2 3 4 5 ccKIcluslon. . . . assumptions. . .. . . .. . . . . . . . . . . statements to support the same posltlon. .. . . . . .. that a deflnlte conclusion cannot be . . . . . . . . wlthout them. . . .. . . .III. . . . .. .. . . . . . . wlthout sufflclent 1 2 3 4 5 1 2 3 4 5 justlflcatlon Relylng solely . . .. .. opln Ion wl th “hard data”. . . . . when 1 2 2 3 3 4 4 5 5 1 1 2 2 3 3 4 4 5 5 ccmplex or far-fetched 1 Being unable to see a pattern appropriate Falllng reached In results or to generalize .. . . . . . . . . . .. . . .. . . . . . . . 1 2 3 4 5 1 2 3 4 5 MakIng lmplauslble Applylng a formula. 1 1 1 1 2 2 2 2 3 3 3 4 4 4 4 5 5 5 5 1 1 1 2 2 2 2 3 3 3 3 4 4 4 5 5 5 Uslng Inconslstent Arguing In support of the obvlous . . . . . . . 1 2 3 4 5 1 2 3 4 5 . .. . . . . . . . . . . algorlthm. . . .. . . . . .. . . 1 1 2 2 3 3 4 4 5 5 1 1 2 2 3 3 4 4 5 5 .. 5 1 2 2 3 4 5 Submlttlng a paper that falled to address the asslgned lssues .. . . or attemptIng to explain . . .. explanations over simpler ones. . . . . . authorltles . .. or descrlptlon In papers and reports 1 2 3 4 5 1 2 3 4 5 when analysis Referrlng .. . . .. . . . . . . . . . . . . .. . 2 3 5 1 3 4 5 Accepting or supportlng an argunent based more on emotional appeal than on evidence ConfusIng anecdote and/or Accepting crltlcally the concIuslons evaluating . . .. . . . . .. or other rule . . . . of recognized . . . . . . . . . . . . . . . . about Eelng unable to perceive Reoognlzlng lt a pattern when an analogy 3 1 4 5 but fal I Ing to draw any conclusions It. . . . . . . Hardly Ever VWY Frequent bb Ef feet Q-eat Ef feet 1 2 3 4 5 1 2 3 4 5 presentations not constructive or wel I founded. .. . . . . . . . . . . . breaks down. . . . . . . . . . . on narratlve 1s appropriate. . . . Ideas from varlous 1 2 3 4 5 1 2 3 4 5 Being unable to Integrate and synthesize sources. . . . . . . . . . .

Years of teaching experience at the undergraduate/graduate levels 6. . of a student’ s Being unable to generate EIelng able to crltlclze alternatIves.. . Ignoring . BACGROUM INFOlMATION 2. .. . . ... Instltutlon Department 5. . tests of hypotheses . . . . .-7- Never/ Hardlv Ever V’ V Freauent lb Ef feet Q-eat Effect Searching for a ccmpl lcated solution l when an obvlous simple one =lsts.g. Altogether. . .*. . . . . .. . when . result. . . .. Acadmlc rank (e.. Associate Professor) 3. Msters Doctoral about how many graduate students at each of the followlng levels have you advised over the past three years? Do you have any comments that you’ d llke to make about measuring students’ analytlcal abllltles? Thanks for your contrlbutlon to our study. . . ... . . . .. . . . . . . Your current teaching load (nunber of contact hours per week) undergraduate graduate 7. . . .. . . . but unable to suggest better . . . . . .*. . . . . details that contradld . 4. hypotheses Independently . . . . . . an expected cr desired Being unwllllng the I Imlts to respond In a new or unknown sltuatlon knowledge has been reached. Eblng unable to suggest appropriate Other (specl fy) Other W=lfy) Iv.

Appendix B Factor Analyses .

Appendix B.l Varimax Factor Loadings for Ratings of Importance of Reasoning Skills .

l of Importance of Reasoning Skills Factor Variable Reasoning Skills--General or abstract principles facts or cases I 11 III IV v Communality Deriving general from disparate 36 28 Reasoning or solving problems in situations in which all facts underlying a problem solution are known explicitly Reasoning or solving problems in situations in which all the needed information is not known Reasoning when inconsistencies the information Knowing the rules of formal are present in 34 48 27 38 35 31 32 53 38 41 38 logic Taking well-known principles and ideas from one area and applying them to a different specialty Being able position to both support and refute a given 51 31 16 40 Critically analyzing and evaluating research or reports in a field Incorporating isolated instances a preexisting framework Monitoring one's problems own progress previous 40 45 30 46 or data into 23 in solving 36 20 Deriving from the study of single cases structural features or functional principles that can be applied to other cases Recognizing events Understanding.Appendix Varimax Factor Loadings for Ratings B. arguments the probabilistic nature of most 41 21 32 25 analyzing and evaluating 64 49 Deducing new information relationships from a set of 33 33 32 Knowing what kind of evidence will support or refute a hypothesis or thesis 53 43 52 .

Appendix B.l (Continued) Factor Variable Snecific Skills--Problem Definition I 11 III IV v Communalitv Clearly identifying central issues and problems to be investigated or hypotheses to be tested Recognizing Generating two or more sides alternative of an issue 32 58 32 30 36 46 24 54 39 hypotheses or situations Breaking down complex problems into simpler ones Identifying problem all the variables 59 44 involved in a 32 56 43 Identifying more than one approach a problem Recognizing one type structural of problem to solving 31 36 39 similarities between or theory and another very precise) 30 48 61 47 Developing operational (or definitions of concepts Setting up formal consideration Recognizing problem 33 models for problems under 69 53 the historical context of a 67 55 Specific Skills or Arguments Supporting --Constructing Theses general assertions with details 57 40 52 Making explicit all relevant components in a chain of logical reasoning Distinguishing irrelevant Perceiving between relevant information and 40 40 42 36 51 58 37 46 52 relationships among observations between similar but not 46 35 Drawing distinctions identical ideas Using analogies 53 39 54 29 appropriately .

Appendix B.l (Continued) Factor Variable Elaborating an argument implications Drawing sound inferences that and developing its 72 64 69 56 I 11 III IV v Communalitv from observations is internally 51 Producing an argument consistent 43 49 Synthesizing two different third position positions into a 32 44 34 Translating graphs or symbolic into words and vice versa Specific Skills--Analyzing central statements 57 36 Arguments argument or thesis 63 34 53 Recognizing the in a work Identifying both stated assumptions and unstated 52 47 Testing the validity of an argument searching for counterexamples Finding alternative observations Recognizing explanations for by 36 44 33 46 38 53 53 64 supporting points in an argument contradictions 70 33 Detecting fallacies in arguments i and logical 58 32 54 Distinguishing argument major and minor points in an 53 43 Being sensitive to the strength of different types of evidence (correlational. testimony) Recognizing shifts in the meaning of a word in the course of an argument Judging whether supported a thesis has been adequately 41 35 42 52 73 65 64 41 61 . causal.

8.l (Continued) Factor Variable Determining whether the conclusions drawn are logically consistent with.30 have been omitted.37 3.Appendix B. and sufficient factors Specific Skills--Drawing necessary.51 25.84 3.21 2. . 43 35 34 49 I II III IV v Communality 36 68 64 Conclusions to account 72 59 Generating valid explanations for observations Supporting conclusions with sufficient or information Recognizing the implications of an interpretation data 31 69 58 or consequences ‘47 45 54 Qualifying conclusions appropriately and recognizing ways in which conclusions could be challenged Generating new questions or experiments extend or support the interpretation of data Considering alternative conclusions to 48 56 57 57 59 37 37 52 60 Comparing conclusions with what is already known Revising a previously held view to account for new observations Common Variance Note. Loadings less than . the data Distinguishing among contributing. and -adequately supported by.41 67 56 58 7.34 49 as have all decimal points.

2 Varimax Factor Loadings for Ratings of Frequency and Seriousness of Reasoning Errors .Appendix B.

Communality 74 56 75 58 Offering irrelevant statements about a person's character or circumstances to oppose his/her conclusion Failing to recognize when a conclusion concealed within a premise Confusing coincidence with causation and/or correlation is 48 40 39 51 40 43 30 53 38 Using an ambiguous term to shift in the same argument Arguing that (rejected) disproved senses 65 47 a conclusion should be accepted because it has not been (proved) 37 34 25 Accepting the central assumptions in an argument without questioning them Failing to evaluate the credibility reliability of a source or text Offering irrelevant a point evidence or 63 40 63 44 to support 67 47 Making generalizations on the basis insufficient evidence of 63 46 Reading into a text one's own views or inappropriately interpreting text or data on the basis of one‘s own experience Allowing anecdotal information more extensive statistical Basing conclusions on analysis of a text or data set Failing to recognize and dissimilarities to argue CommonVariance Note. . Loadings to override data of only part 70 55 42 53 46 58 30 42 relevant similarities when using analogies 50 40 42 less than .Appendix Varimax Factor Frequency B.60 as have all decimal 6.2 Ratings Errors of Loadings for of Reasoning Factor Error Failing to recognize differences between a sample and a population of interest Failing for to take into account the base rate a phenomenon in a population I IT.06 2. 4.66 points.30 have been omitted.

Appendix B.2 of Varimax Factor Loadings for Ratings Seriousness of Reasoning Errors Factor Error Failing to recognize differences between a sample and a population of interest Failing for to take into account the base rate a phenomenon in a population I II Communality 68 48 76 59 Offering irrelevant statements about a person's character or circumstances to oppose his/her conclusion Failing to recognize when a conclusion concealed within a premise Confusing coincidence with causation and/or correlation is 30 63 48 40 56 47 38 61 51 Using an ambiguous term to shift in the same argument Arguing that (rejected) disproved senses 50 48 48 a conclusion should be accepted because it has not been (proved) 35 60 48 Accepting the central assumptions in an argument without questioning them Failing to evaluate the credibility reliability of a source or text Offering irrelevant a point evidence or 70 53 71 56 to support 75 61 Making generalizations on the basis insufficient evidence of 72 56 Reading into a text one's own views or inappropriately interpreting text or data on the basis of one's own experience Allowing anecdotal information more extensive statistical Basing conclusions on analysis of a text or data set Failing to recognize and dissimilarities to argue to override data of only part 65 33 53 43 56 50 55 39 relevant similarities when using analogies 46 44 40 .

3 Varimax Factor Loadings of Frequency and Effects of Critical Incidents .Appendix B.

or other justification rule 48 34 48 53 40 53 Applying a formula. without sufficient 67 45 Relying solely on narrative or description in papers and reports when analysis is appropriate Preferring complex or far-fetched over simpler ones explanations 39 41 37 32 40 33 . 3 of Critical Incidents of Effects Factor Incident Repeatedly making irrelevant discussions Offering that criticism of other was not constructive that remarks during class 60 36 I I.Appendix Varimax Factor Loadings B.1 III Communality students' or well presentations founded the 65 44 Submitting a paper assigned issues failed to address 62 42 Accepting or supporting an argument based emotional appeal than on evidence Confusing anecdote and/or opinion with more on 72 60 52 "hard data" 65 Accepting the conclusions of recognized without critically evaluating them Being unable to integrate various sources Failing to recognize that than one conclusion Using inconsistent position Arguing in support and synthesize authorities 71 56 ideas from 58 40 evidence can support more 55 40 statements to support the same 56 40 39 51 37 of the obvious when an analogy breaks 38 Being unable down to perceive 49 38 47 Recognizing a pattern but failing to draw any conclusions about it or attempting to explain it Making implausible assumptions algorithm.

15 3.05 31 10. .66 37 3. Loadings less than . as have all decimal points.86 Note.Appendix B.3 (Continued) Factor Incident Being unable generalize to see a pattern when appropriate in results or to 40 42 40 I 11 III Communality Failing to recognize that a definite conclusion cannot be reached (or failing to qualify a conclusion appropriately) Writing a one-sided essay when a more balanced treatment is appropriate Searching for a complicated solution obvious simple one exists Ignoring details that desired result contradict when an 42 48 32 51 48 41 44 58 40 an expected or 34 40 39 42 Being unwilling to respond in a new or unknown situation when the limits of a student's knowledge have been reached Being unable to generate hypotheses but unable independently to suggest 59 56 54 35 39 44 Being able to criticize better alternatives Being unable to suggest hypotheses CommonVariance 40 appropriate tests of 41 4.30 have been omitted.

3 of Critical Incidents of Frequency Factor Incident Repeatedly making irrelevant discussions Offering that criticism of other was not constructive that remarks during class 52 31 I II III Communality students' or well presentations founded the 46 22 Submitting a paper assigned issues failed to address 37 36 27 Accepting or supporting an argument based emotional appeal than on evidence Confusing anecdote and/or opinion with more on 71 53 39 "hard data" 59 Accepting the conclusions of recognized without critically evaluating them Being unable to integrate various sources Failing than to recognize that one conclusion and synthesize authorities 41 47 39 ideas from 44 29 evidence can support more 40 36 35 Using inconsistent position Arguing in support statements to support the same 53 53 32 58 35 of the obvious when an analogy breaks 47 Being unable down to perceive 47 33 30 42 Recognizing a pattern but failing to draw any conclusions about it or attempting to explain it Making implausible assumptions algorithm. without sufficient 54 34 Relying solely on narrative or description in papers and reports when analysis is appropriate Preferring simpler complex or far-fetched ones explanations over 39 28 49 30 .Appendix Varimax Factor Loadings B. or other justification rule 32 42 50 28 42 Applying a formula.

75 Note.27 2. Loadings less than . .30 have been omitted.Appendix B.75 3. as have all decimal points.3 (Continued) Factor Incident Being unable generalize to see a pattern when appropriate in results or to 47 45 45 I II III Communality Failing to recognize that a definite conclusion cannot be reached (or failing to qualify a conclusion appropriately) 41 42 43 Writing a one-sided essay when a more balanced treatment is appropriate Searching for a complicated solution obvious simple one exists Ignoring details desired result that contradict when an 65 48 64 43 an expected or 39 32 41 42 Being unwilling to respond in a new or unknown situation when the limits of a student's knowledge have been reached Being unable to generate hypotheses but unable independently to suggest 35 48 72 36 36 54 Being able to criticize better alternatives alternatives Being unable to suggest hypotheses CommonVariance 52 39 appropriate tests of 68 3.72 52 9.

Sign up to vote on this title
UsefulNot useful