Thinking Skills and Creativity 3 (2008) 85 93

Contents lists available at ScienceDirect

Thinking Skills and Creativity

journal homepage: http://www.elsevier.com/locate/tsc

A comparison of a subject-specic and a general measure of critical thinking

Robert D. Renaud a,, Harry G. Murray b,1
a b

Department of Educational Administration, Foundations, and Psychology, University of Manitoba, Winnipeg, Manitoba, Canada R3T 2N2 Department of Psychology, University of Western Ontario, London, Ontario, Canada N6A 5C2

article
Article history:

info

abstract
The majority of studies reporting gains in college students critical thinking due to instruc tional process variables measured critical thinking with subject-specic questions rather than general or decontextualized questions. However, it is uncertain whether these gains were attributable to the use of subject-specic questions or to other distinc tive aspects of these studies (e.g., methodological). The present study provides a direct, controlled comparison between general and subject-specic test questions in the context of a laboratory-based true experiment assessing the effect of higher order review questions on gains in critical thinking. A stronger effect was found when the tests of critical thinking contained questions that were subject-specic (e.g., introductory psychology) rather than questions that focused on general topics. 2008 Elsevier Ltd. All rights reserved.

Received 1 August 2007 Received in revised form 18 March 2008 Accepted 26 March 2008 Available online 1 April 2008

Keywords:
Critical thinking Assessment Subject-specic General

1. Introduction

1.1. Importance in education

Critical thinking has become more important as a skill because individuals are faced with having to make an increasing number of important decisions affecting themselves and society in general (Halpern, 1996, 1998). In other words, a 30-yearold person would have to make more decisions today than a 30-year-old person would have needed to make 20 years ago. As the amount of information and decisions increase, we may be in danger of having the answers, but not knowing what they mean. Dressel and Mayhew (1954) outline four additional reasons for why critical thinking is important. First, critical thinking is valuable as it helps foster other important goals of attending college such as the development of moral and spiritual values, the transmission of knowledge, and the preparation of individuals for adult life. Second, critical thinking skills provide a purpose for acquiring knowledge. Otherwise, acquired knowledge simply becomes a jumble of facts. In other words, the development of critical thinking does not belittle the importance of gaining knowledge, but rather helps make the acquisition of knowledge more meaningful. Third, critical thinking ability is applicable to most activities and problems we encounter. Finally, while subject matter knowledge may be soon forgotten, critical thinking ability is a long lasting skill. Put another way, the absence of critical thinking will likely result in inferior or regrettable decisions (Gouran & Hirokawa, 1983). Beyond these more concrete reasons for why critical thinking is important, numerous national reports and researchers have recommended that instruction needs to have more emphasis on developing critical thinking skills (Facione, 1990; Halpern, 1996, 2001).

Corresponding author. Tel.: +1 204 474 6786; fax: +1 204 474 7564. E-mail addresses: renaudr@ms.umanitoba.ca (R.D. Renaud), murray@uwo.ca (H.G. Murray).
1

Tel.: +1 519 641 4367.

1871-1871/$ see front matter 2008 Elsevier Ltd. All rights reserved. doi:10.1016/j.tsc.2008.03.005

86

R.D. Renaud, H.G. Murray / Thinking Skills and Creativity 3 (2008) 85 93

1.2. Critical thinking: denition and assessment

Although most educators probably agree that critical thinking is an important cognitive skill that schools aim to develop in students, there appears to be a lack of agreement regarding a clear and operational denition of critical thinking ( Griggs, Jackson, Marek, & Christopher, 1998; Halpern, 2001; McMillan, 1987; Moseley et al., 2005; Nordvall & Braxton, 1996). At a broader level, critical thinking has been considered alongside creative thinking as related subordinate constructs within the broader level category of productive thinking, which is interpreted by Moseley et al. (2005) as comparable to the upper levels of Blooms taxonomy, namely analysis, synthesis and evaluation. One of the main distinctions within critical thinking is that between dispositions and skills. As outlined in the various critical thinking models reviewed by Moseley et al. (2005), critical thinking dispositions refer to prerequisite characteristics such as persistence and open-mindedness that reect ones inclination to apply critical thinking skills. Given that the focus of this study is on comparing the measurement of critical thinking skills in two formats, the following denitions pertain to skills rather than dispositions. Based on a review of 25 prior denitions, Griggs et al. provide a summary denition of critical thinking abilities as . . .a process of evaluating evidence for certain claims, determining whether presented conclusions logically follow from the evidence, and considering alternative explanations. Critical thinkers exhibit open-mindedness; tolerance of ambiguity; and a skeptical, questioning attitude. (pp. 256). Consistent with this conclusion, several popular denitions of critical thinki ng (e.g., Ennis, 1985; Furedy & Furedy, 1985; Pascarella & Terenzini, 2005; Watson & Glaser, 1980) contain the following ve common elements: identifying central issues and assumptions, making correct inferences from data, deducing conclusions from data provided, interpreting whether conclusions are warranted, and evaluating evidence or authority. Other elements of critical thinking include: making a statement or argument supported with evidence (Beyer, 1987), recognizing important relationships (Ennis, 1985; Furedy & Furedy, 1985; Pascarella & Terenzini, 2005), dening a problem (Dressel & Mayhew, 1954; Ennis, 1985), and forming relevant hypotheses (Dressel & Mayhew, 1954; Ennis, 1985). While this touches only briey on the concept of critical thinking, it seems that many of these elements could be likened to higher order levels of thinking, which attempt to explain how or why, as compared to lower order knowledge levels, which focus simply on what. As with the varied denitions of critical thinking, so too are the approaches used in its assessment. One issue regarding which type of items most validly measure critical thinking is whether or not the items reect an emotional component. For example, a question on the power of government would be suspected to evoke a stronger emotional component than a question on linear algebra. Dressel and Mayhew (1954) were among the rst to suggest that critical thinking test items should involve some degree of emotion. Subsequently, Watson and Glaser (1980) cite several studies showing that topics evoking strong attitudes or biases can affect ones ability to think critically. While an emotional component may serve to te st ones critical thinking ability under more extreme conditions, it might be difcult to incorporate an emotional component into a critical thinking measure within specic non-controversial subject areas such as geography or chemistry. As with other complex skills, there seems to be general agreement that critical thinking would best be measured in an open-ended essay format compared to a closed multiple-choice answer format (Dressel & Mayhew, 1954; McPeck, 1981). An essay format has the advantage of being more real-world like, in that the respondent can focus on the quality of the justication, and therefore can provide more than one justiable answer. However, two main limitations of essay questions are, rst, that they take longer to score and, second, scoring is less reliable compared to objective testing. Despite the advantages of measuring critical thinking with essay-type questions in tests such as the Test of Thematic Analysis (Winter & McClellend, 1978) and Psychological Critical Thinking Measure (Lawson, 1999), the closed-ended format found in more commonly used tests such as the Watson-Glaser Critical Thinking Appraisal (WGCTA) does allow for easier administration, more reliable scoring, and closer comparability, which is especially preferable in larger samples. Moreover, recent applications of sophisticated psychometric procedures such as item response theory can reduce the error component derived from using multiple-choice items to estimate critical thinking ability (Wagner & Harvey, 2006). Finally, one possible alternative that does not restrict students to closed-ended responses, but is easy to collect and score is the self-report format like that in the Chickering Critical Thinking Behaviors Inventory (Chickering, 1972). While there seems to be fair level of agreement regarding the advantages of emotional content and open-ended responses, it is not as clear whether critical thinking should be measured by generic, decontextualized test items or by items focusing on the same specic content or context, as in a course that intends to enhance critical thinking skills, such a s music history or educational psychology (Dressel & Mayhew, 1954; Garside, 1996; McMillan, 1987; McPeck, 1981; Pascarella & Terenzini, 2005). This contextual point of view assumes that one cannot engage in critical thinking without having the relevant knowledge base. For example, Dressel and Mayhew found that critical thinking in social science was associated with knowledge of social science terminology and content. Conversely, one of the most popular measures of critical thinking, the WGCTA attempts to measure critical thinking in a more general, decontextualized sense, in that it contains questions that focus on situations not concerning any particular course but, rather, everyday situations anyone might encounter, such as the weather or social issues.

1.3. Previous research

One particular area in which the question of whether critical thinking measures should focus either on general or subjectspecic content remains unclear is in the link between college processes an d critical thinking as an outcome. While much research has found that students increase their critical thinking skills as a result of attending college (see reviews by Adams,

R.D. Renaud, H.G. Murray / Thinking Skills and Creativity 3 (2008) 8593

87

Whitlow, Stover, & Johnson, 1996; McMillan, 1987; Pascarella & Terenzini, 2005), relatively little is known about what goes on during the college experience that contributes to this. Gains in critical thinking have been compared with several process variables pertaining to either the institution itself as a whole or a specic instructor. In general, while there is little evidence to suggest that any particular institutional variable is consistently linked with improved critical thinking skills measured either with general or subject-specic content, one particular instructional variable that may be worth further attention is the frequency of higher order questions. Recently, in a series of related studies with varied methods, Renaud and Murray (2007) found that frequency of higher order questions was related to gains in students critical thinking skills. While the Renaud and Murray studies included both general and subject-specic measure of critical thinking skills, these two formats were not compared with one another. Looking at general content, Terenzini, Springer, Pascarella, and Nora (1995) found that both class-related and out-of-class student experiences were related to scores on the critical thinking module of Collegiate Assessment of Academic Prociency. In a more controlled setting, Gadzella, Ginther, and Bryant (1996) found signicant pretest to posttest gains in critical thinking using the WGCTA for a single group of students enrolled in a course where students were given critical thinking problems. However, Gadzella et al. did not have an equivalent control group, which makes it dif cult to know how much the critical thinking problems actually contributed to observed gains. In contrast, several studies focusing on higher order questions asked by the instructor while teaching found little relation with gains in general critical thinking (Foster, 1983; Smith, 1977, 1980). For example, Smith (1977) correlated frequency of higher order questions asked by the instructor during class with gain scores on the WGCTA, and found that while critical thinking was enhanced by student participation, encouragement, and peer interaction, use of higher order questions by the instructor had little effect. However, a serious limitation of the Smith (1977) study was a severe restriction in variance in that questioning occurred in only 2.6% of the total class time. With almost no class time spent on asking questions, it would be difcult to determine if asking higher order questions really did have an impact on critical thinking skills. According to both McMillan (1987) and Pascarella and Terenzini (2005), gains in critical thinking are more likely to be found when critical thinking items focus on content relevant to a particular course, or when the content is consistent with a students academic major. Looking at questions asked in student tests or assignments, the aspect most studied appears to be the effect of having students write their own higher order test questions. The relation between students writing higher order questions and critical thinking was found to be positive when students created their questions either in class (King, 1989, 1990, 1995), or as an assignment out of class (Keeley, Ali, & Gebing, 1998; King, 1995). For example, King (1995) provided students with generic question stems such as What is the best. . .and why? and What is the difference between. . .and. . .? and had students generate higher order questions right after class in a peer questioning session or on assigned readings before class. In the King studies, critical thinking was measured either in the form of self-evaluation (King, 1995), subject-specic pre- and posttests of lecture comprehension (King, 1989, 1990), or pre- and posttest measures of the number of higher order questions generated from reading a short passage. Using a different design, Logan (1976) gave a 10-item critical thinking test to one group of students near the end of a one-semester course in which the professor analyzed material in a critical manner during lectures and to a second group at the beginning of the same course in the following semester. He found that the mean score for the rst group was higher than that for the second group. While Logans results are encouraging as the measure of critical thinking was course-specic, and thus, more sensitive, it is difcult to know just how much the students in the rst group actually increased their critical thinking skills without knowing their incoming levels. Looking at one particular instructional variable, Wesp and Montgomery (1998) measured gains in critical thinking with pre- and posttests of 300-word articles in which students were asked to nd and explain several aws. They found that compared to the control class, the class in which students were engaged in critical thinking exercises showed higher gains in critical thinking skills. More recently, Williams, Oliver, and Stockdale (2004) found larger pretestposttest gains in critical thinking skills among students in classes that were given practice exam questions that involved critical thinking compared to those in the control classes, which received practice exam questions on similar topics, but without the critical thinking focus. Conversely, Tierney, Soter, OFlahavan, and McGinley (1989) conducted an experiment in which students wrote an initial draft of a letter to the editor on an assigned controversial topic. Students in the experimental group were then given higher order questions to answer before revising their initial drafts, while control students were not. According to several criteria followed by trained judges, the quality of revised drafts from the group answering higher order questions did not differ appreciably from those of the group answering no questions. Using a stronger methodology, Garside (1996) compared the effects of group discussions in an interpersonal communications class on critical thinking skills. Controlling for pretest scores and prior preparation (i.e., assigned reading before class), students engaged in group discussions did not obtain signicantly higher scores on a critical thinking test focusing on interpersonal communication concepts than those from class sections that followed the lecture format. Pertinent to this study, there are several limitations that may have prevented researchers from nding a more consistent link between specic process variables and critical thinking that could help determine whether critical thinking should be measured either with general or subject-specic items. One concern has to do with ensuring that the processes believed to have an impact on critical thinking are experienced at the intended level. While Smith (1977) was able to determine the low frequency of higher order questioning during class, the level of treatment exposure in other studies is difcult to conrm (e.g., Garside, 1996). Another methodological limitation is that without basic design controls such as a pretest (e.g., Logan, 1976) or a control group (e.g., Gadzella et al., 1996), it is unclear if the signicant relation between instructional processes and critical thinking are due to the intended treatment or other confounding variables. While many studies have explored

88

R.D. Renaud, H.G. Murray / Thinking Skills and Creativity 3 (2008) 85 93

the impact of instructional processes on either general or subject-specic gains of critical thinking, two recent studies have directly compared the scores obtained from both types of tests. In a eld experiment that compared the pretestposttest gains of three different critical thinking tests across three sections of an educational psychology course, Williams (2003) found larger gains from the critical thinking test focusing on psychology compared to two general measures. Similar ndings were obtained in a later study (Williams et al., 2004). Although the methods of Williamss studies addressed many of the above limitations of previous studies, other threats to the internal validity of the ndings (e.g., contamination) were unavoidably present.

1.4. The present study

To address some of the methodological limitations in previous studies, the purpose of this research was to provide a direct, controlled comparison of general, noncontextual vs. subject-specic, contextual measures of critical thinking in a true experiment assessing the effect of higher order questions on gains in critical thinking. 2. Method

2.1. Participants
The participants in this study consisted of 190 undergraduate students registered in a rst -year introductory psychology course. Students who chose to participate in this study were given credit toward fullling the research participation require ment in this course. Before the experiment began, all participating students signed a consent form indicating that they were taking part in a study that compares study strategies. Participants were randomly assigned to either an experimental group or a control group with 96 and 94 students, respectively. To minimize the degree to which the data used in the analyses were contaminated with outlying values, each subjects data had to meet the following criteria for inclusion. Separate analyses were performed for each of the general and subject-specic critical thinking tests. Looking at the general test, there were three criteria for inclusion. First, because the maximum score on both the general pretest and posttest was 10 points, a subjects data were r emoved if he or she obtained a perfect score on the general pretest, thus precluding pretest-to-posttest gain. During the treatment phase of the experiment, to help ensure that a subject had put forth a reasonable effort toward reading the passage and answering the review questions that pertained to the passage, each subject had to obtain a minimum score of 3 out of 8. Although this criterion was applied to all subjects, it was intended more for those in the higher order group as an indication that they had engaged in a sufcient level of higher order thinking. Of the 26 subjects data who were taken out for failing to meet this criterion, 25 came from the higher order group. Finally, as an indication that a subject had put as much effort into answering the questions on the general posttest as on the pretest, and considering how a subjects scores could decline slightly from pretest to posttest because of factors other than critical thinking ability (e.g., the different questions in each versio n), a subjects data were removed if his or her posttest score was at least 4 points below his or her pretest score. Based on these three criteria, 157 subjects were retained for analyses on the general test with 66 and 91 subjects in the experimental and control conditions, respectively. With respect to the course-specic psychology portion of the critical thinking test, there were two further criteria for inclusion in the data analysis. First, as with the general subtest, each subject had to obtain a minimum score of 3 out of 8 on the review questions. Because the review exercise (i.e., treatment) was the same in both the general and psychology analyses, the number of subjects taken out from each group was the same. Regarding the level of student motivation in completing the posttest, it is possible that a student who is less motivated, and wanting to complete the experiment as quickly as possible, may be inclined to select answers without carefully reading the multiple-choice questions. With each of the 15 items in the psychology subtest having 4 options, the probability of someone obtaining up to 4 out 15 correct by picking answers at random is .69. Therefore, to help ensure that a posttest score was more indicative of a genuine effort rather than a random selection, and given that the students were expected to do reasonably well on the posttest as they had just completed the review exercise, the second criterion for inclusion was that a subjects postte st score had to be at least 5 out of 15. Based on these two criteria, 95 subjects were retained for analyses on the psychology test with 37 and 58 subjects in the experimental and control conditions, respectively.

2.2. Materials
All subjects were given a pretest and a parallel posttest measure of critical thinking, consisting of both a general subtest and a course-specic subtest. Each critical thinking test consists of 25 multiple-choice questions intended to measure the degree to which a student can engage in a particular aspect of critical thinking. Each item is followed by 4 or 5 options representing varying degrees of correctness or applicability from which students selected the most appropriate option. Scores on either test could range from a minimum of 0 to a maximum of 25. T o measure general critical thinking ability, the rst ten questions in the pretest and posttest were adapted from the Watson-Glaser Critical Thinking Appraisal (WGCTA). These questions focus on everyday situations that most people would likely be familiar with. To concur with the overall format of the WGCTA, the general items used in this study reected each of the ve main components of critical thinking covered by the WGCTA (i.e., inference, recognition of assumptions, deduction, interpretation, and evaluation of arguments), with two items reecting

R.D. Renaud, H.G. Murray / Thinking Skills and Creativity 3 (2008) 85 93

89

each component. Because many of the studies reporting signicant gains in college students critical thinking used domain or subject-specic measures (see review by McMillan, 1987), the remaining 15 questions focused on a selected passage from a chapter on personality theory from the introductory psychology textbook that was used in the course. For example, one item reads Which conclusion is most valid concerning research on the ve key dimensions of personality? The chapter on personality theory was chosen because it was not scheduled to be covered in class for at least a month after the completion of this study. With respect to estimating the reliability of the scores obtained from both the general and course-specic subtests, the typical reliability estimates, namely, internal consistency and parallel measures would not be applicable for at least two main reasons. First, the items within each subtest represent multiple dimensions. A principal-components analysis with Promax rotation revealed the following number of largely unrelated factors for each subtest: pretest-general (5), pretestpsychology (6), posttest-general (4) and posttest-psychology (7). Within each dimension or factor, there are too few items to obtain a meaningful estimate of internal consistency. Second, the reliability estimates of dichotomously scored items are inversely related to the range of item difculties (i.e., the proportion of students who answered a particular item correctly) (Nunnally & Bernstein, 1994). In this study, the item difculties ranged from .12 to .90. Additional concerns surrounding the inappropriateness of internal consistency and parallel measures reliabilities of a complex performance such as critical thinking are outlined in Linn, Baker, and Dunbar (1991) and Shavelson, Baxter, and Pine (1992). Between the pre- and posttest measures of critical thinking, all subjects were asked to read a passage on personality theory. This passage consisted of 9 textbook pages, which most students read within 25 min. Along with the assigned reading, each subject was given a set of review questions that pertained to the passage. The independent variable in this experiment was the level of review questions subjects answered as they read the passage. Subjects assigned to the experimental condition were given four higher order critical thinking questions, each of which was in a short answer essay format and answerable within half a page. Subjects in the control condition were given eight lower order recall questions, each of which was also in a short answer open format and answerable within a quarter of a page. The reason for assigning a greater number of lower order questions than higher order questions was to ensure that students in both groups were spending about the same length of time on the reading assignment before starting the posttest measure of critical thinking. It was suspected that the lower order questions would be easier and, therefore, take less time and space to answer compared to the higher order questions.

2.3. Procedure
Subjects were tested in groups ranging in size from 3 to 10. To ensure that experimental and control groups had roughly equal numbers of subjects, and that subjects in both groups were tested at the same times, half of the subjects in each testing session were randomly assigned to the experimental condition and half to the control condition. Before the experiment began, each participant signed a consent form indicating his or her understanding of the experiment and willingness to participate. To encourage students to put forth their best effort, students were told before the experiment began that anyone who scored higher than the predicted population average on both critical thinking tests and the chapter questions would receive a one-dollar lottery ticket at the end of the experiment. Actually, every participant had received a lottery ticket. The predicted population average was a ctitious goal that enabled the experimenter to justify giving every participant a lottery ticket more easily than would an absolute goal such as 50% correct. To begin, all subjects were given the critical thinking pretest, which most subjects completed in about 20 min. After all subjects had completed the pretest, they were given the assigned reading and review questions at the same time. Before the students began this phase, they were instructed to briey look over the questions before reading the passage so they could identify more readily which parts of the passage to pay close attention to in order to answer the questions. Most subjects took approximately 40 min to read the passage and answer all of the questions. After each subject had nished answering his or her review questions based on the assigned reading, they were given the critical thinking posttest. After completing the posttest, each subject received a debrieng form outlining the purpose of this experiment in detail and a lottery ticket. 3. Results

3.1. Review exercises

The mean scores on the review exercises were 7.7 out of 8 for the lower order question group and 4.4 out of 8 for the higher order question. This indicates that while most students in the lower order condit ion had little difculty in correctly answering the review questions, students in the higher order category were not as successful. However, it should be pointed out that the higher order review questions were clearly more difcult compared to those in the lower order condition.

3.2. General measure of critical thinking

The mean score for each group on the general pretest and posttest is shown in Fig. 1. The two groups showed virtually the same amount of gain with the control group going from (M = 5.95, S.D. = 1.81) on the pretest to (M = 6.45, S.D. = 1.91) on the posttest, while the experimental went from (M = 6.53, S.D. = 1.32) to (M = 7.02, S.D. = 1.76) on the pretest and posttest, respectively. Note that the higher mean score of the experimental group compared to the control group on the general

90

R.D. Renaud, H.G. Murray / Thinking Skills and Creativity 3 (2008) 85 93

Fig. 1. Mean pretestposttest critical thinking test scores (general).

pretest was primarily due to the majority of omitted subjects, who tended to score relatively low on this test, coming from the experimental group as explained earlier. Using the ANCOVA procedure with pretest scores as the covariate, there was no signicant difference between the groups posttest scores, with group membership accounting for only 1% of the total variance (2 = .010).

3.3. Subject-specic measure of critical thinking

The mean score for each group on the psychology pretest and posttest is shown in Fig. 2. The group receiving higher order review questions showed a larger gain from pretest (M = 5.38, S.D. = 1.59) to posttest (M = 6.46, S.D. = 1.45) than the group receiving lower order questions (M = 5.16, S.D. = 1.63) and (M = 5.79, S.D. = 0.91) for pretest and posttest, respectively. Using the ANCOVA procedure controlling for psychology pretest scores, the mean posttest score of the group that received higher order review questions was signicantly higher than that of the group that received the lower order review questions,

F(1,92) = 7.94, p < .01, and the effect of the treatment accounted for 7.9% of the total variance ( 2 = .079).

Fig. 2. Mean pretestposttest critical thinking test scores (psychology).

R.D. Renaud, H.G. Murray / Thinking Skills and Creativity 3 (2008) 85 93

91

4. Discussion The present study compared gains in students critical thinking measured with general and subject -specic questions. In agreement with McMillans (1987) review and in recent studies by Williams (2003) and Williams et al. (2004), larger pretestposttest gains were found from tests that contained questions that relate to the course in which the tests were given (e.g., introductory psychology) than tests with questions that focused on general topics. Perhaps one reason for this nding is that although a student is likely to be aware of the topics in general measures (e.g., politics), they may not be required as often to engage in the types of critical thinking in these everyday topics as they would in studying particular subjects (e.g., psychology). In other words, it may be that as instructors teach the skills that foster critical thinking, much of the focus is within a subject-specic context. In addition, the length of time between the pretest and posttest (i.e., the treatment) was only about 40 min and the subjects were rst-year students. Therefore, the opportunity to engage in a variety of learning experiences over a longer time span such as 34 years, ought to contribute to gains in critical thinking skills that may be more detectable with a general measure. In addition to being signicant both in statistical and practical terms, this nding is noteworthy for another reason. Like intelligence, it is not unreasonable to think that an apprecia ble gain in critical thinking skills would occur after having been exposed to a variety of courses and experiences over an extended period. Therefore, given that one could expect only a small effect under the restricted conditions of this experiment, namely minimal exposure to the independent variable in such a short time frame, these results are encouraging. Arguably, such a minimal exposure would lead to a narrower range of the degree to which subjects had experienced the treatment, which would tend to attenuate its relationship with the scores on the dependent variable (Prentice & Miller, 1992). On a related limitation concerning the short duration of the intervention, it becomes more difcult to determine the degree to which the students pretest and posttest scores actually reect the level of critical thinking that was inu enced by the treatment. Alternatively, given that the entire experiment (i.e., pretest, intervention, and posttest) lasted approximately only 90 min, it is possible that these ndings may be a function of other inuences such as the stu dents orientation to what was required on the assessments. In future research, it will be helpful to reduce the effects of these confounding inuences by assessing these skills more comprehensively (i.e., varied measures) and over a longer period. An ongoing concern in controlled experiments like this is the degree to which students are putting forth a genuine effort to complete the tasks involved in the experiment. The students who took part in this study did so to help fulll their research participation requirement in the introductory psychology course. This requirement is based solely on the number of research studies in which the student participates, and has nothing to do with the quality of the participation. As an incentive to make the students in this study try their best in completing the critical thinking tests and the review questions, each student was promised a lottery ticket if his or her scores were above a hypothetical standard on all three tasks (i.e., pretest, review questions, posttest). Admittedly, in this study, the effectiveness of this incentive is questionable for two reasons. First, although both groups showed a pretestposttest improvement on the course-specic psychology subtest, it was somewhat surprising that there was not a larger improvement given that the critical thinking tests referred to a relatively short passage that students had read through for about 45 min. Second, it is possible that the lower scores obtained on review questions by students in the higher order condition (4.35 out of 8, on average) may suggest that these students were not engaged in higher order thinking as much as was expected. One way to address the concern regarding the level of student motivation to try their best in each of the tasks in the experiment might be to provide a more valuable incentive. At the post-secondary level, one of the clearest, most immediate incentives is grades. Therefore, one possibility would be to include the experimental tasks as a small part of a course with a corresponding weight of the nal grade. To deal with the ethical issues involved, such an experiment would require a within-subjects design such that each student receives exactly the same materials from which he or she will be assigned a grade. For example, a study could take place during a class period with the pretest and posttest critical thinking measures focusing only on relevant course material. The review questions could consist of both lower order questions that pertain to one half of the passage (e.g., based on one chapter), and higher order questions that pertain to the rest (e.g., based on another chapter in the text). The ndings in this study suggest that gains in students critical thinking skills are more clearly detected with items focusing on specic course content rather than on general issues assumed to be familiar to a student in any discipline. Concerning research that examines the link between institutional and instructional processes and student outcomes, two implications are worth noting. While Pascarella and Terenzini (2005) conclude in their extensive review that controlling for incoming ability and maturatio nal effects, most studies found a signicant gain in critical thinking going from freshman to senior year, the format of assessing critical thinking used in these studies includes general, subjectspecic, and self-ratings. Thus, it is possible that some of the studies considered in their review that used general measures of critical thinking may have underestimated the extent of student gains. Using subject-specic measures of critical thinking may help colleges to better assess gains in students critical thinking skills both within depart ments (e.g., chemistry, economics) and entire institutions based on measures obtained from several departments. From another perspective, Halpern (2001) suggests that because critical thinking ought to be a skill that students should be able to use indenitely after graduation, and therefore should be transferable to novel contexts outside of a particular course, gains over a longer enrollment (e.g., from rst-year to fourth-year) might be more appropriately assessed with

92

R.D. Renaud, H.G. Murray / Thinking Skills and Creativity 3 (2008) 85 93

a general measure. Conversely, shorter term gains (e.g., one semester), might be more detectable with subject-specic measures. The second implication has to do with identifying valid institutional (e.g., campus tutoring programs) and instructional processes (e.g., higher order questioning in class) in terms of their relation with intended outcomes including gains in students critical thinking skills. Pascarella and Terenzini (2005) point out that while signicant gains in critical thinking have been found as a result of attending college, we are only beginning to learn what actually contributes toward this improvement. A more sensitive, course-based measure of critical thinking could help better determine the effectiveness of various educational processes. This would help to provide better empirical evidence concerning the factors that contribute toward student learning and development, which can be used to make more informed and justied choices both at the institutional level and within a particular course. Acknowledgement This study was supported in part by the Centre for Higher Education Research and Development, University of Manitoba, Winnipeg, Canada. References
Adams, M. H., Whitlow, J. F., Stover, L. M., & Johnson, K. W. (1996). Critical thinking as an educational outcome: An evaluation of current tools of measurement. Nurse Educator, 21(3), 2332. Beyer, B. (1987). Practical strategies for the teaching of thinking. Boston, MA: Allyn and Bacon. Chickering, A. (1972). Undergraduate academic experience. Journal of Educational Psychology, 63, 134143. Dressel, P. L., & Mayhew, L. B. (1954). General education: Explorations in evaluation. Washington, DC: American Council on Education. Ennis, R. H. (1985). A logical basis for measuring critical thinking skills. Educational Leadership, 43, 4448. Facione, P. A. (Ed.). (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. American Philosophical Association, ERIC ID 315 423. Foster, P. (1983). Verbal participation and outcomes in medical education: A study of third-year clinical-discussion groups. In C. Ellner & C. Barnes (Eds.), Studies in college teaching: Experimental results, theoretical interpretations and new perspectives . Lexington, MA: D.C. Heath. Furedy, C., & Furedy, J. (1985). Critical thinking: Toward research and dialogue. In J. Donald & A. Sullivan (Eds.), Using research to improve teaching (New Direction for Teaching and Learning No. 23). San Francisco: Jossey-Bass. Gadzella, B. M., Ginther, D. W., & Bryant, G. W. (1996). Teaching and learning critical thinking skills. In Paper presented at the International Congress of Psychology. Garside, C. (1996). Look whos talking: A comparison of lecture and group discussion teaching strategies in developing critical thinking skills. Communication Education, 45, 212227. Gouran, D. S., & Hirokawa, R. Y. (1983). The role of communication in decision-making groups: A functional perspective. In M. S. Mander (Ed.), Communications in transition. New York, NY: Praeger. Griggs, R. A., Jackson, S. L., Marek, P., & Christopher, A. N. (1998). Critical thinking in introductory psychology texts and supplements. Teaching of Psychology , 25, 254265. Halpern, D. F. (1996). Thought and knowledge: An introduction to critical thinking (3rd ed.). Mahwah, NJ: Erlbaum. Halpern, D. F. (1998). Teaching critical thinking for transfer across domains. American Psychologist, 53, 449455. Halpern, D. F. (2001). Assessing the effectiveness of critical thinking instruction. The Journal of General Education, 50, 270286. Keeley, S. M., Ali, R., & Gebing, T. (1998). Beyond the sponge model: Encouraging students questioning skills in abnormal ps ychology. Teaching of Psychology, 25, 270274. King, A. (1989). Effects of self-questioning training on college students comprehension of lectures. Contemporary Educational Psychology, 14, 116. King, A. (1990). Enhancing peer interaction and learning in the classroom through reciprocal questioning. American Educational Research Journal, 27, 664687. King, A. (1995). Inquiring minds really do want to know: Using questioning to teach critical thinking. Teaching of Psychology, 22, 1317. Lawson, T. J. (1999). Assessing psychological critical thinking as a learning outcome for psychology majors. Teaching of Psychology, 26, 207209. Linn, R. ,L., Baker, E. L., & Dunbar, S. B. (1991). Complex, performance-based assessment: Expectations and validation criteria. Educational Researcher, 20, 1521. Logan, G. H. (1976). Do sociologists teach students to think more critically? Teaching Sociology, 4, 2948. McMillan, J. H. (1987). Enhancing college students critical thinking: A review of studies. Research in Higher Education, 26, 329. McPeck, J. (1981). Critical thinking and education. New York, NY: Saint Martins Press. Moseley, D., Baumeld, V., Elliott, J., Gregson, M., Higgins, S., Miller, J., et al. (2005). Frameworks for thinking: A handbook for teaching and learning. Cambridge: Cambridge University Press. Nordvall, R. C., & Braxton, J. M. (1996). An alternative denition of quality of undergraduate education: Toward usable knowl edge for improvement. Journal of Higher Education, 67, 483497. Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill. Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research San Francisco: Jossey-Bass. Prentice, D. A., & Miller, D. T. (1992). When small effects are impressive. Psychological Bulletin, 112, 160164. Renaud, R. D., & Murray, H. G. (2007). The validity of higher-order questions as a process indicator of educational quality. Research in Higher Education, 48(3), 319351. Shavelson, R. J., Baxter, G. P., & Pine, J. (1992). Performance assessments: Political rhetoric and measurement reality. Educational Researcher, 21, 2227. Smith, D. G. (1977). College classroom interactions and critical thinking. Journal of Educational Psychology, 69, 180190. Smith, D. G. (1980). College instruction: Four empirical views. Instruction and outcomes in an undergraduate setting. In Paper presented at the Annual Meeting of the American Educational Research Association. Terenzini, P. T., Springer, L., Pascarella, E. T., & Nora, A. (1995). Inuences affecting the development of students critic al thinking skills. Research in Higher Education, 36, 2339. Tierney, R. J., Soter, A., OFlahavan, J. F., & McGinley, W. (1989). The effects of reading and writing on thinking criticall y. Reading Research Quarterly, 24, 134173. Wagner, T. A., & Harvey, R. J. (2006). Development of a new critical thinking test using item response theory. Psychological Assessment, 18(1), 100105.

R.D. Renaud, H.G. Murray / Thinking Skills and Creativity 3 (2008) 8593
Watson, G. B., & Glaser, E. M. (1980). Watson-Glaser critical thinking appraisal. San Antonio: The Psychological Corporation. Wesp, R., & Montgomery, K. (1998). Developing critical thinking through the study of paranormal phenomena. Teaching of Psychology, 25, 275278. Williams, R. L. (2003). Critical thinking as a predictor and outcome measure in a large undergraduate educational psychology course . (ERIC Document Reproduction Service No. ED 478 075). Williams, R. L., Oliver, R., & Stockdale, S. (2004). Psychological versus generic critical thinking as predictors and outcome measures in a large undergraduate human development course. Journal of General Education, 53, 3758. Winter, D., & McClellend, D. (1978). Thematic analysis: An empirically derived measure of the effects of liberal arts education. Journal of Educational Psychology, 70, 816.

93