THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY 2006, 59 (2), 411 – 425

The effect of contextual factors on the judgement of informal reasoning fallacies
Yair Neuman, Michael P. Weinstock, and Amnon Glasner
Ben-Gurion University of the Negev, Beer-Sheva, Israel

Informal reasoning fallacies are arguments that are psychologically persuasive but not valid. In order to judge the validity of these arguments one has to be sensitive to the context in which they appear. However, there is no empirical study that examines students’ sensitivity to contextual factors and whether contextual factors actually influence their ability to identify informal reasoning fallacies. We hypothesized that when explicitly presented with different argumentative contexts, students’ performance would reflect their sensitivity to the contextual nature of informal reasoning fallacies. The two experiments that we conducted support this hypothesis and emphasize the mediating role of perspective taking in students’ ability to identify fallacious arguments.

Informal reasoning may appear in a nonverbal form (e.g., through images). However, the observation that daily reasoning is verbally mediated motivated researchers to identify informal reasoning with “argumentation” (Baron, 1991; Kuhn, 1991; Means & Voss, 1996; Perkins, 1989). According to this suggestion, when reasoning informally an individual actually is involved in a deliberation process aimed at “increasing (or decreasing) the acceptability of a controversial standpoint . . . by putting forward a constellation of propositions intended to justify (or refute) the standpoint before a rational judge” (Eemeren, Grootendorst, & Henkemans, 1996, p. 5). For example, when asked to argue whether the death penalty reduces the murder rate, a person should put forward a set of reasons (theoretical and empirical) supporting her or his claim concerning this debated issue.

A major subfield of argumentation is the study of informal reasoning fallacies (Eemeren & Grootendorst, 1992; Eemeren et al., 1996; Walton, 1987, 1989): arguments that are “psychologically persuasive but logically incorrect; that do as a matter of fact persuade but, given certain argumentative standards, shouldn’t” (Copi & BurgessJackson, 1996, p. 97). For instance, the ad ignorantiam argument (the argument from ignorance) is an argument in which a conclusion about the truth (or the existence) of a proposition “A” is derived from the fact that “A” is not known to be false. For example, from the premise “It has never been proved that God does not exist”, we cannot conclude, “God exists”. In this example, the conclusion (“God exists”) is not derived soundly from the premise (“It has never been proved that God does not exist”), although most students may find no problem with it

Correspondence should be addressed to Yair Neuman, Department of Education, Ben-Gurion University of the Negev, Beer-Sheva 84105, Israel. Email: yneuman@bgumail.bgu.ac.il The authors would like to thank the anonymous reviewers for their helpful comments. # 2006 The Experimental Psychology Society http://www.psypress.com/qjep

411
DOI:10.1080/17470210500151436

NEUMAN, WEINSTOCK, GLASNER

(Neuman, 2003). We use the term “soundly” rather than deductively or inductively derived, because soundness is the criterion for judging the validity of arguments (Copi & Burgess-Jackson, 1996). In contrast with formal-deductive reasoning, there is no formal criterion for the “soundness” of an argument. A formal fallacy in domains such as mathematics, logic, or statistics can be detected by examining the form of the argument. In contrast, an informal fallacy cannot be detected by examining the form of an argument. Therefore, formal and informal fallacies are mutually exclusive. If a fallacy is formal, then it is not an informal fallacy and vice versa (Copi & Burgess-Jackson, 1996). Informal fallacies can be detected by examining either the context of the argument (i.e., the pragmatics of the argument) or the content of the argument (e.g., the meaning or vagueness of words). The example above illustrates the argument from ignorance fallacy because, in the context of critical discussion, our epistemological norm is that knowledge is usually derived from something, and it cannot be derived from ignorance. Recently it was argued by Oaksford and Hahn (2004) that the argument from ignorance can be viewed from a probabilistic-Bayesian perspective as a weak rather than unsound form of reasoning. Framing the argument from ignorance in a Bayesian perspective is a novel approach to this fallacy. It shifts the discussion from the pragmatic norms of argumentation to psychological processes underlying argumentation. In the current study, we suggest that people’s evaluations of the soundness or strength of an argument defined as fallacious will be influenced not just by how the reason bears on the probability of a claim, but the context in which the reason is given.

Informal reasoning fallacies and contextual factors
Context is crucial for any process of meaning making, and recently there is an increasing interest in the definition of this construct and its role in activities of meaning making. This interest in context is evident even in fields that have

traditionally ignored it such as artificial intelligence (Akman & Surav, 1996; Iwanska & Zadrozny, 1997; McCarthy, 1987). However, the term “context” is too general and problematic and therefore might have a low informative value (Burke, 2002). This difficulty should motivate researchers to define clearly what they mean by “context”. In argumentation, context may be operationally defined by using the dimensions suggested by Walton (1989). Walton proposed three dimensions that differentiate between different types of dialogue/argumentative contexts. These dimensions are (a) the initial situation that motivated the dialogue; (b) the method of dialoguing; and (c) the goal of the dialogue. For example, a quarrel is a type of dialogue, which is motivated by an emotional disquiet (i.e., the initial situation). The goal of each participant in a quarrel is to strike out at the other by using the method of personal attack. In this context, it is legitimate to use a personal attack against your adversary (i.e., the ad hominem argument) even though this argumentative move might have no relevance for the point one is trying to make. For example, during a quarrel one may describe one’s adversary as intellectually incompetent. Whereas this move might be irrelevant to determining the truth of the issue in debate, it is coherent with the context of the dialogue and, therefore, cannot be described as a fallacy. Using an informal reasoning “fallacy”, such as an attack against the person, also may be legitimate in the context of a debate, which is a different type of a dialogue. In the context of a debate, the initial situation is one of a forensic contest in which one is trying to impress the audience by trying to achieve verbal victory over one’s adversary. In contrast, it is illegitimate to use argumentative moves such as the ad hominem argument, in the context of a critical discussion, or reasoned dialogue. In a reasoned dialogue, the initial situation is of difference of opinion. In this context, the goal is to persuade the other by bringing a set of reasons that may support one’s claim and refute your adversary’s claim. Cases in which argumentative moves do not support the claim, however much they may

412

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

JUDGEMENT OF INFORMAL REASONING FALLACIES

have a rhetorical impact on the audience, may be considered as fallacies. In a series of studies, it was found that participants’ ability to identify informal reasoning fallacies is influenced by several factors such as the representation of the argument in the cognitive system (Neuman & Weizman, 2003), the truth value of the argument’s premises (Neuman, Glassner, & Weinstock, 2004), and students’ familiarity with norms of argumentation (Weinstock, Neuman, & Tabak, 2004). Although “context” plays a critical role in the definition and the identification of informal reasoning fallacies, there is no empirical study that investigates the role of context in students’ ability to identify informal reasoning fallacies. Therefore, it is not clear to what extent students are influenced by the argumentative context in their judgement of how relevant a reason is to a claim and how convincing they find a reason to be. An investigation into the argumentative context will add a social element missing from the body of research on informal reasoning fallacies: the element of persuasion. Recall that a defining characteristic of an informal reasoning fallacy is that it may be psychologically persuasive despite violating standards of argumentation. An informal argument should be judged on the quality of the reason supporting a claim. Ideally, one would find an argument more convincing based on its soundness and strength—that is, if the reason addressed the claim with pertinent, truthful, and meaningful information. However, the context of the argument might influence how convincing one finds an argument, either in addition to or at the expense of an evaluation of the argument’s soundness. For example, according to language expectancy theory (LET), whether an attempt at persuasion conforms to cultural and sociological expectations about language behaviours will affect how persuasive the communication will be (Burgoon, Denning, & Roberts, 2002). In addition, whether a fallacious argument is found persuasive or is criticized might depend on one’s role in the context of the argument. According to the theory of motivated reasoning

(see Klaczynski, 2000; Kunda, 1990), the directional goal of an argument—that is, whether one is supporting or opposing a claim—will influence the application of systematic reasoning and inferential rules in evaluating reasons for the claim. From this, it might be predicted that people who can recognize arguments as fallacious from a detached perspective will differ in their recognition of such arguments when asked to take sides. Thus, in addition to the context of the argument as reasoned dialogue or nonreasoned dialogue, we may include directional goals that would influence the acceptance of types of reasons. Moreover, reasoning might be motivated by the degree of involvement in the argument (Johnson & Eagly, 1990). In motivated reasoning research, the factor generally considered to bias reasoning is prior belief. Thus, in this line of research the directional goal of an argument was determined by one’s belief concerning a claim whether deeply held (e.g., Klaczynski, 2000) or manipulated (e.g., Jain & Maheswaran, 2000). The current study departs from this line of research in its focus on the argument contexts rather than personal beliefs. The experiments in this study investigate the effect of argumentative context on participants’ ability to identify informal reasoning fallacies and on their evaluation of the persuasiveness of fallacious arguments. The general hypothesis is that students’ judgements about the fallaciousness and persuasiveness of informal reasoning fallacies will vary significantly according to the argumentative context. The first experiment examines students’ abilities to identify fallacies in different contexts and to distinguish between the contexts. The second experiment focuses on the convincingness of fallacious arguments in the context of a reasoned dialogue and with the participants in different roles—the proponent of a claim, and the adversary—in the argument context.

EXPERIMENT 1
Our theoretical point of departure is that participants’ difficulties in identifying informal reasoning

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

413

NEUMAN, WEINSTOCK, GLASNER

fallacies may be explained partially by the lack of an explicit argumentative context. The context should provide the redundancy that is necessary for critically analysing the argument. In other words, an invalid argument may be more psychologically convincing if there is no explicit context that may support its critical analysis. If the context is explicitly presented as a context of a reasoned dialogue then participants’ performance in identifying the fallacies would be much better than it would be in a context of a nonreasoned dialogue. Thus, we hypothesized that students will recognize that the fallacies conflict with the goals of the reasoned dialogue and will be capable of handling the fallacies in this context.

the initial situation of the dialogue and the issue in debate. In lines 2– 3 the claims were presented, line 4 presented the goal of the dialogue, and line 5 the method of argumentation. Line 6 presented the fallacious argument. A general schematic structure of the scenarios appears in Appendix B. There were two scenarios (stories) for each fallacy and two versions for each scenario. One version corresponded to the context of a reasoned dialogue, and the other version corresponded to a context of a nonreasoned dialogue, whether a quarrel or a debate. For example, the UFOs scenario was presented once in the context of a quarrel and once in the context of a reasoned dialogue. The quarrel reads: 1. Yossi and Avi, who don’t like each other, are arguing about whether UFOs exist. 2. Yossi argues that UFOs exist. 3. Avi argues that UFOs do not exist. 4. The aim of each participant is to strike out at the other. 5. In order to achieve this aim they personally attack each other. 6. During the dialogue Yossi argues: “Avi, you argue that UFOs do not exist because you are a person with no imagination.” The same scenario appeared in the context of a reasoned dialogue: 1. Yossi and Avi disagree with regard to the question of whether UFOs exist. 2. Yossi argues that UFOs exist. 3. Avi argues that UFOs do not exist. 4. The aim of each participant is to convince the other to accept his claim. 5. In order to achieve this aim each one of them should bring reasons that justify his claim. 6. During the dialogue Yossi argues: “Avi, you argue that UFOs do not exist because you are a person with no imagination.” The questions. After reading the scenario, participants answered questions aimed at measuring their ability to identify the fallacy. The participants were randomly assigned one of three experimental tasks

Method
Participants A total of 71 eleventh graders from a regional secular high school participated in the study. A total of 48% of the participants were female, and the rest were male. The average age of the participants was 16.5 years. We chose high-school students as our participants because this population does not have a formal background in either formal or informal logic, yet it is has been shown to be capable enough to handle informal reasoning tasks (Neuman, 2003). Materials The informal reasoning fallacies identification tasks The scenarios. In this experiment, we used a variation of Neuman and Weizman’s (2003) informal reasoning tasks. Each participant received 12 informal reasoning tasks. Each task included an argumentative scenario followed by a question or questions. Each scenario corresponded to one of the three types of informal reasoning fallacy: (a) the ad hominem fallacy (argument against the person); (b) the ad populum fallacy (the appeal to people); and (c) the ad ignorantiam fallacy (the appeal to ignorance). Descriptions of all the fallacies that we used in this study appear in Appendix A. The structure of the scenarios was constant and represented the three dimensions of the argumentative context. The first line presented

414

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

JUDGEMENT OF INFORMAL REASONING FALLACIES

in which a goal of each was to identify the fallacy. In these subexperiments we used three different measures of students’ ability to identify the fallacy, in order to increase the external validity of our study. Approximately one third of the students (n ¼ 23) received a simple identification question of the form: Do you think that there is a problem with the argument raised by X (i.e., the proponent)? (For example: “Do you think that there is a problem with the argument raised by Yossi?”). In response, the students circled “yes” or “no”. The second third of the students (n ¼ 23), received a version in which they were asked the following two questions: 1. To what extent does the argument raised by X help achieve his or her aim? [For example: “To what extent does the argument raised by Yossi (‘Avi, you argue that UFOs do not exist because you are a person with no imagination’), help him to achieve his aim (to convince his colleague to accept his claim)?”] 2. To what extent does the reason used by X support the claim that Y? [For example: “To what extent does the reason raised by Yossi (‘you are a person with no imagination’) support his claim (‘that UFOs exist’)?”] These questions differentiate between the extent to which the argument helps to achieve the goal and the extent to which the reason supports the claim. Students were asked to rate their judgements by using a 5-point Likert scale ranging from 1 (“to a very minor extent”) to 5 (“to a great extent”). The third group of the students (n ¼ 25) was asked to answer the question: If you were Y, what would you have say in order to counter X’s argument? (For example: “If you were Avi, what would you have said in order to rebut Yossi’s argument?”) This question was an open-ended question that aimed to examine whether the

student would respond by pointing at the fallacy in the argument. The opinions item sheet. Prior opinions may be considered as an interfering variable in this study because they may influence students’ ability to identify the fallacy. For example, in Oaksford and Hahn (2004) it was found that prior beliefs influence the acceptance of the argument from ignorance. In order to control prior opinions, we evaluated students’ opinions concerning the issues that appear in the scenarios, by using an opinions item sheet. The sheet included six statements. Each statement expressed the proponent’s conclusion about the issue being debated. The students were asked to rate their agreement with each statement on a 7-point Likert scale. For example, students were asked to rate their agreement with the statement “UFOs exist”. The argumentation norms task. The argumentation norms task (Weinstock et al., 2004), presents students with a paragraph (see Appendix C) that defines a reasoned dialogue. The paragraph is followed by a forced-choice question in which students are asked whether they consider it justified in the context of reasoned dialogue to: 1. Support a position based on a majority per se (the argument from popularity fallacy). 2. Attack your adversary rather than attack his/her argument (the argument against the person fallacy). 3. Argue that a position that has not been discounted is necessarily true (the argument from ignorance fallacy). For each possible position students were to select a response from the choices: (a) yes, (b) no, or (c) I do not know. It was previously found (Weinstock et al., 2004) that students’ familiarity with norms of argumentation is associated with their performance in identifying informal reasoning fallacies. Therefore, we used the argumentation norm task in order to make sure that our participants were familiar with the argumentative norms.

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

415

NEUMAN, WEINSTOCK, GLASNER

Design In each group we used a within-subjects design, which is usually considered to be the most powerful research design. Each participant received a mix of scenarios in which each scenario was presented once in the context of a reasoned dialogue and once in the context of a nonreasoned dialogue. The order of the scenarios was random except that in no instance did a scenario presented in one context immediately follow the same scenario presented in the other context. The use of a within-subject design might be criticized as being susceptible to the artifact of order effect. In order to address this possibility, betweensubjects measures were used to compare cases in which the first scenario was a reasoned dialogue with cases in which it was a nonreasoned dialogue. In the second subexperiment, in which the participants went beyond identifying the fallacies and evaluated their effectiveness in the given context, we used the Kruskal –Wallis test to compare participants’ responses to the two questions concerning each of the 12 scenarios across the two different orders of presentation. None of the comparisons was found to be statistically significant.

After completing the task, the students received the opinion item sheet and the argumentation norm task. After the students started work on the tasks, the experimenter’s role was limited to questions of clarification during the session and a postsession debriefing.

Results
Before we analysed the data we examined students’ familiarity with the argumentative norms. It was found that 94% of the students rejected the legitimacy of using the ad populum argument, 94% rejected the legitimacy of using the ad hominem argument, and 89% rejected the legitimacy of using the ad ignorantiam argument in the context of a reasoned dialogue. These results indicate that the students were familiar, at least theoretically, with norms of argumentation that reject the use of fallacies in a particular context. Subexperiment 1 Table 1 presents the percentages of participants in the first group who answered “yes” to the question of whether they thought that there was a problem with the reason in support of a claim across contexts of reasoned and nonreasoned dialogue. We hypothesized that a higher percentage of students
Table 1. Percentages of students who claim that there is a problem with the argument by context Context Reasoned Scenario DRG UFO STU TAX DEA GOD Average % 83 83 52 74 96 65 76 No. 19 19 12 17 22 15 17 % 61 52 35 48 78 70 57 Nonreasoned No. 14 12 8 11 18 16 13

Procedure The experiment took place during a classroom session. Participants were asked to work individually on the informal reasoning task. Participants received the task of identifying the informal reasoning fallacy and worked 4 minutes on each informal reasoning scenario (48 minutes in total). All of the scenarios in the identification task appeared in written form. Each scenario appeared on a separate sheet of paper. The students received the following instructions: On the following pages, you will be presented with several scenarios. Each scenario presents a dialogue between two characters. Read each scenario and answer the question/s. Try to answer the question/s without any relation to your opinion of the issue in debate.

Note: N ¼ 23. God (GOD) and students (STU) are the ad ignorantiam scenarios. Death (DEA) and drugs (DRG) are the ad populum scenarios. TAX and UFO are the ad hominem scenarios.

416

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

JUDGEMENT OF INFORMAL REASONING FALLACIES

would respond that there was a problem with the argument in the context of a reasoned dialogue. It can be seen that in five out of six scenarios a higher percentage of participants in the reasoneddialogue context did argue that there was a problem with the argument with a difference that ranged from 17% to 31%. The one exception was the God scenario. This exception might be explained by the participants’ prior opinions about the topic. It was found that participants’ rating of the claim “God exists” was the highest among the other opinion items (M ¼ 5.59, SD ¼ 2.03). By using a binary logistic regression it was also found that prior opinion of this item significantly predicted students’ performance in the context of a reasoned dialogue, x2(1, N ¼ 23) ¼ 6.46, p ¼ .01, with a 65% correct classification rate. Prior opinion of the God item also significantly predicted performance in the context of a nonreasoned dialogue, x2(1, N ¼ 23) ¼ 6.62, p ¼ .01, with a 70% correct classification rate. In other words, the participants were highly supportive of the proponent’s claim that God exists, to start with, and their support significantly predicted their judgement of whether there is a problem with the argument. Those who expressed a stronger belief in the existence of God were less attentive to the problem with the argument, whether the argument was presented in the context of a reasoned or a nonreasoned dialogue. It is possible that this strong belief, which was not present in the other items, biased the participants’ judgement of the reason given in support of the claim. As performance on this issue was found to be sensitive to prior attitudes we ran a Sign Test without the responses to the God scenario. The Sign Test was used to find a significant pattern for the remaining scenarios (p ¼ .013), in each of which more participants argued that there was a problem with the argument. Subexperiment 2 Our hypotheses were that a participant’s identification of an argument as fallacious and judgement of the extent to which such a reason supports a given claim would be influenced by the context

of the dialogue. Accordingly, participants should have judged a fallacious argument as supportive of the goal in the context of a nonreasoned dialogue in which the use of fallacies is legitimate. However, there should be no differences in students’ judgements of the extent to which the reason (i.e., the fallacy) supports a given claim. In neither the reasoned dialogue nor the nonreasoned dialogue context does the reason legitimately support the claim. The second group was asked to evaluate the extent to which the reason supported the goal of the argument and supported the claim. Table 2 presents the means and standard deviations of participants’ responses to these two questions across contexts. It can be seen from the means for Question 1 (Q1) that across the scenarios the participants judged the argument as more helpful for achieving the goal when the argument was presented in the context of a nonreasoned dialogue. In contrast, there is no significant difference between participants’ rating of the extent to
Table 2. Mean ratings of the extent to which reasoning fallacies achieve the goal of the argument (Q1) and support the claim of the argument (Q2) by context Context Reasoned Scenario DRG UFO STU TAX DEA GOD Average goal claim goal claim goal claim goal claim goal claim goal claim goal claim M 2.00 2.22 1.83 2.04 2.61 2.70 2.09 2.35 2.09 2.09 2.52 2.43 2.18 2.30 SD 0.90 1.24 1.03 1.26 1.12 1.11 1.28 1.37 0.85 0.79 1.12 1.20 1.05 1.16 Nonreasoned M 2.74 2.30 3.30 1.83 3.09 3.04 3.52 2.30 2.52 2.04 2.78 2.48 2.99 2.33 SD 0.96 1.26 1.15 1.03 1.08 1.15 1.12 1.22 1.18 1.11 1.04 1.27 1.09 1.17

Note: N ¼ 23. DRG ¼ drug scenario. UFO ¼ alien scenario. STU ¼ exceptional students scenario. TAX ¼ tax scenario. DEA ¼ death scenario. GOD ¼ God scenario.

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

417

NEUMAN, WEINSTOCK, GLASNER

which the reason supports the claim across the contexts as indicated by the means from the responses to Question 2 (Q2) of each scenario. In order to examine the significance of our results we grouped participants’ performance on the first question and used a simple arithmetic sum of their answers as our dependent variable. This procedure was used in order to avoid the problem of multiple comparisons and the particularities of each scenario. The totalled mean was higher in the context of a nonreasoned dialogue (M ¼ 17.95, SD ¼ 3.93), than in the context of a reasoned dialogue (M ¼ 13.13, SD ¼ 4.18). The results of a paired-sampled t test indicated that this difference was statistically significant, t(23) ¼ 4.87, p , .001. By using the same procedure for the second question, we found a negligible and insignificant difference between participants’ performance in the contexts of nonreasoned and reasoned dialogues (M ¼ 14.00, SD ¼ 4.66, and M ¼ 13.82, SD ¼ 5.08, respectively).

who included in their response an explicit reference to the fallacy of the argument scored “1”, and the others scored “0”. Interjudge reliability was 95% for 10% of the sample. Across the six scenarios, more participants in the context of a nonreasoned dialogue responded to the argument by pointing at its fallacy. By using the Sign Test, this result was found statistically significant ( p ¼ .02). In the nonreasoned dialogue context, over three quarters in all the scenarios and over 90% in four of the scenarios produced a rebuttal directed toward the fallacy. This result seems to contradict our hypothesis, and the discussion below aims to address this.

Discussion
When asked to distinguish between the soundness of an argument and its appropriateness for a particular argumentative goal, the participants appeared sensitive to the context, recognizing on one hand that while the reasons might not have supported the claims, they may have helped further the arguer’s goal in nonreasoned dialogue. This sensitivity was also displayed by the larger percentages of students who argue that there is a problem in reasoned dialogue than those who argued that there is a problem in nonreasoned dialogue. Moreover, in general, a majority of students identified the reasoning fallacies in the reasoneddialogue contexts. However, contrary to the hypothesis, among those who had to counter the arguments there was a notably higher percentage of those who directed their rebuttals at the reasoning fallacy in the nonreasoned dialogue than in the reasoned dialogue. This result can be explained by task characteristics. In the context of a reasoned dialogue students were told that in order to achieve the aim each participant in the dialogue should present reasons that would justify his position. That is, the method of argumentation is positive and constructive. In this context, it is possible that when students were asked to take the adversary’s role they did not consider it legitimate to respond by attacking their opponent and pointing at its fallacious argument. In contrast, the context of a nonreasoned dialogue is a

Subexperiment 3 Table 3 presents the third group of participants’ responses to the questions asking them to produce an argument against the proponent across the argumentative contexts. Participants
Table 3. Percentages of students who rebutted the opponent’s argument by pointing at its fallacious character Context Reasoned Scenario DRG UFO STU TAX DEA GOD Average % 71 24 9 36 58 52 42 No. 18 6 2 9 14 13 10 % 92 91 96 83 95 76 89 Nonreasoned No. 23 23 24 21 24 19 22

Note: N ¼ 25. DRG ¼ drug scenario. UFO ¼ alien scenario. STU ¼ exceptional students scenario. TAX ¼ tax scenario. DEA ¼ death scenario. GOD ¼ God scenario.

418

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

JUDGEMENT OF INFORMAL REASONING FALLACIES

context of a quarrel or debate. It should be remembered that in these contexts the aim is to win or to hit your opponent, and therefore the method of argumentation permits and encourages the use of attack and to defend yourself against attack. It may be the case that this context encouraged the students to adopt a more aggressive and critical line that motivated them to respond by exposing the fallacy in their opponent’s argument. This explanation suggests that students’ performance were actually in line with our hypotheses. The first experiment established that students are context sensitive in the sense that they could identify problems with arguments, reject the relevance of the fallacies and respond to the argument by exposing its fallaciousness. Moreover, they could distinguish between whether the argument supported a claim and whether it achieved the goal of a given context. However, a fallacy may be identified as fallacy, and as inappropriate in certain contexts, but it still may have a persuasive effect. We hypothesized that context effects may be “washed out” by the factor of perspective taking. The second experiment was aimed to test the effect of taking a perspective on students’ ability to identify informal reasoning fallacy.

reason does not support the claim. Although previously shown that students are sensitive to the differences between the contexts of reasoned and nonreasoned dialogue, the act of adopting a perspective might supersede the argumentation norms of the reasoned-dialogue context.

Method
Participants A total of 52 twelfth graders from another regional secular high school participated in the study. A total of 50% of the participants were female, and the rest were male. The average age of the participants was 17 years. Materials The informal reasoning fallacies identification tasks. In this experiment, we used four stories corresponding to one of four types of informal reasoning fallacy: (a) the ad hominem fallacy (argument against the person); (b) the ad populum fallacy (the appeal to people); (c) the ad ignorantiam fallacy (the appeal to ignorance); and (d) the post hoc ergo propter hoc fallacy (false cause; see Appendix A). After presenting the scenario, participants received the first task in the form of: Let us assume X (i.e., the proponent) is right and it is true that Y (i.e., the reason). Try to explain why this reason supports/does not support the claim? (For example, “Let us assume Yossi is right and it is true that Avi is a person with no imagination. Try to explain why this reason supports the claim that UFOs exist.”) This task was open-ended and aimed to prompt the student to adopt the proponent’s or adversary’s standpoint. After this manipulation, the participant was asked to judge to what extent the reason supports the claim on a 5-point Likert scale. Design Each participant received a mix of scenarios. In a half of the scenarios the manipulation was to

EXPERIMENT 2
The second experiment tested whether the persuasion characteristic of informal argument overrides contextual sensitivity to the soundness of argument. The experiment was designed to keep the participants focused on the qualities of the reason while involving them personally in the argument. In this case, the students’ role in persuasion was more active than as the audience; they were assigned the role of taking the perspective of the proponent or adversary in alternate arguments. It was hypothesized that when prompted to explain, in the context of a reasoned dialogue, how a fallacious reason could possibly support a claim, participants would be less likely to identify the fallacious nature of the reason and more likely to find the reason actually supportive of the claim than those asked to explain why the

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

419

NEUMAN, WEINSTOCK, GLASNER

adopt the proponent’s perspective. This manipulation was conducted through the instruction to explain how the reason used by the proponent supports his claim. In the other, the manipulation was to adopt the adversary’s perspective through the instruction to explain why the reason does not support the claim. In other words, half of the students received a given scenario and were directed to adopt the adversary’s perspective (the anti condition), and the other half received the same scenario while directed to adopt the proponent’s perspective (the pro condition). All of the scenarios were presented in the context of a reasoned dialogue. The order of the scenarios was random. Procedure The experiment took place during a classroom session. Participants were asked to work individually on the informal reasoning task. Participants received the task of identifying the informal reasoning fallacy and worked 4 minutes on each informal reasoning scenario (16 minutes in total). All of the scenarios in the identification task appeared in written form. Each scenario appeared on a separate sheet of paper. The students received the same instructions that they had received in the first experiment.

Table 4. Mean ratings of the extent to which reasons supported claims by condition Condition Pro Scenario GRA STU UFO COM Average M 3.04 2.62 2.42 3.13 2.80 SD 1.33 1.20 1.18 0.99 1.18 M 1.90 2.22 1.60 2.40 2.03 Anti SD .94 1.17 .82 1.19 1.03

Note: N ¼ 52. GRA ¼ graphology scenario. UFO ¼ alien scenario. STU ¼ exceptional students scenario. COM ¼ computer scenario.

Results and discussion
If participants are influenced by the perspective that they adopt, then participants who were prompted to adopt the proponent perspective should judge the reason in support of the claim to be more relevant than should participants who were prompted to adopt the adversary perspective. Table 4 presents means and standard deviations of participants’ ratings across experimental manipulations. Although the context of the scenarios was that of a reasoned dialogue, it seems that participants were influenced by the perspective that they were prompted to adopt. Across the four stories participants’ ratings were higher when they were prompted to adopt the proponent’s perspective. That is, they were less likely to judge the reasons to be fallacious. By using one-way analyses of variance (ANOVAs), these differences were found significant in the GRA

scenario, F(1, 49) ¼ 10.53, p ¼ .002, h2 ¼ .20, the UFO scenario, F(1, 48) ¼ 8.02, p ¼ .007, h2 ¼ .15, and the COM scenario, F(1, 49) ¼ 5.34, p ¼ .02, h2 ¼ .10. A more refined analysis examines the quality of participants’ explanation in the task. Content analysis allowed the division of the participants’ explanations into three categories. Participants scored “0” if their explanation did not provide any rationalization of why the reason supports/ does not support the claim. Participants scored “1” if they provided any reasonable explanation of why the reason supports/does not support the claim. Participants scored “2” if their explanation reflected the fact that the reason provided by the proponent involved an informal reasoning fallacy. In the case where participants were assigned to explain how an apparent fallacy might have supported a claim, in a “2” answer either they acknowledged that the reason could not, in fact, support the claim, or they explained conditions under which the fallacy was not fallacious (such as, in the case of the ad populum scenarios, that the majority of people supporting a position were people with specialized knowledge). Two raters had complete agreement on the scores of 10% of the sample. Table 5 presents the percentages of students who identify the fallacy (i.e., scored “2”) or support a reasonable explanation in favour of the

420

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

JUDGEMENT OF INFORMAL REASONING FALLACIES

Table 5. Percentage of students who identified fallacy or offered a different explanation when explaining why a reason did or did not support a claim Condition Identify fallacy Scenario GRA STU UFO COM Average Anti 83 67 65 87 76 Pro 17 33 35 13 25 Anti 27 48 47 21 36 Explain Pro 73 52 53 78 64

Note: N ¼ 52. GRA ¼ graphology scenario. UFO ¼ alien scenario. STU ¼ exceptional students scenario. COM ¼ computer scenario.

Oakhill, & Morley, 1998; Evans, 1993; Franks, 1997; Luria, 1976; Markovits & Vachon, 1989; Neuman et al., 2004), in which one’s knowledge regarding the truth of a conclusion affects one’s evaluation of the validity of a syllogism or argument. On the other hand, what appears from the results is that when one is inclined to justify a claim, one becomes less sensitive to the soundness of the reasons and less critical of one’s own argument. This result is specifically relevant for those who are interested in teaching critical thinking, because it means that teaching critical thinking while using issues in which the pupils have an emotional involvement may not be the most appropriate context.

GENERAL DISCUSSION
reason, across the experimental conditions. The results show that when participants were asked to adopt the proponent’s perspective they provided more explanations, on average, that did not refer to the fallacy but justified the relevance of the fallacious argument (64% vs. 36%, respectively). However, when they were asked to adopt the adversary’s critical perspective (the anti condition) their responses reflect the fact that on average they identified the fallacy (76% vs. 25%, respectively). This result supports our argument that the results of Subexperiment 3 were created by task characteristics. In a comparison of these average proportions, the first and the second differences described above were found to be statistically significant (z ¼ 1.96, p ¼ .02, and z ¼ 2.85, p ¼ .003, respectively). It may not be so surprising that when directed to adopt the proponent’s perspective, students do not focus on problems with the reason. What is most meaningful here is that those in the role of proponent also stated that the reason actually did support the claim to a greater extent than those taking the adversarial perspective. Moreover, despite the fact that the claim was said to be true, when instructed to analyse the reason for problems, an average of three quarters identified the fallacy. The instruction to be critical apparently countered belief bias (Cherubini, Garnham, In general, one may identify two trends of research in argumentation. The first “universal” trend seeks general cognitive mechanisms that may explain students’ performance in argumentation tasks. This trend is evident in Kuhn’s (1991) attempt to explain argumentation through epistemic levels of development. The other “contextualist” trend is focused on identifying situational factors that may influence students’ performance in argumentation tasks (e.g., Brem & Rips, 2000). The current study follows the second trend by examining students’ sensitivity to the argumentative context and the way that this context influences their ability to identify informal reasoning fallacies. It was found that when presented with an explicit argumentative context participants can differentiate between the extent to which an argument supports a claim or a goal, and they can identify the problem with a fallacious argument in a context of a reasoned dialogue. The current study also points at the shortcoming of participants when they are asked to take an active role in the dialogue. The results of the second experiment suggest that students’ perspective plays a crucial rule in their ability to critically evaluate the relevance of the argument. In one conception of biased reasoning, the stance that one takes in an argument might lead

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

421

NEUMAN, WEINSTOCK, GLASNER

to bias in the use of inferential rules. Kunda (1990) theorizes that extensive findings that people’s reasoning is biased in favour of their prior positions do not necessarily indicate that it is merely the selection of beliefs or evidence that is biased. Rather, the directional goal of an argument might influence the types of inference that one might make. In the current study, most students found that a claim should not be inferred from a fallacious reason and, in the first experiment, discounted such a reason’s role in supporting a claim. Nevertheless, in the final experiment, when comparable groups of students were assigned directional goals in favour or opposed to arguments containing fallacious reasons, they differed in their criticism of this aspect of the argument. In particular, as the assignment to directional goal was not based on prior belief, unlike many studies of biased reasoning (Klaczynski, 1997; Klaczynski & Gordon, 1996; Kunda, 1990), this finding seems to lend support to the claim that the goals of an argument can bias the reasoning process and not just the selection of beliefs or evidence to support prior beliefs (Kuhn, 1989; Kunda, 1990). The current study differs in important ways from the biased reasoning studies. The variable considered here—the context of the argument as reasoned or nonreasoned dialogue—would seem to be more impersonal than the sort of variables considered in investigating the motivations behind biased reasoning. Biased reasoning studies tend to look at personal, motivational goals such as accuracy (Klaczynski, 1997; Kunda, 1990), accountability (Tetlock, 1983), or prior beliefs (Klaczynski, 1997; Klaczynski & Gordon, 1996; Kuhn, Amsel, & O’Loughlin, 1988), and other personal factors such as intellectual ability (Klaczynski, 1997; Klaczynski & Gordon, 1996), epistemological beliefs (Klaczynski, 2000), and evaluations of self (Kunda, 1990). In contrast, the experiments reported in this study stress the goals of the argument and manipulate the directional goals by assigning a position in an argument. Variations in what appear to be relatively superficial goals induced differences similar to those found in studies investigating

the influence of deeper, personal goals on biased reasoning. This further specification of the influence of goals in biased reasoning and, in particular, the inclusion of the argument context suggests that the pragmatics of argument rather than just internal or deep personal goals might account for variations in critical reasoning. What is known in pragmatics as the “context-change theory of speech acts” (Levinson, 1997) might explain the loss of critical perspective that appeared among students who had or could be assumed to have a strong enough sense of the inferential rules in argument to recognize the problem with fallacious reasons given for claims. This theory conceives of speech acts as operations on context. In other words, they are functions in the transformation of contexts, where context is understood as a set of propositions describing the beliefs, knowledge, and the commitments of the participants in the discourse. According to this theory a sentence changes the context, and this change is its speech act force or potential. This theory originally was used to explicate the notion of presupposition; if sentence S presupposes that q, then an assertion of S is appropriate only in a context in which q is already accepted. In short, the theory suggests that although context has a top-down influence on the utterances, the utterances have a bottomup influence on the context. When students were given the context they were able to critically examine the fallacies from their position as outside observers. They worked according to a form of top-down processing of the argument, from context to utterance. However, when taking an active role in the context, they seemed to have lost their top-down perspective. In this case, they worked according to the pragmatic principle in which the utterance has a strong bottom-up effect on transforming the context of the dialogue. The major conclusion of the above analysis is that explicating the context may help the students to critically examine fallacious arguments as long as they are not disturbed by taking an active position in the internal logic of the dialogue. These initial suggestions and conclusions may invite researchers interested in the cognitive process of

422

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

JUDGEMENT OF INFORMAL REASONING FALLACIES

argumentation to examine empirically the ways in which bottom-up and top-down processes conflate in the processing of arguments.
Original manuscript received 9 September 2004 Accepted revision received 13 March 2005 PrEview proof published online 20 September 2005

REFERENCES
Akman, V., & Surav, M. (1996). The use of situation theory in context modeling. Computational Intelligence, 12, 1 –13. Baron, J. (1991). Myside bias in thinking about abortion. Thinking and Reasoning, 1, 221– 235. Brem, S. K., & Rips, L. J. (2000). Explanation and evidence in informal argument. Cognitive Science, 24, 573– 604. Burgoon, M., Denning, V. P., & Roberts, L. (2002). Language expectancy theory. In J. P. Dillard & M. Pfau (Eds.), The persuasion handbook (pp. 117– 137). Thousand Oaks, CA: Sage. Burke, P. (2002). Context in context. Common Knowledge, 8, 152– 177. Cherubini, P., Garnham, A., Oakhill, J., & Morley, E. (1998). Can any ostrich fly?: Some new data on belief bias in syllogistic reasoning. Cognition, 69, 179– 218. Copi, I. M., & Burgess-Jackson, K. (1996). Informal logic. Englewood Cliffs, NJ: Prentice Hall. Eemeren, F. H. van, & Grootendorst, R. (1992). Argumentation, communication, and fallacies. Hillsdale, NJ: Lawrence Elbaum Associates, Inc. Eemeren, F. H. van, Grootendorst, R., & Henkemans, F. S. (1996). Fundamentals of argumentation theory. Mahwah, NJ: Lawrence Elbaum Associates, Inc. Evans, J. S. (1993). Bias and rationality. In K. I. Manktelow & D. E. Over (Eds.), Rationality: Psychological and philosophical perspectives (pp. 6 – 30). London: Routledge. Franks, B. A. (1997). Deductive reasoning with prose passages: Effects of age, inference form, prior knowledge, and reading skill. International Journal of Behavioral Development, 21, 501– 535. Iwanska, L., & Zadrozny, W. (1997). Introduction to the special issue on context in natural language processing. Computational Intelligence, 13, 301– 308. Jain, S. P., & Maheswaran, D. (2000). Motivated reasoning: A depth-of-processing perspective. Journal of Consumer Research, 26, 358– 371.

Johnson, B. T., & Eagly, A. H. (1990). Involvement and persuasion: Types, tradition, and the evidence. Psychological Bulletin, 107, 375– 384. Klaczynski, P. A. (1997). Bias in adolescents’ everyday reasoning and its relationship with intellectual ability, personal theories, and self-serving motivation. Developmental Psychology, 33, 273– 283. Klaczynski, P. A. (2000). Motivated scientific reasoning biases, epistemological beliefs, and theory polarization: A two-process approach to adolescent cognition. Child Development, 71, 1347– 1366. Klaczynski, P. A., & Gordon, D. H. (1996). Selfserving influences on adolescents’ evaluations of belief-relevant evidence. Journal of Experimental Child Psychology, 62, 317– 339. Kuhn, D. (1989). Children and adults as intuitive scientists. Psychological Review, 96, 674– 689. Kuhn, D. (1991). The skills of argumentation. Cambridge, UK: Cambridge University Press. Kuhn, D., Amsel, E., & O’Loughlin, M. (1988). The development of scientific thinking skills. Orlando, FL: Academic Press. Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108, 480– 498. Levinson, S. C. (1997). Pragmatics. Cambridge, UK: Cambridge University Press. Luria, A. R. (1976). Cognitive development: Its cultural and social foundations. Cambridge, MA: Harvard University Press. Markovits, H., & Vachon, R. (1989). Reasoning with contrary-to-fact propositions. Journal of Experimental Child Psychology, 47, 398– 412. McCarthy, J. (1987). Generality in artificial intelligence. Communication of the ACM, 30, 1030– 1035. Means, M. L., & Voss, J. F. (1996). Who reasons well? Two studies of informal reasoning among students of different grade, ability, and knowledge levels. Cognition and Instruction, 14, 139– 178. Neuman, Y. (2003). Go ahead prove that God does not exist! On student’s ability to deal with fallacious arguments. Learning and Instruction, 13, 367– 380. Neuman, Y., Glassner, A., & Weinstock, M. (2004). The effect of a reason’s truth-value on the judgment of a fallacious argument. Acta Psychologica, 116, 173– 184. Neuman, Y., & Weizman, E. (2003). The role of text representation in students’ ability to identify fallacious arguments. Quarterly Journal of Experimental Psychology, 56A, 849– 865.

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

423

NEUMAN, WEINSTOCK, GLASNER

Oaksford, M., & Hahn, U. (2004). A Bayesian approach to the argument from ignorance. Canadian Journal of Experimental Psychology, 58, 75 – 85. Perkins, D. N. (1989). Reasoning as it is and could be: An empirical perspective. In D. M. Topping & D. C. Crowell (Eds.), Thinking across cultures: The third international conference on thinking (pp. 175– 194). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Tetlock, P. E. (1983). Accountability and the perseverance of first impressions. Social Psychology Quarterly, 46, 285– 292. Walton, D. N. (1987). Informal fallacies: Toward a theory of argument criticisms. Amsterdam: John Benjamin Publishing Company. Walton, D. N. (1989). Informal logic: A handbook for critical argumentation. Cambridge, UK: Cambridge University Press. Weinstock, M., Neuman, Y., & Tabak, I. (2004). Missing the point or missing the norm: Epistemological norms as predictors of students’ ability to identify fallacious arguments. Contemporary Educational Psychology, 29, 77–94.

as evidence for supporting a given claim, the majority, in and of itself, is not strong evidence in favour of a claim, because the majority may be wrong or its decision might be based on an irrational process. In the above discussion, the alleged fact that in most countries drugs are legal tells us nothing about whether drugs should be legalized.

The argument from ignorance fallacy
The ad ignorantiam argument (the argument from ignorance) can be defined as an argument in which a conclusion about the truth (or the existence) of a proposition “A” is derived from the fact that “A” is not known to be false. For example: Avi and Ely are philosophers. During a philosophical seminar, they are debating the question: “Does God exist?” Avi argues that God exists. Ely argues that God does not exist. During the debate Avi argues: “No one has proven that God does not exist; therefore we can conclude that God exists.”

The argument against the person fallacy
The argument against the person (the ad hominem argument) suggests that the argument that a person makes cannot be trusted because of the arguer’s character or situation. For example: Marie and Iris are scientists. During a university seminar, they are debating the question: “Is it justified to avoid paying taxes?” Marie argues that it is justified to avoid paying taxes. Iris argues that it is not justified to avoid paying taxes. During the debate Marie argues: “Iris, you think it isn’t justified to avoid paying taxes because you come from a wealthy family.” The implication that P cannot be true because someone has a motive to believe it neither addresses nor disproves the substance of the argument.

APPENDIX A Description of the fallacies
In this study, we investigate students’ ability to identify four types of informal reasoning fallacies: (a) The ad populum fallacy (appeal to the people); (b) the ad ignorantiam fallacy (appeal to ignorance); (c) the ad hominem fallacy (argument against the person); and (d) the post hoc ergo propter hoc fallacy (false cause).

The false cause fallacy
The false cause fallacy (Walton, 1987, 1989) involves the inference of causation from temporal succession alone. According to Copi and Burgess-Jackson (1996) this fallacy can be schematically described as follows: 1. F comes after E (“post hoc”). Therefore (“ergo”) 2. F comes because of E (“propter hoc”). For example: Jerry and Leonard are high-school students. During a lesson, they are debating the question: “Is it necessary to teach young students computers in order to develop their intelligence?” Jerry argues that it is necessary to teach young students computers in order to develop their intelligence. Leonard argues that it is not necessary to teach young students computers in order to develop their intelligence. During the debate Jerry argues: “It is necessary to teach young students computers in order to develop their intelligence, because it is known that most of the

The argument from popularity fallacy
The argument from popularity (the ad populum argument) appeals to a sympathetic feeling of group solidarity in order to prove a point. However, norms or accepted beliefs do not necessarily establish the truth of a matter: Arguing that P is correct because most of the people believe in P may be an irrelevant argument. For example: Hannah and Martha are philosophers. During a philosophical seminar, they are debating the question: “Whether the use of drugs should be legal.” Hannah argues that the use of drugs should be legal. Martha argues that the use of drugs should not be legal. During the debate Hannah argues: “In most countries in the world the use of drugs is legal.” The above debate involves the use of an ad populum argument. Although there are instances in which the majority may be used

424

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

JUDGEMENT OF INFORMAL REASONING FALLACIES

intelligent students in the university learned computers when they were young.” In this case, it is false to argue that the fact that most of the intelligent students in the university learned computers when they were young implies that the use of computers is the cause of their superior intelligence. The false cause fallacy results from confusing between sufficient and necessary conditions for causality.

APPENDIX C The argumentation norms task
In life, we often encounter a situation in which two people or more have different opinions concerning the same issue. In this type of a situation, each person’s goal is to persuade the others that he/she is right and that the others are wrong. In order to prove that he/she is right, each participant in the discussion must provide support for his/her claim and discount the opposing claim. The scenarios that you have just read are examples of these types of discussion. In this type of discussion, just portrayed, do you think it is legitimate: 1. To argue that a certain claim is right just because most people think so? Yes/No/I don’t know 2. To attack the adversary’s opinion by attacking his character? Yes/No/I don’t know 3. To argue that a claim that cannot be proven is correct by default? Yes/No/I don’t know

APPENDIX B The structure of the arguments in a nonreasoned dialogue
1. 2. 3. 4. 5. X and Y, who don’t like each other, are arguing about Z. X argues that P. Y argues that P. The aim of each participant is to strike out at the other. In order to achieve this aim they personally attack each other. 6. During the dialogue X argues: Q.

THE QUARTERLY JOURNAL OF EXPERIMENTAL PSYCHOLOGY, 2006, 59 (2)

425

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer: Get 4 months of Scribd and The New York Times for just $1.87 per week!

Master Your Semester with a Special Offer from Scribd & The New York Times