This action might not be possible to undo. Are you sure you want to continue?

# International Journal of Mathematical Education in Science and Technology, Vol. 38, No.

1, 15 January 2007, 65–83

Introductory statistics, college student attitudes and knowledge – a qualitative analysis of the impact of technology-based instruction

M. MELETIOU-MAVROTHERIS*y, C. LEEz and R. T. FOULADIx

yCyprus College, Nicosia, Cyprus zCentral Michigan University, Mt. Pleasant, USA xSimon Fraser University, Burnaby BC, Canada (Received 14 July 2005) This paper presents findings from a qualitative study that compared the learning experiences of a group of students from a technology-based college-level introductory statistics course with the learning experiences of a group of students with non-technology-based instruction. Findings from the study indicate differences with regards to classroom experiences, student enjoyment of statistics, and student understanding of the many roles that technology plays in statistics. However, no significant differences were found between technology-based and non-technology-based instruction on students’ grasp of fundamental statistical concepts. In particular, these findings agree with the findings of several other studies, which indicate that incorporation of statistical software in the introductory statistics classroom might not always be very effective in building student intuitions about important statistical ideas related to statistical inference.

1. Introduction The developments in computing technology during the recent decades have radically transformed the culture of practicing statisticians. The calculating power of the computer has relieved the burden of extensive computations underlying many widely used statistical procedures. Computers have also improved the modelling of complex systems, and have resulted in an improved ability to generate data simulating the behaviour of these systems. These advances have opened new and exciting avenues for increased understanding of stochastic phenomena and the robustness of analytic data procedures. Similarly, technology has provided the opportunity to create an entirely new learning environment; it has significantly increased the range and sophistication of possible classroom activities [1]. While many statistics instructors still choose to ignore the calls for reform, many others have responded to the calls and have developed and implemented curricula that take advantage of technology to expose students to a more data-driven approach to statistics [2–5]. In a recent survey of introductory statistics courses in the USA, Garfield et al. found that most of the

*Corresponding author. Email: meletiou@cycollege.ac.cy

International Journal of Mathematical Education in Science and Technology ISSN 0020–739X print/ISSN 1464–5211 online ß 2007 Taylor & Francis http://www.tandf.co.uk/journals DOI: 10.1080/00207390601002765

66

M. Meletiou-Mavrotheris et al.

courses used some type of technology, and most of the faculty anticipated increasing or changing the use of technology in their courses [6]. Despite the wide use of technology in many statistics classrooms, relatively little published research exists describing its impact on student learning. New curricula are often developed and implemented without the benefit of research on their effects in terms of student learning. Even when done, evaluation of technological interventions is often very superficial and based on the performance in closed-ended tasks. According to the research literature, there are many examples of alarming situations in which statistics students get a correct judgment using an incorrect procedure, or produce a correct solution but do not understand either the solution or even the question behind it [7–10]. Having a vision of what technology is capable of is not the same as knowing how to take advantage of these capabilities in a teaching context [1]. The amount of research done, which investigates in depth whether students’ conceptual understandings change because of the use of computers, and if so in what ways, is still limited. Several statistics educators [1, 11–13] have alluded to the fact that there is still a lot to be learned about the use of technology as an instructional aid. Konold warns that the assertion that technology interventions that we design and implement will be effective without first testing them with students is a naive one [14]. Garfield argues, ‘without scrutinizing what is really happening to our students and using that information to make changes, it is unlikely that instruction will improve and we will be able to achieve our new vision of statistical education’ [15]. More research is needed in order to gain a better understanding of the source of learners’ difficulties with probability and statistics, and use this understanding to build improved learning environments. The article describes findings from a qualitative study, which compared the learning experiences of a group of students from a technology-based introductory statistics course with the learning experiences of a group of students with nontechnology-based instruction. In particular, this paper addresses the following two research questions in the context of a course on elementary statistics. (a) How does technology affect students’ motivation and attitudes towards statistics? (b) How does technology affect students’ understanding of concepts related to statistical inference? 2. Methodology 2.1. Context and participants We conducted individual in-depth interviews with 22 students who had taken a college-level introductory statistics course within three months after their completion of the course. Our intent was to investigate students’ statistical reasoning, attitudes, skills and thought processes, and to make a qualitative evaluation of the potential impact due to technology usage. The students interviewed were of varied mathematical backgrounds and had taken the introductory statistics course in one of five different departments between two campuses in the USA. They volunteered to participate in the study and were selected on the basis of their overall performance in the statistics course. This ensured that an

Introductory statistics, college student attitudes and knowledge Table 1. Type of course Technology-based No. of students 7 Interviewees’ background. Gender 3 Males 4 Females 2 1 2 1 1 3 2 2 1 4 3 1 Major Education Computer Science Accounting Meteorology Math Education Sociology Nursing Advertising Buiness Math Geology

67

Grade received in statistics course 3 As 2 Bs 2 Cs or lower 5 As 5 Bs 5 Cs

Non-technology-based

15

7 Males 8 Females

approximately equal number of the students interviewed were in As, Bs, and Cs or lower (see table 1 for some information on interviewees’ background). Seven of the 22 students had taken a course following the PACE approach, while the rest of the students experienced courses where non-technology-based instruction was implemented. PACE (Projects, Activities, and Cooperative learning using computers and Exercises) [16] is an approach that attempts to provide a structured framework for integrating projects and hands-on activities conducted cooperatively in a computer-based classroom environment. The nature of intellectual activity in the PACE classroom was very different from that in the non-technology-based classrooms. Whereas in the other courses instruction was primarily lecture-based and incorporation of technology was minimal or nonexistent, in the PACE course technology was an integral part of instruction. Each class meeting included laboratory time where students would work collaboratively on activities designed to help them explore different statistical concepts. Students had access to calculators and computers and used them to perform calculations, create graphical displays, analyse real datasets, and conduct simulations that illustrated statistical concepts such as sampling distributions and the Central Limit Theorem. Minitab was the main statistical software employed in the PACE course. 2.2. Instruments, data collection and analysis procedures We conducted in-depth semi-structured interviews of each of the study participants. During the interviews, we asked open-ended questions following the interview protocol in Appendix A as a guide. Despite the explicit design of the interview protocol, the questions were posed in such a way as to provide direction to respondents rather than restricting their responses. We were sensitive and responsive to issues identified by respondents, and often changed the order of specific questions depending on the interviewees’ responses. Nonetheless, before concluding each of the interviews, we ensured that we covered all the topics included in the interview protocol. Each of the 22 interviews, which were audiotaped, lasted for approximately two hours. During the interviews, students were encouraged to use paper-and-pencil as needed. In order to make students feel more comfortable, but also in order to get

68

M. Meletiou-Mavrotheris et al.

some information on their background, their attitudes and beliefs about statistics as well as their experiences from the course, the interviewer started by asking some general questions. Then, the following task was posed: Some instructors believe that today’s (name of the institution) students are not as good in math as (name of the institution) students were 20 years ago. You are assigned a project to find out if there exists evidence to support this belief. Students were then told that the interviewer was interested in their thinking and reasoning rather than in their getting the ‘correct’ answer. Although the interviews proceeded depending on students’ responses, the interviewer guided the dialogue so that the majority of the topics and main ideas encountered in an introductory statistics course were brought into the conversation. A vast amount of information regarding the culture of the statistics classroom and its impact on student attitudes and learning was collected. All of the interviews were transcribed and then analysed using the inductive data processing method of constant comparison analysis [17] which involves unitizing, categorizing, and coding by choosing words or phrases that specifically address the research questions, and which assisted us in the search for patterns and themes that were used to develop the study’s interpretation. Using related behaviours and comments within each topic, we wrote descriptions of the students’ attitudes, problem-solving strategies, and knowledge. These descriptions formed the findings of the study. Some of the findings that relate to learning of key concepts regarding statistical inference are described in the next section. In this section, names have been changed to protect the identity of the participants. The symbol - P accompanies each student from the PACE course. Since the study was a qualitative interview study, and the number of students in the study not as large as any quantitative study, the findings from the in-depth interview should not be trivially extended to a general student population. Nonetheless, they do provide some useful information to statistics educators that it may not be possible to obtain using quantitative data gathering techniques. 3. Results The interviews demonstrated differences between technology-based and nontechnology-based instruction with regard to college student introductory statistics classroom experiences, student enjoyment of statistics, and student understanding of the many roles that technology plays in statistics. However, no differences were found between technology-based and non-technology-based instruction on students’ grasp of fundamental statistical concepts related to statistical inference. 3.1. The contrast of technology and non-technology-based instruction – effects on student motivation and skills Our findings indicate that students that had completed the non-technology-based classes experienced a technique-oriented approach to statistics instruction, which focused on formulas rather than empirical applications. Due to the time necessary

Introductory statistics, college student attitudes and knowledge

69

for training students in computational methods, classroom examples and activities were carefully chosen to make manipulation of formulas easier and students never got to work with datasets that had more than just a few values. Several students described the learning environment as rigid and most of the activities as contrived. Others commented that, although recognizing how important statistics is, they felt that the tools they obtained from this class were inadequate for them to be able to solve any real world problems since they never had the opportunity to collect, explore, and present any real data: A lot of things we learned in the class were the basic technical things but we never got like here’s a case, here’s all the data to it, now how do you figure through, no practical application of statistics. (Lee Anna) It was a little dry. A lot of times it was hard to visualize the concepts we were talking about in statistics. (Andrew) The professor lectured the whole time. In such a course you might be able to do some more interesting things than just lecturing and working out the easy problems on the board. Which is what he would always do and then we would go home and have the really hard problems to do on our own. I’m sure there’s something he could have done to make it more interesting . . . like present a real problem and have us do something with it. (Kim) When asked about statistical concepts, students from the non-technology-based courses showed a strong dependency on formulas, often making several unsuccessful attempts to remember formulas and rules that they had never understood. These students also showed unfamiliarity with technology and lack of appreciation of its role in statistical endeavours. For example, when asked what tool they would use to get a random sample, only four of the fifteen students in the nontechnology-based group suggested using technology. These students realized that ‘that’s a lot to do physically, so we’d probably do it on a computer somehow’ (Bill). The majority of the students, however, seemed to be completely unaware of the fact that one could use a computer to generate random numbers. I’m not sure if there’s really a tool that would help you do a random selection. There must be such a tool, but I don’t know of one’ (Ariadne). The response of the following student, who suggests scrambling up all 50 000 names, is a typical one: ‘I’m going to get pieces of paper and a pen and write their names on the paper and pick my sample from that’ (Bob). Unlike students in non-technology-based courses, use of technology gave students in the PACE course much more experience with data collection, manipulation, and representation. Rather than spending most of their time learning techniques, PACE students were given the opportunity to conduct investigations and gain experience with the wide applicability and usefulness of statistical methods. Engagement with activities using technology seems to have led to an increased enjoyment and appreciation of the subject. Six of the seven PACE students reported that despite their generally negative attitudes towards statistics before taking the class, they ended up enjoying the course much more than they had anticipated. John-P, for example, before taking statistics had the impression that it was going to be nothing more than ‘number-crunching’, ‘playing with a bunch of statistics and figuring out averages.’ But, unlike his

70

M. Meletiou-Mavrotheris et al.

expectations, he ended up finding the course quite interesting. Computers had a lot to do with this: Yeah, it’s not so much as taking average. I mean, I really thought a lot of it was just averages and stuff like that, but there’s more than just averages. You have your errors, your skewed to the rights and lefts and all that stuff. Computers, I really didn’t think much of . . . I mean I knew computers could help you, but I didn’t realize with the graphs and stuff like that. I didn’t find it that . . . I mean it wasn’t boring . . . I mean, I was expecting to fall asleep in class. Well, I didn’t . . . (John-P) Sara-P, a prospective secondary education mathematics teacher, also changed her view of statistics after the PACE approach. Whereas before taking the PACE course, whenever hearing the word statistics she would ‘automatically think of just an abstract concept . . . numbers to samples, you know’, her experience with the course helped her realize ‘what lies behind those numbers, how they get them.’ She believed that her experiences with technology would help her a lot as a teacher: ‘We’re getting computers more and more in the classroom and since we did it on the computers, it’s going to be easier to do it that way than it was to try and do it by formulas’. Ann-P reported her surprise at ‘the amount of computer hands-on stuff’ employed in the class. She considered computers to be extremely helpful for instruction because ‘they make the class more interesting than if it was just numbers’ and manipulation of formulas. When asked whether she thinks that the teaching of statistics could have been as effective without the use of technology, she responded: ‘I don’t think so because if you don’t use the computers, you’re not going to see the application of statistics.’ Students in the PACE course came to view technology as an indispensable tool in statistical problem solving. When, for example, asked what tool they would use to get a random sample, six out of the seven PACE students mentioned a calculator or a computer without further prompting. They were well aware of the fact that using computers or calculators makes the process of generating random data much more efficient and that ‘trying to do it yourself, that would take a lot of time.’ They realized that, given the technologies available in modern society, it is silly for one to still do statistical computations by hand: Interviewer: So, before you took this class, what was your impression about statistics based on what you heard from your friends or from others? Mary-P: Very negative I think. Yeah. Because you hear all these horror stories about people in statistics and they don’t quite do good or it’s hard or it’s . . . . you know . . . . I guess I don’t really know. I guess I never really had statistics before and I was just basing it upon what others said. Interviewer: Did that affect your attitude towards this class? Mary-P: I think at first, it probably did. Interviewer: Were you scared? Mary-P: Yeah. Maybe just . . . I think at first, I was like, why am I here? This has nothing to do with what I am studying. I’m just going to be doing all this hard work and it has no real benefit. I don’t know. I guess that’s what I was kind of thinking at first. And then I found out it wasn’t as bad as I thought it was going to be.

Introductory statistics, college student attitudes and knowledge

71

Interviewer: So, after taking the class, you have some different thoughts about statistics? Mary-P: Yeah. Well, now that I can see the value that it has, like in meteorology for example, and seeing how it could benefit me in some of the . . . Like, I do a lot of personal research, too. And, when I did this file I created for the city of Holland last summer for record highs and lows, I went though . . . The way that I did it . . . . I did it all the long way. It took me three months to do. Instead of getting all the data and letting the computer do it for me, I went through it all one by one. I started in 1948 and then every time I saw a temperature above a certain level, I wrote it down for highs and went through every year, and when I saw a record broken, I would replace it. And I went through every year like this. And it took me three months to do that. And this was before I took statistics. So, if I had Minitab, I could have gotten all of this on disc or file or something . . . When I ordered the data, I ordered it on paper because I didn’t know how to work with a computer at that time to do this. But if I could have gotten it on disc and could have gotten the computer do it all for me, and it would have been done just like that.

3.2. The contrast of technology and non-technology-based instruction – effects on student learning of concepts related to statistical inference We observed in this study that most students, regardless of whether they were coming from the non-technology-based courses or from the PACE course, had superficial and poorly interconnected knowledge of many statistical concepts. Especially noticeable was students’ confusion about the nature and purpose of inferential statistics. Almost all of the 22 students interviewed seemed very confused about the concepts related to statistical inference. Though students in the PACE course clearly gained a lot from the implementation of technology-based instruction compared to students in non-technology based courses, use of technology did not seem very effective in improving student learning of key concepts related to statistical inference. Nearly every single student in the PACE group expressed frustration and failure to see the ideas underlying the computer activities designed to help them improve their understanding of inferential statistics. Their opportunity to use computer simulations as a tool for drawing repeated samples from a specified population and summarizing the results, did not seem to be very effective in helping them develop the idea of sampling distribution and to see its connection to confidence intervals, p-values, and hypothesis testing. Although students knew how to complete the computer activities, they failed to see the purpose underlying these activities: Yeah, all that. When we were doing it on the computer, I knew how to do it, but I felt like I didn’t know why I was doing it. To me, it was just that we were using too much of the computer and I didn’t understand what was going on because I think that we didn’t have enough of it in the lecture. (John-P) Findings from the interviews coincide with those obtained from an earlier study, in which a survey had been administered to students who had just completed the PACE course in order to investigate the impact that the course had on their attitudes

72

M. Meletiou-Mavrotheris et al.

towards statistics and on their motivation [18]. Most students reported that they enjoyed the PACE course and were satisfied with the integration of computer tasks, projects, and other in-class activities. The survey also required students to provide a self-assessment of their level of understanding of the most important concepts encountered during the course. Students reported having a poor understanding of sampling distributions and the Central Limit Theorem and expressed the opinion that the computer simulations employed in the course did not contribute towards improved understanding of these concepts. In order for students to be able to deal effectively with the statistics that they encounter, understanding of statistical inference is essential. Students should be able to make the necessary jump ‘beyond characterizing individual observations and samples’ towards ‘using the data at hand to make inferences about the general characteristics or ‘parameters’ of the population from which those data were drawn’ [19, p. 76]. However, since ‘statistical inference is almost by definition imperfect – all sampling introduces some error’ [20, p. 4], students need to become aware of potential threats to the validity of statistical inferences. They have to realize that when sampling, there is always the possibility of ending up with a sample estimate that differs from the population parameter because of bias or lack of precision. Furthermore, they should understand that random sampling methods help avoid biased samples and sample size controls the precision of the statistics obtained from the sample. In the following section, student comments regarding method of sampling, relationship between sample estimate and population parameter, and effect of sample size on sample variability and on variability of the sampling distribution of the mean are summarized. Because the analysis of the interviews indicated no major difference in the level of understanding of inferential statistics between students in the PACE and the non-technology based course, the presentation of findings proceeds by considering all 22 students as a single group. 3.2.1. Method of sampling. When prompted about the method of sampling they would use, the majority of students interviewed (18 students; 6 out the 7 students from the PACE course, 12 (if applicable) out of the 15 students from non-technology based courses) proposed random sampling because it would help to get a ‘good’ representation of the population. Several students correctly pointed out that random selection helps get a better representation of the data by avoiding bias. For example, Helen-P, when asked whether randomization helps to get a representative sample, responded: ‘Randomization would just try and eliminate bias.’ In the extreme were students who seemed to believe that, given that the sample selection was done randomly, one is almost bound to get a sample which is representative of the population. Sara-P, for example, argued that demographic factors should not affect our choice of sample if we do it randomly because randomization provides ‘an accurate representation of the population . . . a fair . . . a fair and accurate representation of the population’. Ann-P said, ‘random sampling ensures that you have more accurate results’. Ariadne believed that randomness allows studies with even a small size to be ‘pretty accurate’. Bill argued that randomization allows you ‘to throw out the extreme highs or lows and you just get like an honest mix of it’. This group of students perceived random sampling as being synonymous to ‘accuracy, correctness, and lack of error’ [21]; deviation of the sample estimate from the parameter value did not have the statistical sense of random variation but implied that something went wrong during the sampling process.

Introductory statistics, college student attitudes and knowledge

73

However, some of the students did not trust simple random sampling and preferred stratified sampling, claiming that simple random sampling might lead to very unrepresentative samples. These students seemed to ‘equate random with haphazard or without pattern’ [20, p. 4]. Margaret, for example, expressed distrust towards simple random sampling and preference for a method that takes factors such as geographic area into account, ‘so you get more of a spread out for people coming to . . . [name of institution], so you’re taking samples of people from all over, wherever they come from’. She did not recommend simple random sampling ‘because you could get a cluster of students that were the same or that came from the same place or had the same teachers or something like that.’ 3.2.2. Sample estimate vs. population parameter. During the interviews, students were asked to describe what they would do if they found the mean of a sample taken from a certain population to be different from the hypothesized population mean. Three students (1 from the PACE course, 2 from non-technology based courses) replied that if sample mean and population mean are any different, one should conclude that the sample comes from a different population. These three students did not consider the possibility that the observed difference might be the result of random variation. The rest of the students realized that finding a difference between population and sample mean is not adequate to make such a strong inference. However, they remembered very little about what one should do to assess how likely it is that an observed difference has occurred by chance. George-P, for example, knew that ‘just because it’s a lower number, it doesn’t mean that they’re a lot worse’, but when asked how he would decide, ‘how low is really low enough’, responded: ‘Um . . . I’m not sure. I remember that we did it, but I can’t . . . I don’t know if I remember how it was set up’. Most students remembered very little about ideas such as the Central Limit Theorem and hypothesis testing. Responses such as Alan’s were typical: Interviewer: Do you remember anything about the Central Limit Theorem? Alan: Oh that . . . I didn’t like it then. Interviewer: This is actually related to that (the sampling distribution of the mean) . . . they go hand in hand . . . Alan: I’m blocking out because it wasn’t my favourite part.

3.2.3. Effect of sample size on sample variability. All students understood that different samples from the same population could (and usually do) vary. They realized that the sample mean is not the same thing as the population mean and nobody expected the two to be exactly equal. All students also understood that larger random samples tend to produce better estimates and stressed that the larger the sample size, the more representative of the population it is going to be. Despite this, several misunderstandings regarding the effect of sample size on sample variability were prevalent. Many of the students in our study were found to have lack of statistical faith and over-reliance on sample variability. They did not seem to appreciate the fact that, despite the short-term unpredictability, stochastic events have a long-run regularity

74

M. Meletiou-Mavrotheris et al.

and, since statistical tools do take sampling variation into account, samples of even a few hundred subjects do have a lot to say about the population. Seven of the students interviewed (two from the PACE course, five from non-technology-based courses) felt that a random sample of fewer than a few thousand subjects, even if carefully chosen, would be too small and would very possibly lead to misleading conclusions. Additionally, several of the students interviewed did not realize that taking a small, in proportionate terms, sample of the population could often be adequate for drawing valid conclusions. In total, nine students (three from the PACE course, six from non-technology-based courses) said that one ought to consider population size before deciding what sample size to choose, because sample size should be at least a certain proportion of the population size. They went ahead and specified the proportion of the population that they would sample, giving percentages ranging from 10 to 50%. For one student, even taking half of the student population did not seem adequate; anything other than a census was not convincing enough. On the other extreme were nine interviewees (three from the PACE course, six from non-technology-based courses) who had more faith than they should have had in small samples. John-P, for example, stated, ‘probably anything over 50 would be good. You want to get as many as possible so you have less error, but if you put too many people in there, you’re going to be sitting there hunting people down forever and never getting your information.’ For Bob, ‘100 students would be OK’. Sara-P said: ‘You could probably . . . Probably if you went with 100 that would be large enough . . . I don’t think you would get much more improvement over that.’ When the interviewer told her that for this study we would typically need to ‘pick like 400 to 600 students’, her response was: ‘That many?’ These students’ tendency towards deterministic thinking can be interpreted as a reflection of their lack of awareness of variability and its relationship to sample size. Similarly to the students who expressed overconfidence in the accuracy of random samples of even a small size, they also considered small samples to be more representative of the population than they really are because they applied characteristics of large-sample variation to them. It was not apparent to them that sampling variation depends on sample size and that it takes a reasonably sized sample for random phenomena to display regularity. 3.2.4. Effect of sample size on the sampling distribution of the mean. The ‘recognition that the estimates of a population parameter will vary and that this variation will conform to a predictable pattern’ [2, p. 137] is a critical step towards developing the theory of statistical inference. However, when inquired about how to construct the sampling distribution of the mean for a given sample size, most students gave responses that revealed their ignorance and confusion regarding the notion of sampling distribution Several students seemed to confuse sample means with individual scores. Helen-P, for example, when asked to explain the meaning of sampling distribution of the mean responded: ‘When I think of sampling distribution, I think of like range of numbers or something’. Bob argued that the distribution of 200 sample means, each of sample size 100, is ‘going to be similar to the population’s distribution’. Lee Anna, when asked whether she thinks that the spread of the sampling distribution of the mean for a sample size of 400 will be as wide as that for individual scores responded: ‘They are going to be down to 200 (the lowest possible SAT score)’. In general, the different opinions that the interviewees expressed about

Introductory statistics, college student attitudes and knowledge

75

the relationship between sample size and spread and shape of the sampling distribution can be categorized as follows: (i) correctly identifying the relationship between sample size and spread of the sampling distribution and pointing out that the shape of the sampling distribution will be normal (eight students; two of the seven students from the PACE course, six of the 15 students from non-technology-based courses); (ii) understanding that the spread of the sampling distribution decreases with sample size, but not mentioning that the shape of the distribution becomes more normalized (two students); (iii) claiming that the spread of the sampling distribution increases with sample size (six students); (iv) claiming that the spread of the sampling distribution increases with sample size, and the shape becomes more normalized (two students); (v) believing that sample size has no effect on the sampling distribution of the mean (three students); (vi) Other responses (one student). 4. Discussion The purpose of this study was to investigate how technology is changing the learning of introductory statistics at the college level. The conclusions drawn concerning the impact of technology should be treated carefully as the number of students interviewed was a small voluntary sample that might have benefited from technology differently from the rest of the students in these courses. Also, the effect of technology cannot be judged in isolation from curricular materials and instructor characteristics. What made the experience of students in the PACE technology-based classroom distinctive was not the use of technology alone but the overall nature of the course with its emphasis on problem solving and working with real-life problems. However, this study does provide some insights into the effects of technology on student learning of statistics, and highlights some of the continued difficulties in helping students grasp fundamental statistical concepts. In the study, we saw that engagement with technology had a positive impact on the PACE students’ motivation, on their enjoyment and appreciation of statistics, and gave them increased exposure and familiarity with the practical aspects of the subject. At the same time, we observed that not only the non-technology-based, but also the PACE students’ knowledge of statistical ideas seemed to often be superficial and not well interconnected. In particular, technology did not adequately contribute into improving student understanding of key concepts related to inferential statistics, and particularly the notion of sampling distribution. Most students did not seem aware of the transformation involved when moving from the distribution of individual samples to the distribution of sample means; only a few students had complete understanding of the process of forming a sampling distribution. The opportunity PACE students had to work on tasks specifically designed to help them improve their understanding of sampling distribution did not prove very effective as an aid to achieve of this learning objective. Almost all students in the PACE course expressed frustration and failure to grasp the ideas underlying the computer activities; they noted that although knowing how to complete the computer assignments, they failed to see the purpose underlying these assignments.

76

M. Meletiou-Mavrotheris et al.

Nonetheless, the use of technology had a positive impact on the PACE students’ motivation, helping them acquire a much better grasp of the practical aspect of statistics, to come to ‘perceive statistics as a valuable tool rather than a bother’ [22, p. 57]. The experience of using technology in the classroom had an impact on students’ appreciation of the use of technology as a tool to support research processes and statistical analyses. However, arousing students’ interest is a necessary but not sufficient condition for learning [23] and statistical competence with technology is not adequate in its own right. As Starkings stresses, ‘statistics is a subject that is both theoretical and practical by nature and needs to be addressed in both of these aspects’ [12, p. 250]. Being statistically literate requires not only the ability to process data using some software, but also the ability to understand the assumptions behind the statistical analyses and the constraints imposed by the underlying theory of the analyses [24, p. 307]. The results from our study agree with the findings of the considerable research that has been done in the last thirty years, which attests to the many difficulties people, even ones with substantive formal training, have with understanding probability and statistics. It has repeatedly been documented that people’s intuitions are weak and misleading and too often run counter to stochastic reasoning, and that probabilistic thinking is not usually within the repertoire of the problem-solving strategies people employ [25]. It has been stated, time and time again, that people tend to think deterministically and lack awareness or understanding of variation and its relation to sample size. They tend to (i) believe that any difference in means between two groups is significant, (ii) have unwarranted confidence in small samples, (iii) have insufficient respect for small differences in large samples, and (iv) mistakenly believe that there is no variability in the ‘real world’ [26]. However, most of this prior research was done with students in non-technological learning environments. The implication from our study is that most of these misunderstandings might still persist despite the use of technology and empirical activities, that fluency with technology does not necessarily imply thorough understanding of the underlying statistical concepts. Other statistics educators have also questioned the effectiveness of computer simulations in helping students develop the important ideas of statistics. Del Mas, Garfield, and Chance, point out that ‘despite the accepted approach used to integrate simulation software into a statistics class, there is little published research describing and evaluating such an approach’ [27, p. 4]. In those few cases where formal research was conducted to study the kinds of understanding that develop as a result of the use of computer-based simulations, findings have not always been very impressive [e.g. 2, 27–32]. Lipson for example, after conducting a study to examine the effect of computer-based strategies on students’ understanding of sampling distributions, found that ‘many of the propositions that seem paramount in an a priori analysis to an understanding of sampling distribution do not seem to have been evoked by the computer sessions, even though the sessions had been specifically designed with these in mind and students were led to these propositions by the focus questions’ [2, p. 146]. The software typically employed in statistics classrooms, including the PACE course that some of the students in our study had completed, are not pedagogical tools but rather professional statistical software designed primarily for statistical analysis rather than instruction. These professional packages support the conventional technological approach of treating the computer package as a black-box.

Introductory statistics, college student attitudes and knowledge

77

As the findings from our study, as well as other studies indicate, incorporation of such professional software in introductory statistics classroom might not be very effective in building student intuitions about fundamental statistical ideas such as sampling distribution and statistical inference. Scheaffer notes that although technological advances have made data exploration ‘fun and quick’, ‘inference is still a black box, whether done by hand or on the computer’ [33, p. 157] and stresses the need for more thoughtful and constructive use of technology in the classroom. Ben-Zvi argues that statistics instruction ought to employ technological tools which support active knowledge construction, provide opportunities for students to reflect upon observed phenomena, and contribute to the development of students’ metacognitive capabilities [34]. Computer-based learning environments should encourage students to: (1) practice data analysis with an exploratory, interactive, open-ended working style, and combine exploratory and inferential methods, graphical and numerical methods; (2) extensively use multiple linked representations and simulations to construct meanings for statistical concepts and ideas; and (3) construct models for statistical experiments, and use computer simulations to study them [34]. Advances of technology provide us with new tools and opportunities for supporting the development of student statistical reasoning. Lately, several newly developed packages that were explicitly designed to enhance learning and make statistical thinking accessible to students [35] have appeared. Their design encourages students to experiment with statistical ideas, to make conjectures and use the investigative tools provided by the software to test and modify these conjectures. Meletiou-Mavrotheris [36] describes an instructional experiment that explored the capabilities of one of these learning environments, the educational statistical package FathomÕ [37]. Fathom’s designers draw on current constructivist theories of learning as well on several years of academic research about the way students learn and process statistical concepts and the main difficulties they face. The software has affordances towards more constructive pedagogical approaches compared to professional statistics software. Through attributes like the ability to dynamically link multiple representations, to provide immediate feedback, and to transform a whole representation into a manipulable object, it allows students to directly interact with data and to see the immediate impact of their actions on the different data representations displayed on the screen. Findings from the study indicate that use of FathomÕ led students to the construction of a fairly coherent mental model of sampling distribution and its connection to statistical inference. Different types of software have different roles to play in statistics courses, and whatever software is used, it should be appropriate for its educational purpose. For example, one piece of software might be best to use to help students understand sampling theory, but this might not be the best piece of software to enable them to carry out an analysis of variance. Although the use of professional statistical software in the classroom supports the acquisition of technology-based data analytic skills, students’ understanding of fundamental concepts is not supported more than in a non-technology-based course. Since a primary aim of many introductory statistics courses is to help students of varied backgrounds and aspirations understand some fundamental concepts related to probability and statistics, the software used in the introductory statistics classroom should be aligned with constructive views of learning if it is to support this aim. It should support content and pedagogy by affording learners with tools they can use to construct their own

78

M. Meletiou-Mavrotheris et al.

conceptual understanding – tools for not only data display and visualizations, but also thinking and problem solving. Thus, it is preferable for instructors of such courses to include educational software developed for concept development rather than solely rely on professional software designed for professional data analysis. Use of the former type of software in the classroom supports active knowledge construction by ‘doing’ and ‘seeing’ statistics in a powerful and flexible learning environment [34]. It encourages students to build, refine, and reorganize their prior understandings and intuitions about the stochastic, in contrast to black-box simulations that do not allow direct connections between the formal and the informal. This interaction between the data and the theoretical model is more convincing than black-box simulations, and it helps students construct more powerful meanings. By including both educational and professional software in a technology-based learning environment, students’ motivation, understanding, and skills may be simultaneously enhanced.

Appendix A: Interview Protocol (A) Questions on students’ background, attitudes, and motivation 1. 2. 3. 4. 5. 6. 7. 8. What is your major? Why did you take this course? Before taking this class, what was your impression about statistics? After taking the class, do you have any different thoughts about statistics? Do you consider statistics to be the same thing as mathematics? Do you think statistics will be useful for your future career? What type of learning style do you like the most? What would be the important motivation for you to be willing to spend time to study for a course?

(B) Questions assessing understanding of concepts taught in the introductory statistics course Now we will start talking about some of the things that you learned in your statistics course (explain that purpose is not to test him/her whether he/ she knows or doesn’t know something but to see how he/she thinks about the different issues; goal of study is to find how students think about statistics and use the findings to come up with ways to improve instruction): Some students believe that today’s (name of institution) students are not as good in math as students 20 years ago. You are assigned a project to find out if there exists evidence to support this belief. 1. What would be the first thing you would do? 2. Which of the things you have proposed do you consider being the most reasonable one? Why? 3. Let’s use the SAT score as our measurement. And, also, let’s assume we have the mean score of (name of institution) students 20 years ago; it was 580 in math. What is a reasonable target population from which you are going to draw your sample? (All current students, all freshmen, all students in the state?) 4. So, you are ready to take a sample from the target population. How are you going to take this sample? What would be a reasonable sample size? Why?

Introductory statistics, college student attitudes and knowledge

79

5. Why take a random sample? Why not just your friends or just people from a residence hall? 6. Any other way besides simple random sampling to choose your sample? 7. What demographic factors may affect your choice of sample? 8. How can you take a sample that guarantees that your sample includes all (demographic factors mentioned above)? (purpose of question is to see what they know about stratified random sampling) 9. Now, you have collected 400 SAT scores using simple random sampling or stratified sampling for the purpose of comparing with the mean math SAT score of 20 years ago. You are ready to summarize these 400 scores. What information do you think will be important for describing these scores? 10. Do you know how the mean of the SAT scores is computed? 11. Let’s say that the mean score is 540. Explain what the value 540 means. 12. What other summary would you suggest? 13. Do you know how the median is computed? 14. Let’s say the median score is 535. What does this median score of 535 tell you? 15. Any other summary you think would be useful? (Variance, standard deviation) 16. Here is the formula for computing standard deviation (s.d), you do not need to remember it (give formula). Can you explain what sample s.d. means? What does it measure? 17. Let’s say the sample s.d. is 40. Can you explain what this value means for our study? 18. What other type of information is important? 19. How would you graphically display the distribution of these 400 scores? (Histogram, stem-leaf plot) 20. Verbally describe how a histogram is constructed. 21. For this sample of 400 scores, we have a mean of 540, a median of 535, and a s.d. of 40. How do you think the histogram for this sample would look like (symmetric, skewed-to-right, skewed-to-left)? 22. Could you draw a symmetric histogram? Could you give an example of a population that might have a symmetric distribution? How about a skewedto-right histogram? How about a skewed-to-left histogram? 23. What about the scores in a test? Can you give a test situation where you would expect the distribution of scores to be symmetric/skewed-to-right/skewedto-left? 24. If the histogram of the SAT scores ends up being symmetric/skewed, what does this shape tell you about the population of SAT scores? 25. Suppose one of the scores in the sample is 590. You want to find out where this score stands among all 400 scores. What do you need to find out? (Percentile) 26. Verbally explain what percentile means. 27. In this case, can 590 be in the 25 th percentile? Why or why not? 28. What would be a reasonable percentile for a score of 590? Why? 29. Given our sample characteristics, do you think it is reasonable to assume that all (name of institution) scores follow a normal distribution? Why? 30. Suppose that the SAT scores of all (name of institution) students followed a normal distribution with a mean of 520 and a s.d. of 50. Can you draw a normal curve of the distribution? 31. Describe the difference between 520 (population mean) and 540 (sample mean).

80

M. Meletiou-Mavrotheris et al.

32. Describe the difference between 50 (population s.d.) and 40 (sample s.d.). 33. Can you estimate, given the population parameters, the chance of having SAT scores higher than 520? 34. Let’s consider the area that is within two standard deviations of the mean (420 to 620). What approximate percentage of scores do you expect to fall between 420 and 620? Why? 35. Is it reasonable to say around 5%, 50%, 95%, 100%? 36. Let’s go back to our sample. We have obtained a sample of 400 scores and calculated the following statistics: mean ¼ 540, median ¼ 35, s.d ¼ 40. Do you expect the population parameters for all (name of institution) students to be the same as these sample statistics? 37. Based on your response, if we take a different sample, but with the same sample size, then we will probably obtain a different estimate of the overall mean. How would you decide which one is a more accurate estimate of the population mean? 38. Can you suggest any way that would have increased the precision of using the sample mean to estimate the entire (name of institution) population mean score? (Increase sample size) 39. Say we take another sample of n ¼ 3000 students and obtain an average score of 512. Which one do you expect to be a more precise estimate of the entire population mean? 540 or 515? Explain why. 40. Here we are talking about the precision of using the sample mean to estimate the entire population mean. How can we measure the degree of precision? 41. What would you expect the sample mean to be, if you kept on increasing the sample size? 42. Can you verbally explain how the sampling distribution of the sample mean is formed using the example of a sample size of n ¼ 400 drawn from the entire school population? 43. What do you think the expected mean of all possible means, each of n ¼ 400, would be? 44. What do you think the shape of this sampling distribution of the mean will look like? Why? 45. The C.L.T. says: ‘the shape of the sampling distribution of the sample mean will become normal, regardless of the original population from which we drew the sample, as long as the sample size is large enough.’ Can you elaborate your understanding of this statement? What does this mean to you? 46. If you increase the sample size, what will you expect to happen to the sampling distribution of the mean as far as the shape is concerned? 47. Can you draw two relative distribution curves for the sample mean, one for n ¼ 400 and the other for n ¼ 10? 48. Now, back to our original purpose: ‘To decide if there is strong enough evidence that current (name of institution) students’ mean SAT score is significantly lower than that of 20 years ago, when the mean was 580.’ What kind of problem is this? What type of test is it? (Right sided? Two-sided? Left-sided?) Why? 49. Can you give the hypothesis that we are testing? Ho: H1: 50. How are you going to decide if you should choose Ho or H1? (describe the general idea)

Introductory statistics, college student attitudes and knowledge

81

51. Remember that we obtained a sample of 400 scores with a mean of 540. Is it reasonable, since 540<580, to claim that today’s SAT mean score is significantly lower than that of 20 years ago? Why? Why not? 52. Will knowing the sampling distribution of the sample mean help us in making our decision? How? (You may draw a picture to explain your thinking). 53. In testing this hypothesis, we often use a level of significance of, let’s say, ¼ 5%. What does this value mean? Why not take ¼ 30%? 54. Can you give an analogy to hypothesis testing using a court case?

References

[1] Hawkins, A., 1997, Myth-conceptions. In: J.B. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical Institute), pp. vii–viii. [2] Lipson, K., 1997, What do students gain from simulation exercises? An evaluation of activities designed to develop an understanding of the sampling distribution of a proportion. In: J. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical Institute), pp. 137–150. [3] delMas, R., Garfield, J. and Chance, B., 1999, A model of classroom research in action: developing simulation activities to improve students’ statistical reasoning. Journal of Statistics Education, 7, 3. Online: http://www.amstat.org/publications/jse/secure/v7n3/ delmas.cfm [4] Saldanha, L., 2001, ‘Is this sample unusual?’: An investigation of students exploring connections between sampling distributions and statistical inference. Paper presented at the Research Forum on Statistical Reasoning, Teaching and Literacy. Armidale, Australia. [5] Doane, D.P., 2004, Using simulation to teach distributions. Journal of Statistics Education, 12, 1. Online: www.amstat.org/publications/jse/v12n1/doane.html [6] Garfield, J., Hogg, B., Schau, C. and Whittinghill, D., 2002, First courses in statistical science: the status of educational reform efforts. Journal of Statistics Education [Online], 10, 2: http://www.amstat.org/publications/jse/v10n2/garfield.html [7] Estepa, A. and Batanero, M.C., 1994, Judgments of association in scatter plots: an empirical study of students’ Strategies and Preconceptions. In: J.B. Garfield (Ed.) Research Papers from the Fourth International Conference on Teaching Statistics (Minneapolis: The International Study Group for Research on Learning Probability and Statistics). [8] Garfield, J. and del Mas, R., 1990, Exploring the stability of students’ conceptions of probability. Research Papers from the Third International Conference on Teaching Statistics (Dunedin, New Zealand: University of Otago). [9] delMas, R. and Garfield, J., 1990, The use of multiple items to identify misconceptions in probabilistic reasoning. Research Papers from the Third International Conference on Teaching Statistics (Dunedin, New Zealand: University of Otago). [10] Meletiou-Mavrotheris, M. and Lee, C., 2002, Student understanding of histograms: a stumbling stone to the development of intuitions about variation. Proceedings of the Sixth International Conference on Teaching Statistics [CD ROM]. Durban, South Africa. [11] Burrill, G., 1997, Discussion: how technology is changing the teaching and learning of statistics in secondary school. In: J.B. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical Institute), pp. 71–74. [12] Starkings, S., 1997, How technological introduction changes the teaching of statistics and probability at the college level. In: J. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical Institute), pp. 233–254.

82

M. Meletiou-Mavrotheris et al.

[13] Chance, B., 2001, Stats education: current and future directions. MSOR Connections [Online] 1, 3 http://www.mathstore.ac.uk/newsletter/aug2001/pdf/statseddirections.pdf [14] Konold, C., 1995, Issues in assessing conceptual understanding in probability and statistics. Journal of Statistics Education, 3, 1, http://www.amstat.org/publications/jse/ v3n1/konold.html. [15] Garfield, J.B., 1994, Beyond testing and grading: using assessment to improve student learning. Journal of Statistics Education, 2, 1, http://www.amstat.org/publications/ jse.v2n1/garfield.html. [16] Lee, C., 1999, A computer-assisted approach for teaching statistical concepts. Journal of Computers at Schools, 16, 193–208. [17] Glaser, B. and Straus, A., 1967, The Discovery of Grounded Theory: Strategies for Qualitative Research (New York: Aldine). [18] Lee, C., 1998, An assessment of the PACE strategy for an introductory statistics course. Proceedings 5th International Conference on Teaching Statistics, pp. 1215–1221, Singapore, July. [19] Nisbett, R.E. and Ross, L., 1980, Human Inference: Strategies and Shortcomings of Social Judgment (Englewood Cliffs, NJ: Prentice Hall). [20] Jacobs, V., 1997, Missed opportunities on the teaching and learning of data and chance. In: J. Garfield and J. Truran (Eds) Research Papers on Stochastics Education, pp. 3–37. [21] Ballman, K., 1997, Greater emphasis on variation in an introductory statistics course. Journal of Statistics Education, 5, 2, http://www.amstat.org/publications/jse/v5n2/ ballman.html. [22] Lajoie, S., 1997, The use of technology for modelling performance standards in statistics. In: J.B. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical Institute), pp. 70–75. [23] Hawkins, A., 1997, Children’s understanding of sampling in surveys. In: J.B. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical Institute), pp. 1–14. [24] Glencross, M.J. and Binyavanga, K.W., 1997, The role of technology in statistics education: A view from a developing region. In: J. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical Institute), pp. 301–308. [25] Pfannkuch, M. and Brown, C.M., 1996, Building on and challenging students’ intuitions about probability: can we improve undergraduate learning? Journal of Statistics Education, 4, 1, http://www.amstat.org/publications/jse/v4n1/pcannkuch.html. [26] Shaughnessy, J.M., 1992, Research in probability and statistics: reflections and directions. In: D. Grows (Ed.) Handbook of Research on Mathematics Teaching and Learning (New York: Macmillan), pp. 465–494. [27] delMas, R., Garfield, J. and Chance, B., 1999, A model of classroom research in action: developing simulation activities to improve students’ statistical reasoning. Journal of Statistics Education [Online], 7, 3. http://www.amstat.org/publications/jse/secure/v7n3/ delmas.cfm [28] Lipson K., Kokonis S. and Francis, G., 2003, Investigation of statistics students’ experiences with a web-based computer simulation. Proceedings of the International Association of Statistical Education Satellite Conference on Statistics Education: Statistics education and the internet, Berlin. [CD-ROM] (Voorburg: International Statistical Institute). [29] Konold, C., Pollatsek, A., Well, A. and Hendrickson, J., 1990, The origin of inconsistencies in probabilistic reasoning of novices. In: J.B. Garfield (Ed.) Research Papers from the Fourth International Conference on Teaching Statistics (Minneapolis: The International Study Group for Research on Learning Probability and Statistics). [30] Hodgson, T.R., 1996, The effects of hands-on activities on students’ understanding of selected statistical concepts. In: E. Jakubowski, D. Watkins, and H. Biske (Eds) Proceedings of the Eighteenth Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education, pp. 241–246 (Columbus, OH: ERIC Clearinghouse for Science, Mathematics, and Environmental Education).

Introductory statistics, college student attitudes and knowledge

83

[31] Green, D., 1990, Using computer simulation to develop statistical concepts. Teaching Mathematics and its Applications, 9, 58–62. [32] Behrens, J.T., 1997, Toward a theory and practice of using interactive graphics in statistical education. In: J.B. Garfield and G. Burrill (Eds) Research on the Role of Technology in Teaching and Learning Statistics (Voorburg, The Netherlands: International Statistical Institute), pp. 111–121. [33] Scheaffer, R.L., 1997, Discussion. International Statistical Review, 65, 156–158. [34] Ben-Zvi, D., 2000, Toward understanding the role of technological tools in statistical learning. Mathematical Thinking and Learning, 2, 127–155. [35] Rubin, A., 2002, Interactive visualizations of statistical relationships: what do we gain? Proceedings of the Sixth International Conference on Teaching Statistics, Durban, South Africa. [36] Meletiou-Mavrotheris, M., 2003, Technological tools in the introductory statistics classroom: effects on student understanding of inferential statistics. International Journal of Computers for Mathematical Learning, 8, 265–297. [37] Finzer, W., 1999, Fathom. [Software] 1993 (Berkeley, CA: Key Curriculum Press).