Scribd Logo
Upload

Welcome to Scribd!

  • Research Method Chapter 19

    Uploaded by

    PRIYA TARISINI
    159 views24 pages
    GRU

    Original Title

    RESEARCH METHOD CHAPTER 19

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    GRU

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    159 views24 pages

    Research Method Chapter 19

    Uploaded by

    PRIYA TARISINI
    GRU

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    of 24
    chapter 1] 9 Action Research IMPORTANT 1. Action research is more closely tied to educators’ practice than other types of JAS —_‘es22rch, soit possesses great potential to solve problems of practice. 2. Educators carry out action research projects for various purposes—personal, ; professionel, and political 3. After carrying out an action research project, educators report their findings } to colleagues in some form and often decide on a new course of action for continuing their research. 4, Systematic data collection, analysis, and reflection are what distinguish action research from educators’ other approaches to problem solving, 5. A good action research project provides educators an understanding of new practices and empirical data thet show the results of those practices on teaching and learning in a real-ife setting 6. Action research benefits educators by improving their theories of education, their work with students, and their interactions with colleagues. 7. By following the principles of action science, educators can identify and address discrepancies between their espoused beliefs and their actual behavior. 8, Action research is carried out primarily by insiders with an internal perspective on the problems of practice being studied, but outsiders can help in research design, interpretation of findings, helping insiders address strong feelings about the topics being studied, and contributing support for personal and social change. 9. Action researchers can use a number of validity criteria to help them design their research to achieve maximum credibility and trustworthiness. Bl 487 488 Part VI_ Using Other Research Methodologies to Study Problems of Practice Key TERMS action research action science catalytic validity collaborative action research democratic validity dialogic validity espoused theory insider process validity insider research reflection intentionality solfstudy research outcome validity systematicity outs teacher research Participatory action research ‘theory-in-action practitioner research The History of Action Research ‘Action research is a form of research carried out by educators in their everyday work settings for the purpose of improving their professional practice. This form of research corresponds to what some researchers call practitioner research (Zeichner & Noffke, 2001), teacher research (Cochran-Smith & Lytle, 1999), insider research (Kemmis & ‘MeTaggart, 2000), and self-study research (Zeichner & Noffke, 2001). When researchers, and participants collaborate in solving a problem of practice, itis sometimes called par- ticipatory action research (Reason & Bradbury, 2001). Social psychologist Kurt Lewin (1946) helped popularize action research during World ‘War I. He observed that much research resulted in scholarly publications that had litle effect on professionals’ work or on the broader society. Lewin developed action research a8 form of investigation that community members and working professionals could do together to promote positive social change. In one study, Lewin assembled small groups of housewives to consider the use of organ meats in family meals. At the time, the U.S. government was seeking to promote these cheaper cuts of meat because ofa wartime shortage in meat supplies. Lewin's action research project showed the value of group discussion in changing people's attitudes and behavior about a significant socal problem. ‘Action research was popular in the social sciences during the 1940s and 1950s in the United States. Is se declined subsequently, because most academic researchers, and the public as well, considered research to mean primarily experimentation cartied out in laboratory settings. Australian and British educators brought action research back to the forefront inthe 1960s, and they continue to use this approach extensively. Acton research has also again become popular in the United States, especially among teachers who carry cout studies as pat of their teacher education preparation program. Using Action Research to Address Problems of Practice ‘Action research has more practical purposes than formal research. The quality of an ac- tion research project depends on how welt it serves educators’ immediate, local needs. As ‘a result, action research is easier to design and carry out as part of an educator's everyday practice than a formal research study. ‘Table 19.1 summarizes the characteristics of action research compared to those of for- mal research. These characteristics highlight its relevance as a tool for educators seeking to Chapter 19 Action Research 489 FABLE 19.1 Characteristies of Action Research Compared to Formal Research Research Overview __Action Research Formal Research Purpose Solve a local problem of practice Produce generalizable knowledge Focus of study A problem or goal related to one’s own problem or question of concern to practice educational researchers, Topic selection Limited review of research literature, Extensive review of research literature, ‘emphasizing secondary sources emphasizing primary sources Researcher Charactaristics Researcher afiliation One or more school-based educators, perhaps in collaboration with university, University researchers faculty Researcher Practical experience with the problem; Substantial knowledge of the research qualifications basic knowledge/experience in esearch _ literature and training in research methods Research Characteristics Sample selection Convenience sample of one’s own clients/ Random or representative sample from a students defined population Research method Easily implementable procedures, Rigorous research design and controls, long emergent design, short time frame time frame Measures Simple or available Selection based on evidence of validity and reliability Data collection and Emphasis on descriptive statistics and Emphasis on in-depth qualitative coding analysis practical significance of the results and interpretation or on tests of statistical significance Research report Informal sharing with colleagues or Published report or formal presentation at a publication through an online network conference Application of results Make changes to one's practice if the ‘Add to the knowledge base of education findings justify them improve some aspect of their practice. As the table shows, action research has a different cientation, and is typically simpler, thn formal research in every respect. Examples of Action Research Studies ‘Action research studies are done by practitioners in various professional disciplines, includ- ing education. Educators might conduct studies for personal reasons orto fulfill degree and licensure requirements. They might test a theory of their own or draw on other theoreti cal perspectives. Critical esearch, which advocates democratic changes in education (see Chapter 16), is one such theoretical perspective (Carr & Kemmis, 1988). The following sections present several examples of action research studies done by teachers. Bullying in Middle School Drosopoulos, J. D., Heald, A. Z., & McCue, M. J. (2008). Minimizing bullying behavior of middle school students through behavioral intervention and instruction, Chicago: St. Xavier University. ERIC Document Reproduction Service No. EDS00895, 490 Part VI_ Using Other Research Methodologies to Study Problems of Practice Three teachers conducted this study as a master’s degree requirement. They initiated the study by administering questionnaires to students and parents about the incidence of bullying, Among other findings, they learned that half of the students had been bullied at least once. ‘The teachers experimented with various interventions to reduce the incidence of bul- lying, including some that directly involved students as research participants. For example, the students created and posted antibullying posters in bullying hot spots in school, and they ‘wrote and performed an antibullying rap song in class. ‘The teachers made direct observations of students at various sites inthe school to deter- mine the incidence of bullying before and after these interventions. They found that various types of bullying decreased by 31 percent overall, and name-calling decreased the most. ‘Teachers presented their own perceptions of the change. For example, one teacher stated, Once they were taught the definition of bullying, the many different kinds of bullying, and hhow itcan cause life long mental sears, most [students] were ashamed of the actions they had taken against other people at some point in the past... . [NJow that T know the impact this intervention program has had on students, I will implement it every semester for the est of, ‘my teaching career. (Drasopoulos etal, 2008, p. 70) ‘The teacher researchers submitted their paper to ERIC, and the full ext is available on its website (www.eric-gov) The Needs of College Students in China Schippers, M. (2008). Student support in China: Addressing the perceived needs of undergraduate English department students. ERIC Document Reproduction Service No, ED499780. ‘This action research study was conducted by Margriet Schippers, a Western-culture instructor who had been teaching at a Chinese university for 10 years. To learn more about her students’ needs she initiated her own study by taking the role of a teacher-researcher. ‘Combining both quantitative and qualitative methods, she interviewed a sample of students and administered a questionnaire to another sample. ‘Schippers found that the students expressed needs for study skills, including time man- ‘agement, and guidance on how to adapt to college life, develop confidence, and plan for a ccareer. The students expressed dissatisfaction with current university efforts to foster their ‘emotional and social development and withthe university's use of monitors (teacher aides). ‘These findings provided the basis for Schippers to propose action in the form of offering a ‘new student support course that would teach “teamwork (students leam about role differentia- tion and find out their preferred... team role, Gardner's Multiple Intelligences (students learn ‘more about their strong and weaker points), basic counseling skills, study skills such as time ‘management skills and note taking, presentation skills, job interview techniques, Maslow’s hierarchy of needs, the reflective diary tool, and goal setting” (p. 48). Schippers also noted a personal benefit—“professional development for me as a teacher in that I have become more aware of students’ wishes and weaknesses that need addressing in class” (p. 19). This study illustrates the fact that action research need not involve an immediate change {in professional practice. An action research study can involve careful analysis of an existing situation to discover needs and opportunities that set the stage for subsequent design of an intervention and application in the context of professional practice The Effectiveness of Reciprocal Teaching Holt, C. (2008). Does reciprocal teaching increase student achievement in Sth ‘grade social studies? Retrieved December 18, 2000, from http://actionresearch altec.org Chapter 19 Action Research ‘The teacher-researcher, Crystal Holt, conducted an action research study in her class- room of students who were struggling with social studies because of poor reading skills. She decided to try a teaching technique called reciprocal teaching, which had been found to be effective in formal research studies. In this technique, the teacher and students take turns in leading conversations about the meaning of text passages. The leader makes efforts to summa- rize the text, ask questions about it, clarify its statements, and predict what will come next. Holt conducted an experiment with her 22 students in which half the students read text- book material, listened to the teacher talk about it, and took whole-group notes. The other half read the same textbook material but engaged in reciprocal teaching. She consulted with three other teachers in designing the reciprocal teaching intervention, Holt found that the reciprocal teaching group made much greater gains from a pretest to a posttest on social studies than did the conventional instruction group. Moreover, she achieved en insight into her professional practice: Before doing this research, Thad a lot of problems with students talking out and making inap- ‘propriate comments during instructional time. Now thet I have implemented the reciprocal teaching technique into my daily routine I heve noticed that the inappropriate talking has ceased while higher-order thinking and comments are coming more and more into play. truly believe that the students behave better when given an outlet to talk, ‘These comments illustrate an important feature of action research. Although one can read about the benefits of a technique like reciprocal teaching in professional publications, ac- tion research makes these benefits tangible by allowing researchers to see them occurring. Jn their own work site Design Features of Action Research (Our explanation of how to design an action research project draws on the model of action research developed by Jeffrey Glanz (1998). To his six steps we have added a final step, namely, reporting the findings of action research. Action researchers do not always perform the steps inthe order presented here. Sometimes action researchers begin by taking a new action before collecting any data (Schmuck, 1997). Similarly, while in Glanz’s model re- flection follows action taking, reflection is appropriate at various points during a project. ‘Some models of action research describe a definite beginning and end, This description is generally accurate when action research is carried out as part of a degree or course re- quirement. However, when educators incorporate action research into their everyday work, it is more likely to become an ongoing cycle of activity ‘We describe the typical steps in an action research project by referring toa study con- ducted by Wallace Shilkus (2001). Step One: Selection of a Focus for the Study Shilkus (2001), an industrial arts middle schoolteacher, had recently returned to graduate education as a student after 17 years of teaching, Shilkus was “curious about the ways in Which the industrial arts were relevant to middle school students” (p. 143). He had several goals for his action research project, which he undertook as part of coursework toward a raster’s degree, Specifically, he wanted to explore tesching methods to activate students’ multiple intelligences and explore different ways to reach all students, including hard-to- motivate students Undertaking his research at a time when industrial arts was being cut from many schools’ curriculum offering, Shilkus also wanted to demonstrate the contribution of in- dustial arts to students’ intellectual development, especially in middle school. He wanted to help others become aware of the importance of the “endangered subject” (p. 144) of, industrial arts 491 492 Part VI_ Using Other Research Methodologies to Study Problems of Practice In his action research study, Shilkus used both cross-tutoring and peer-tutoring ap- proaches in guiding students’ design and construction of CO,-powered race cars. He ar- ranged to have his seventh- and eighth-grade industrial ars “veteran” students first tutor fourth-grade “rookies” at his school and then the adults in his own graduate education class. Step Two: Data Collection Shilkus's middle school students, their fourth-grade “rookie” partners, and his graduate school classmates all engaged in providing data for the action research project and in shar- ing their discoveries to learn and help others learn. Inhis transportation technology classes Shilkus had observed that a wide range of stu- dent abilities was needed to produce a successful CO,-powered race cer. He found Howard Gardner's theory of multiple intelligences (Gardner, 1983) helpful in defining the types of skills students would need to lear to experience success in the class. Shilkus reports, “Student journal keeping assisted me ... in identifying students’ strengths and weaknesses” (2001, p. 145). This process was important, because he wished to ensure pairings of stu- dents who would be compatible with each other. Also, Shilkus continually observed the classroom activities and administered pre- and postsurveys to al tree groups of research participants. ‘After analyzing each set of data, Shilkus added other data-collection strategies. He asked his adult classmates to make journal entries describing the classroom atmosphere. ‘Also, he made a videotape of a two-hour lab in which the middle school students tutored the adults in design and construction of their race cars, as well as a subsequent videotape of an elimination car race forall participant, Step Three: Analysis and Interpretation of the Data Shitkus wrote descriptions of students’ design, construction, and tutoring activities in clas, with specific examples: ‘In many cases, saw a marked improvement in class participation and behavior. Students were forming a tutoring system in the classroom too, not just with their tutees. I observed many students who asked for additional work; they wanted and enjoyed helping Someone else. Iwas ‘witnessing a form of community being born. (2001, p. 146) tthe elimination car race, “some members of my middle school class commented that the fourth-grade skeiches were better than the adult sketches” (p. 148). postacivity question- naire revealed that participants liked making the cars more than designing them, ‘Adults reported insights that they gained from being in the role of student in an un- familiar context. For example, the comment of a home economics teacher in Shitkus's araduate education class confirmed for him the importance of teachers being able to put thetnselves in their students’ shoes when they design instruction; “know now some of the frustrations my kids must fel in my sewing class... .As much as I like to think Ican see things through a student's eyes, it's refreshing to be proved wrong on this point!" (2001, pp. 148-149), Shilkus concluded from his research project that basing his class teaching activities on the multiple intelligences framework helped him reach more of the hard-to-motivate students, promoted students’ socialization skills, and appealed to their varied learning styles. Step Four: Taking Action ‘Action research typically involves making changes in one’s behavior and observing the consequences, As described, Shilkus engaged in a variety of actions: using both cross tutor- ing and peer tutoring, having veteran students tutor @ fourth-grade class of “rookies,” and. hhaving the veterans tutor the adults in his own graduate education class. Shilkus coached Chapter 19 Action Research 493 his veteran students in their role as tutors, and helped the “rookies” and the adults take the role of industrial arts students, a role with which most of them were unfamiliar, Step Five: Reflection Reflection is a process in which educators step back from the fast-paced and problem-filled world of practice in order to ponder, and possibly share with others, ideas about the mean- ing, value, and impact oftheir practice. This type of reflection can lead educators to make ‘new commitments, discover new topics to explore through action research, and achieve new insights into the strengths and weaknesses of their current practice. ‘Shilkus does not specifically mention his use of reflection in his report, but he gives ‘many examples of how his thought processes changed during, and after, the action research project. For example: {Pe noticed changes in myself and my students as a result ofthis project. Presenting informa- tion in different manners has made it possible to reach more students, not just the students ‘who excel in traditional classes can only hope my students learn as much as I did during this project. (2001, p. 149) David Hobson (2001) views journaling as a critical tool for generating a “writen re- cord of practice” (p. 19). Figure 19.1 provides our summary of his suggestions for keeping an action research journal as a basis for reflection FIGURE 19. 1 Suggestions for Keeping an Action Research Journal Use 8 %4" x 11" pages and put them in @ 3ring binder so that pages can be removed, added, or rearranged, For ease of carrying, you might prefer a 6° x9" binder, blank lesson plan book, Post-It note pad, or spiral-bound notebook. 2, Date and time each entry to facilitate viewing developments processes over time. Start each entry on a new page, so that the pages can be grouped to reflect recurring pattems or reconstruct sequences. 3. Make time for journaling by picking 2 regular time free of interruption or writing in class while your students are writing 4, Use descriptive writing to record directly observed/experienced details for later 5. Use reflective writing to comment, associate, and make meaning 6. Use double-entry journal writing with @ description in one column and reflection in the other column 7. Keep a daily log to help reveal priorities and to note what absorbs your attention and what continuing issues predominate. 8. Name each important teacher you have known, and describe each one. Look for commonalities; describe the stepping-stones in your experience of teaching and refiect on your development over time. 9. Examine the materials you have been reading and bring the results of your investigations into your journal. 10. Develop a “journal of the journals,” going back through your entire journal to seek themes and highlight passages, and have a friend read aloud the lines you have highlighted. 11. Ask your students to turn in "exit lips" at the end of each class reflecting on their learning, questions, and expectations. Source: Adapted from Hobson, D, 2001), Action and reflection: Narative and journaling in teacher research. In G. Burnaford, J. Fischer, & D. Hobson (Eds), Teachers doing research (2nd ed, pp. 7-27). Mahwah, NU: Lawrence Erlbaum. 494 PartVI_ Using Other Research Methodologies to Study Problems of Practice « Step Six: Continuation or Modification of Practices ‘The postactivity survey showed that Shilkus’s research participants preferred making the race car to designing it. Therefore, he decided that he needed to make the design of the race car “even more interesting for my students” (2001, p. 148). His observations of the quality of his students’ suggestions for the race car design and competition led Shilkus to add a critique unit to all his car-racing classes. From the students’ critiques he learned that they appreciated seeing a finished product before they began their own car design and construction, Therefore, he changed his teaching unit to allow students to examine the cars being designed by others as they progress. This activity provides feedback and ‘motivation to all the students, both those designing the cars and those observing the design process. Step Seven: Preparing a Report of the Findings Shilkus's action research report is one of a number of detailed examples in Teachers Doing Research (Burnaford, Fischer, & Hobson, 2001). Its thus a polished, published example of action research and illustrates the value of an action researcher being part of a professional development program. Through his program Shilkus was able to interact with, lear from, and include as research participants his graduate student colleagues. He also received guid- ance from the course instructors, who edited the book in which Shilkus’s research report is published. Reports of action research projets can be disseminated in other media besides jour- nals and books. Cochran-Smith and Donnell (2006) observe that action research has led to “new ways to store, retrieve, code, and disseminate practitioners’ inquiries in the form of CD-ROMs, Web sites, and other electronic innovations as well as new modes of public presentation and publication, such as multivoiced conversations, readers theater, poetry, and so on” (p. 512). How Action Research Differs from Educators’ Other Approaches to Problem Solving ‘Table 19.1 describes how action research differs from formal research, Here we consider briefly how action research differs from educators’ typical approaches to solving problems of practice. Typically teachers talk to colleagues, attend workshops, pick up ideas from professional magazines, or rely on their own hunches as a guide to trying something new in their practice. ‘The question must be asked: Do the ideas derived from these approaches actually affect student learning? Mixed evidence from the National Assessment of Educational Progress and the No Child Left Behind Act as to the overall quality of student learning inthe United States suggests that such approaches, despite their popularity, are not highly effective. Ac- tion research possesses two important features beyond those of most other approaches to solving problems of practice—systematicity and intentionality Cochran-Smith and Donnel (2006) have used these concepts to clarify the unique char- acteristics of practitioner research, including action research. They describe systematicty as involving “ordered ways of gathering and recording information, documenting experi ences inside and outside of the contexts of practice, and making some kind of written record... [and] ordered ways of recollecting, rethinking, and analyzing events for which there are only partially written records” (p. 510) Intentionality, in turn, “refers to the planned and deliberate rather than spontaneous nature of practitioner inquiry” (Cochran-Smith & Donnel, 2006, p. 510). Many teachers frequently examine assignments, test results, and the other forms of student documents routinely generated in schools. Action research encourages teachers to generate additional Chapter 19 Action Research data to look specifically at what they do and what results they observe in students from ‘what they do, They thus can gather new data on student learning and highlight the specific relationships between their varied teaching choices and student learning. In some reports of action research, the authors limit their description of findings to statements that students “responded positively” or “did better” after a new action was un- dertaken, To have the greatest impact on their own and others’ practice, we recommend that in reports of action research authors explain specifically the actions they took, as well as the procedures of data collection, analysis, and interpretation that provided the basis of the results they obtained. Purposes and Benefits of Action Research ‘Action research is sometimes regarded as having three main purposes—personal, profes- sional, and political (Zeichner & Noffke, 2001). In practice, as reflected in the set of actual research studies cited in this chapter, most educators carry out action research for reasons that appear to blend these purposes. Below we briefly summarize some of the benefits of action research, referring back to Shilkus’s (2001) study to illustrate each one. 1. Contribution tothe theory and knowledge base that educators need to enhance their practice. Educators who carry out action research learn to reconstruct educational theory and findings in terms that are understandable to them, Based on this understanding, they can develop more effective practices in their work settings. Shilkus’s use of cross tutoring and peer tutoring illustrates his application of new concepts to his teaching repertoire. Educators who have the opportunity to try out new teaching strategies can in turn contribute to the tesching profession and to the education research literature, For example, Madeline Hunter (1994) was the director of the laboratory school at the University of California, Las Angeles, when she began informally experimenting with ways to improve teachers’ classroom instruction. Her individual creativity resulted in the development of a method of instruction called ITIP (Instructional Theory into Practice), which has had ‘a major impact on teaching practice and stimulated many formal research studies of its effectiveness. 2. Support of the professional development of educators. Shilkus’s report shows how his action research increased his competence in applying research findings, camying out research himself, and reporting the research. He not only developed needed skills in doing research, but he also improved his ability to read, interpret, and apply the research of others. Furthermore, the education course that he completed while doing his research contributed to his completion of a master's degree. 3. Build a collegial networking system. Collaborative action research is carried out by two or mote educators involved in the same type of practice (¢.g., K-12 teaching) o from more than one type of practice (e.g., K-12 teachers and university teacher educators). ‘The collaboration might also extend to the clients for whom the research activities are intended, For example, Shilkus’s research collaborators included his own middle school stu- dents, the fourth-grade student “rookies” and their teacher, and his graduate education classmates, They were involved not only as research participants but also in the design and. conduct of the research itself. Shilkus thus built a rich communication network, reducing the isolation often experienced by individual teachers and providing opportunities for future collaborative work. 4, Help identify problems and seek solutions in a systematic fashion. Shilkus's action 1e- search required that he and his students and teaching colleagues define problems of practice clearly, identify and try out possible solutions systematically, and reflect on and share the 495 496 Part VI_ Using Other Research Methodologies to Study Problems of Practice results of their efforts. Thus, systematic action research enables educators to break out of the rut of institutionalized, taken-for-granted routines. It generates hope and motivation to solve seemingly intractable problems in the workplace. 5. Can be used at all levels and in all areas of education practice, Shilkus’s study not only cut across two levels of schooling (fourth-graders and his middle school students), but also included both his school and the university where he and his classmates were studying, ‘Action research can be carried out in specific classrooms or departments, throughout an educational institution, or atthe regional, national, or international level ‘Applying Action Science to Action Research ‘Action science, a theory developed by Chris Argyris and Donald Schiéin (1974), can help action researchers design their action research to produce effective change in their practice. ‘Action science is an approach to help educators discover and reconcile differences between their espoused theory, that is their beliefs about how they deal with specific problems of practice, and their theory-in-action, that is, their actual behavior as they engage in their work, For example, suppose that Sophie, a ninth-grade history teacher, believes that she fos- ters higher-order thinking when questioning her students. If in fact she actualy asks mostly kanowledge and comprehension questions, she is not tapping the higher cognitive levels that are specified in models of thinking (e.g, Bloom, 1956; Wiggins & McTighe, 1998). ‘Action research could help demonstrate this discrepancy and the problems it causes. ‘Suppose that Sophie undertook action research to improve her questioning strategies. She ‘ould collect data on both her specific ations and students’ responses to them. For example, she could videotape herself as she carries out a lesson for subsequent self-analysis—a method known as microteaching that was once widely used in teacher education (MacLeod, 1995). She could il out an observation scale to assess the extent to which she carried out the actions that she intended to carry out. She could also ask colleagues or teacher educators from a university to observe and give her feedback. Sophie could look specifically for instances that reveal differences between her es- ppoused theory and her theory-in-action. Then she could take new actions to bring her be- havior closer to her beliefs about what she wants to accomplish, ‘A study by Thomas Hatch (1998) illustrates the value of action science in identify. ing problems of practice. In 1991 the New American Schools Development Corporation issued a request for proposals. In response, four large-scale school-improvement organiza- tions formed a collaborative effort called the Authentic Teaching Learning and Assessment for All Students (ATLAS) Communities Project. Their stated goal was to effect change in schools nationwide. Hatch outlines major differences in the espoused theories of the leaders of these organizations, which contributed to very different approaches to basic schooling dilemmas. Hatch concludes that these differences “made it extremely difficult to make decisions and to carry out the collaborative work that school improvement required” (p.24), The Insider-Outsider Issue in Collaborative Action Research ‘Whether the promise of action research is realized depends to a great extent on the partici- pants in action research projects. Some educators who engage in action research (typically, K-12 teachers) view themselves as insiders, tha is, individuals with an internal perspec fe on the problems of practice being studied. They might argue against collaborating Chapter 19 Action Research ‘With outsiders, such as university professors or researchers from a research organization, ‘who are viewed as having an external perspective on those problems. Stephen Kemmis and Robin McTaggart (2000) argue that reliance on the interpretations of outsiders might disempower teachers and might imply that “outsider” research is more valid than teacher research In a subsequent publication, McTaggart (2003) addresses the insidesoutsider issue froma different perspective. He argues that scholars such as university professors can best add value to K-12 educators’ practice through participatory action research endeavors MeTaggart asserts that educational practice involves not only K-12 teaching, but all the major functions of various educational systems or institutions. He explains that all educa tors engage in some aspect of practice and can affect other aspects of practice through col- laboratve efforts. Therefore, he argues that to be effective, the action undertaken through action research must affect all the critical aspects of educational practice: curriculum, administrative practice, teacher education, and the conduct and publication of educational research, McTaggart (2003) identifies three ways in which scholars such as university professors, who would be considered outsiders in relation to K-12 teaching, can contribute through ac- tion research to solving K-12 teachers’ problems of practice. First, he sees such scholars as able to help ensure that action research insiders adequately test the credibility of evidence in suppor of particular esearch findings and interpretations. This involves “an ongoing sociopolitical process... in situ with participants” (p. 9) that depends on systematic data collection and analysis. Second, McTaggart urges scholars to suppor insiders and research participant in ex- pressing ther feelings about what is being studied, which are often strong and can help or hinder subsequent action. McTaggart argues that outsiders can also help insiders determine Which of those feelings are justified and which are better to reframe or set aside in the in- terests of social change. Third, McTaggart sees scholars as able to contribute critical support for the develop- ‘ment of personal political agency among research participants and provide critical mass for a commitment to change. He admits, though, that outsiders’ typical privileged insti- tutional settings may have deskilled them for such collaborative roles. He nonetheless asserts that if they will “insinuate themselves into political life” (p. 14), they wall gradu- ally be able to become equal players with insiders in a taly participatory action research process. Evaluating the Credibility and Trustworthiness of Action Research Projects ‘Action researchers need to consider ways to design and carry out their research so thatthe resulting actions, and the reports on them, are credible and trustworthy both to the research- crs and to others. Some action rescarch projects rely primarily on quantitative research designs. You can evaluate these projects using the criteria shown in Appendix 1 and the design-specific criteria for descriptive research (see Chapter 10), group comparison re- search (see Chapter 11), correlational research (see Chapter 12), and experimental research (see Chapter 13). Other action research projects reply primarily on qualitative research designs. You can evaluate these projects using the criteria shown in Appendix 3. The suggested strategies for evaluating case studies (see Chapter 14) are also relevant. In addition, you can apply five validity criteria developed by Gary Anderson and Kathryn Herr (1999) specifically for ‘evaluating action research studies: (1) outcome validity, (2) process validity, (3) democratic validity, (4) catalytic validity, and (5) dialogic validity. We describe each validity criterion next. We also cite actual research studies and refer to specific strategies used in each study 1 increase that type of validity. 497 498 Part VI_ Using Other Research Methodologies to Study Problems of Practice ‘As you read about these criteria, keep in mind that you may encounter action research projets that are not presented in a complet, formal report. You can overcome this limitation by interviewing the action researcher—and perhaps colleagues and clients, too—in order to learn more about the project. That process should give you the necessary information to evaluate a specifi action research study using these or other validity criteria, Outcome Validity Outcome validity concems the extent to which actions occur that lead to a resolution ofthe ‘problem that prompted the action research study. Rigorous action research, of course, seeks not only to solve a specific problem. It also aims to help researchers reframe the problem in a more complex way, which often leads to a new set of questions or problems to be ad- dressed, Thus, this criterion also stresses the importance of reflection and the continuing introduction of new actions to address ongoing or emerging problems. Barbara Levin and Tracy Rock (2003) studied five pairs of preservice and experi- enced teachers who engaged in collaborative action research projects during the preservice teachers’ professional development internship in the experienced (mentor) teachers’ class- rooms. Levin and Rock analyze interviews with, and written and audiotaped reports from, the participants. They present findings about the costs of collaboration (for example, time pressures, dependence on someone else to fulfill one’s responsibilities, or limited access to one’s mentor). As important, they demonstrate that the project had beneficial outcomes for the preservice teachers (for example, generating their mentors’ perspective, support, and feedback). Process Validity Process validity concerns the adequacy of the processes used in different phases of an ac: tion research project. Framing and solving problems ina way that promotes the researchers’ ‘ongoing learning is one aspect of process validity. Triangulation (the inclusion of multiple perspectives or data sources) also contributes to process validity. If the action research ‘project is reported through narratives such as poems, folktales, or anecdotes, readers need to know whether they depict accuratcly what occurred, rather than being purely subjective accounts or interesting exaggerations. ‘Martha Stevens (2001) carried out action research while teaching mainstreamed sixth- agradors with learning disabilities. She used a continual process of exploration of students’ Tearning, and corresponding change of her actions, to improve the learning environment. For example, she modified the curriculum to encourage greater use of self-management by students, Based on her review of research on teaching reading to students with learning disabilities, she also scaffolded information in the regular education texts for her students by recording it on audiotape and rewriting the assignments at a simpler reading level. The adequacy of these processes is shown in six years of data reflecting good student progress in both reading and writing skills, including improvements in test scores and grades both in her classes and in mainstreamed settings, Democratic Validity Democratic validity refers to the extent to which an action esearch project is done in col- Inboration with all the parties who have a stake in the problem being investigated, It also involves determination of whether the multiple perspectives and material interests of all stakeholder groups have been taken into account, Here multiple perspectives are viewed not as a basis for triangulation of data sources but as an issue of ethics and social justice. Shilkus’s action research (2001) provides an outstanding example of including mul- tiple perspectives in the problem of making industrial arts instruction an active learning ‘process forall participants. He included as datasources, and actually as coresearchers, not just his middle schoo! students, but also fourth-grade “rookie” students and his graduate education classmates. Chapter 19 Action Research Catalytic Validity Catalytic validity involves the extent 10 which an action research project rerients, focuses, ‘nd energizes participants so that they become open to a transformed view of reality in relation to thei practice. Action researchers strengthen this aspect of validity by keeping @ research journal 1 record their reflections and changing perceptions. This criterion also addresses the extent to which practitioner research realizes an ema cipatory potential. In other words, catalytic validity addresses an action research project's success in fostering the widespread engagement of educators and education stakeholders in an active quest for ending oppression and promoting social justice. For example, Anchalee Chayanuvat and Duangta Lukkunaprasit (1997), two English- language instructors at a university in Thailand, conducted action research to help enhance the English-language skils of gifted students entering their university. Their findings in- cluded recommendations fora special English class for gifted students, with ‘more emphasis on speaking and writing, inclusion of external reading materials which are more difficult and challenging, and exploitation of students’ learning activities outside class in our English program, eg. an oral discussion following the watching ofan assigned film. (p. 164) CChayanuvat and Lukkunaprasit provide evidence thatthe provision of this type of class helped emancipate their students by giving them increased opportunities to express their giftedness and make learning gains not otherwise possible. Dialogic Validity Dialogic validity refers to the use of extensive dialogue with peers in the formation and review of the action researcher's findings and interpretations. It can be met by doing ac- tion research collaboratively. [tis also enhanced by the researcher engaging in critical and reflective dialogue with other researchers or with a“crtical friend” who serves asa devil's advocate for alternative explanations of research data, Efforts to ensure the “goodness-of- fit’ of the action research problem and findings withthe intuitions ofthe practitioner com- munity also improves dialogic validity. Stevens's (2001) action research projects with her mainstreamed sixth-graders involved an impressive amount of dialogue between her and her students, the students other teachers, and the students’ parents, The other teachers’ positive responses to her program outcomes and materials also reflect “goodness-of-fit” with the teaching community in her middle school. An example of HOW ACTION RESEARCH CAN HELP IN SOLVING PROBLEMS OF PRACTICE Students and others often question the value of homework, as illustrated in these excerpts from a recent newspaper article: For years, students, parents, teachers and Ph.O.s have debated the value of homework in genera, yelding mixed but impassioned opinions. When it comes to giving home- work over Thanksgiving and other school breaks, opinions are equelly mixed—and passionat Some teachers, such as English teacher [BNI, don't give anything beyond extra credit or makeup work Others, such as [SC], a second-grade teacher... say they assign homework over breaks for students’ own good. Schencker, L (2008, November 2). Schoo!’s out, but homework’s not. Salt Lake Trbune. Retrieved from www strib.com 499 500 Part VI_ Using Other Research Methodologies to Study Problems of Practice “The article also mentions parents, students, end professors who are either for or against assigning homework to be completed during holiday breaks from school. They express a surprising range of opinions about this issue. This controversy can stimulate an action research project, as we illustrate. Suppose you are a teacher with the choice to assign homework over a holiday break. You know that students are expected to be prepared for school tests and for state competency examinations. On the pro sie, you believe that homework helps them review and extend their learning beyond what you and your students cover in class. On the con side, you feel that you and your students need a tre break during holiday periods. ‘To help you decide on a strategy you could frst review some research on homework as to the pros and cons of assigning homework at all, and specifically during school breaks. ‘You could also talk to other teachers in your school and get their opinions. In the process, suppose that you discover another teacher at your grade level (loan) who also wonders about the value of homework. You could do collaborative action research with Joan, in one of her classes and one of your classes. Your goal could be to determine whether assigning homework over school breaks is a good idea and if so, what type of homework is most appropriate. ‘You and Joan could design a questionnaire asking students to rate their attitudes about homework in general and over holiday breaks. It could also ask students to rate different types of homework as to how positively or negatively they would feel about having each type during a holiday break. To help you find out more about individual students’ attitudes, you could also include room for student comments on the questionnaire, If you ask students to sign their ques- tionnaires, you could look for differences between the responses of higher-achieving and lower-achieving students ‘When the next holiday break comes, you could give no homework while Joan tries out an approach to homework that stadents rated. You could then have students in your class rate how positively they felt about getting no homework while the other class rates the ap- proach Joan used. You could also give an examination on the material covered before the holiday break and compare the performance of the students who got homework during the ‘break with the performance of students who had no homework. ‘The findings from this action research might help you and Joan develop a homework policy that fosters student learning. You and Joan could also share the results with your students, which might reduce their resistance to homework you assign in the future. Che ats 1. Action research has all the following purposes except 3. Reflection by action researchers 4. supporting the professional development of practitioners. a. is particularly important at the stat of an action research '. building theory and generalizable knowledge, project. , building a collegial networking system among educator. ', occurs primarily during date analysis and interpretation. 4. helping practitioners identify problems and seek €, involves pondering the meaning, value, and impact of solutions systematically. ‘one’s actions. 2, The quality of an action esearch project is least dependent . requires dialogue withthe research participants onits 4, The problem to be addressed by an action research project . use of well-designed methods of data collection and typically is identified by analysis. ‘educators’ consideration of obstacles to achieving their », promotion of collaboration between the researcher and ‘work goals. his or her colleagues. ', roviews ofthe education literature. ¢. contribution tothe knowledge base for education, ©, asystematic needs assessment 4. impact on the researcher's practice. 4. consultation with outsider. '. Action researchers who publish their studies in the research literature are primarily motivated by the desire to present generalizable findings to the widest possible audience ». encourage other educators o undertake action research ‘on problems of practice «. demonstrate the rigor that ation research can involve. 4. enance their status in their local educational context 6, For an action research project to be considered a success, itis important that che researchers 1, receive extensive preparation to develop their research knowledge and skills. », review the education literature before designing the action to be taken, «. discuss the theoretical implications of their results. 4, apply the findings to their own practice. 17, According to Robin McTaggart, scholars can best contribute to collaborative ation research with K-12 educators by ‘promoting adherence to positivist principles of objectivity. ». providing structure forthe research design ©. providing critical mass for a commitment to change. 4. Allof the above. Chapter 19 Action Research S01 8. Considering the multiple perspectives and interests ofall stakeholders in an action research project is a strategy that iretly increases its validity a. democratic b. catalytic «. dialogic 4. process {9, Educational action research differs most from formal research in 8 its focus on improving teaching and leaning. bits use of deliberate inquiry and ordered ways of gathering and recording information, «. its conduct of research on aspects ofthe researchers" own practice. 4. its applicability to all levels and types of education practice. 10. Action science can best inform action research with respect to 4. identifying and resolving discrepancies between participants’ espoused theories and theories-n-action. b. determining the appropriate participants fr collaborative action esearch, . enabling insiders to free themselves from the disempowerment of working with outsiders 4. increasing the political impact of research findings Chapter Reference: Anderson, G. L., & Herr, K. (1999). The new paradigm wars: Is ‘there room for rigorous practitioner knowledge in schools and uni- vetsities? Educational Researcher, 28(3), 12-21, 40, Axgyris,C, & Sch, D. A. (1974). Theory in practice: Increasing professional effectiveness. San Francisco: Jossey-Bass. Bloom, B. S. (Ed). (1956). Taxonomy of educational objectives: Classification of educational goals. Handbook 1: Cagnitive do- ‘main. New York: Longman. Burnaford, G., Fischer, 1, & Hobson, D. (Eds, (2001). Teachers doing research (2nd ed.). Mahwah, NJ: Lawrence Eribaum. Carr, W., & Kemmis, S. (1988). Becoming evtical: Educational knowledge and action research. London: Falmet. CChayanuvat, A., & Lukkunaprasit, D. (1997). Classroom-centered research at Chulalongkorn University Language Institute. In S. Hollingsworth (Ei), International action research: A casebook {for educational reform (pp. 157-167). London: Falmer. Cochran-Smith, M., & Donnell, K. (2006). Practitioner inquiry Blurring the boundaries of research and practice. In J. L. Green, G. Camilli, & PB. Elmore (Bds,), Handbook of complementary ‘methods in education research. Washington, DC: American Fdu- cational Research Association Cochran-Smith, M.,& Lytle, SL. (1999). The teacher research, movement: A decade late. Educational Researcher, 28(7), 15-25. Drosopoutos, J.D, Heald, AZ, & MeCue, MJ (2008). Minimis {ng bullying Behavior of middle school students through behavioral intervention and instruction. Chicago: St. Xavier University. FRIC Document Reproduction Service No. EDSO0895. Gianz, J. (1998). Action research: An educational leader's puide to school improvement. Norwood, MA: Christopher. Gordon. Hatch, T. (1998). The differences in theory that mater in the prac- tice of school improvement. American Educational Research Jour: nal, 35,3-31 Hobson, D. (2001). Action and reflection: Narrative and journal- ing in teacher research. In G. Burnaford, J. Fischer, & D. Hobson (E4s,), Teachers doing research (2nd ed., pp. 7-27). Mahwah, NI Lawrence Erlbaum. Holt, C. (2008). Does reciprocal teaching increase student ‘achievement in Sth grade social studies? Retrieved from http! actionresearchaltec.org Hunter, M. (1994). Enhancing reaching. New York: Macmillan. Kemmis, $., & McTaggart, R. (2000). Participatory action re search, In N. K, Dengin & Y. S. Lincoln (Eds.), Handbook of ‘qualitative research (2nd ed, pp. 567-605). Thousand Oaks, CA Sege. Levin, B. B, & Rock, T. C. (2003). The effects of collaborative action research on preservice and experienced teacher partners in professional development schools. Journal of Teacher Education, 54, 135-149. Lewin, K. (1946). Action esearch and minority problems. Journal ‘of Social Issues, (8), 34-46, 502 Part VI_Using Other Research Methodologies to Study Problems of Practice MacLeod, G. (1995). Microteaching in teacher education. In L.W, Anderson (E4), International eneycolopedia of teaching and teacher educarion (2nde4, pp. 573-S77). Tarrytown, NY: Elsevier Science. ‘McTaggart, R. (2002). Action research scholar: The role of the scholar in aetion esearch. In M. P. Wolfe & C. R. Pryor (Eds), ‘The mission ofthe scholar: Research and practice (pp. 1-16). New ‘York: Peter Lang, Reason, P., & Bradbury, H. (Eds.). (2001). Handbook of action research: Participative inquiry and practice. Thousand Oaks, CA: Sage Schippers, M. (2008). Student support in China: Addressing the perceived needs of undergraduate English department students [ERIC Docurnent Reproduction Service No, ED499780. Schmuck, R.A. (1997) Practical action research for change. At- lington Heights, IL: IRUSkylight. Shitkus, W. (2001). Racing to research: Inquiry in middle school {industrial arts. In ©, Burnaford, J. Fischer, & D, Hobson (Eds. ‘Teachers doing research (2nd ed., pp. 143-149). Mahwah, NF: Lawrence Exbaum, Stevens, M. C. (2001). Laptops: Language ats for stents with learning dissbiliies: An action research curriculum development project, In G, Burnaford, J. Fischer, & D. Hobson (Eds.), Teachers doing research (2nd ed, pp. 157-170). Mahwah, NJ: Lawrence Enbeum, Wiggins, G., & MeTighe, J. (1998). Understanding by design Alexandria, VA: Association for Supervision and Curriculum Development ‘Zeichoer, K. M., & Noflke,S. E (2001). Practitioner research. In V. Richerdson (Ba), Handbook of research on teaching (4th ed, pp. 298-330). Washington, DC: American Educationst Research Association, Resources for Further Study ‘Burnaford, G., Fischer, J., & Hobson, D. (Bds.). (2001). Teachers doing research 2nd ed.) Mabwab, NI Lawrence Exlbaum, Presents guidelines for the conduct of teacher research in local, university, and national or international settings. The many detsiled examples of teachers doing rescarch are well esigned, cleanly writen, and inspiring. Fishman, S.M., & McCarthy, L. (2000). Unplayed tapes: A per- sonal history of collaborative teacher research, Urbana, IL: Na- tional Couneil of Teachers of English, ‘Summarizes the authors' experiences in conducting and guid- ing others in the conduct of “insider-outsider” collaborative teacher research Marion, R, & Zeichner, K. (2001). Practitioner resource guide for ‘action research, Oxford, OH: National Staff Development Coun- cil, ERIC Document Reproduction Service No, ED472207. ‘A compilation of resources for practitioner research, includ- ing information about practitioner research networks, jour- nals, and online sites presenting action research studies, funding sources, and a bibliography on sourees for designing, section research, Sample Action Research Study Student-Generated Discussion in the Senior Secondary English Classroom Lehmann, A. W. (2000), Student-gensrated discussion in the senior secondary English classroom. Networks: An On-line Jourral for Teacher Research, 32). Available at http://journals lorry misc edu/indexphp/networks/artcl/view/142/141 ‘The following journal article reports # qualitative action research study that was published online. The article is reprinted in its entirety, just as it appeared when originally published. Its preceded by comments specially prepared by the article's author for this book. Many teachers would like students to become active learners in class by speaking to each other and the teacher about ideas expressed in their curriculum materials. Also, they would like students to initiate more questions. In this article, we learn how a teacher— researcher conducted an experiment in her classroom to achieve these goals. Chapter 19 Action Research RESEARCHER'S Tt was my good fortune a few years ago to pursue a PSSHERS | Ramet omen uc ee er ish Columbia, Of the courses in the stream through which I personally navigated, several focused on the Prepared by | pragmatics of speech in the classroom. Some investi- A.W. Lehmann | gations were devoted to habits of instruction that can 50 easily become paradigms of classroom manage- ‘ment. Others explored the differences between didactic instruction and narrative. Construc- tivist literature emphasized the significance of student engagement and participation in classroom activity, and so on. ‘Overall, these courses’ most lasting impression on me was the conviction that the ‘common discourse of classrooms, including both the oral interchanges between instructors and students and conversations among students themselves, is of exceptional importance to learning that lasts. In constructing my own learning about tis issue, I wished to explore further, by examining my own teaching, the kinds of departures from habitual practices that might more effectively generate student engagement, I wanted to see the effects of mean- ingful oral activities that were both student-driven and productive toward curricular goals, 4 combination that can prove elusive. Teachers love fo promote curricular aims, but often students’ interests li elsewhere. Students love to talk, but frequently not as enthusiastically about classwork as about their personal lives and social priorities, Discourse dynamics are extremely complicated, despite their surface simplicity. Be~ fore one even begins to assess or interpret the denotations of the messages being exchanged, there are numerous elements to consider. The nature ofthe venue, the participants’ ages and. ‘genders, the relative social power of the speakers and listeners, and all the tacit understand- {ngs implicit in culture, tradition, and social expectation combine to provide the framework in which messages are exchanged. ‘Then, once such verbal exchange is under way, other considerations arise. For exam- ple, whose agenda is being served (isthe conversation aimless or directed, and if the latter, by whom)? Is there an understood common purpose to the interchange? Is there mutual understanding of vocabulary and idiom? Of course, if each time we teachers began a lesson in a classroom situation we tried to attend to all these concerns, we would probably stop up cour mouths and ears and cower in despair. The priorities of getting on withthe job prevent ‘most such analysis Despite such a daunting truth, the challenge of improving student speaking and listen- ing in class and developing student participation in discourse while increasing its utility in positive directions is one that doesn’t go away. For teachers to rely on the occasional “teach- able moment” to generate student involvement in curricular goals through class discussion has always seemed too haphazard and too inadequate for professional practice. However, if and when such moments occur, we naturally embrace them strongly. Can we, though, devise useful strategies that are more consistently successful? This is a useful question, and perhaps itis natural that we recognize that questioning itself is key. ‘Although we all use questions quite naturally in our daily lives—asking directions ry- ing to locate products, investigating details in personal relationships, and so on—in educa- tion questions often have a kind of delayed utility For example, if one asks a store clerk in which aisle one can find the canned tuna and the clerk responds “Aisle 4,” the answer fulfills the question's purpose, and the episode is finished, Some questions in the language arts classroom are of just that sort, pragmatic tols for helping youngsters leam to use both the jargon and the concepts of grammar, syntax, composition, analysis, and criticism. However, in the open-ended universe of literary study, it often seems as if our queries never cease, with one question leading to the next in @ continual chain toward an elusive goal of more comprehensive understanding—of what? Of ourselves? Of society? Of the world? Such open-ended questions, the ones with no easy answers, are often, paradoxically, both the most frustrating and the most satisfying. As Leonard Cohen once sang, “I never had a secret charvto get you tothe heartof this or any other matter.” The world of literature is mysterious not only because it reflects a mysterious world, but also because our continual ‘wrestling match with language often leaves us unsatisfied as to the delivery of completely 503 504 Part VI_ Using Other Research Methodologies to Study Problems of Practice definitive meanings. It is our questions, though, that create the boundaries and forms of our exploration, ‘Teachers, naturally, direct a great deal ofthe questioning in classrooms, sometimes to ‘wonderfully beneficial effect. Its tobe expected, though, that although curricular purposes will shape much of the direction and form of their questions, teachers’ personal biases, judg- ‘ments, and values will also mold them. By contrast, students may have questions whose boundaries are not necessarily congruent with those of the instructor or oftheir peers. ‘Adventures in English Literature is one of the common texts used in British Columbia to teach a historical overview of English literature. Surely if teachers expect students to participate in this edventure in a whole-hearted way, we must encourage the exploration of literature as an adventure, and encourage, embrace, and address genuine student questions about it T use the adjective genuine in contrast to the “ted herring” questions youngsters sometimes ask in the hope of drawing the class away from its intended purposes. In addressing the puzzle ofthe role of questions ami the greater discourse in senior secondary language arts, I decided to pursue the theory that perhaps student questions have been insufficiently appreciated and underutilized. Teachers have always valued genuine questions from their students. Our difficulty has been, however, to design into our class- oom planning successful, systematic, and reliable methods of encouraging such questions, ‘of allowing them a respected place on the floor of discussion, and of using them to enhance overall learning, ‘To that end I devised the strategy outlined in the paper that follows. It isto be hoped that the methods outlined may prove useful as guides toward more successful approaches, to the ultimate benefit ofthe whole learning community. Student-Generated Discussion in the Senior Secondary English Cl Teyen] A.W. Lehmann | ABSTRACT i The purpose of this research study was to devise and test a method of encour- ‘aging, and subsequently managing, stuent-generated discussion of English literature within 2 senior secondary classroom. The students would provide not only the discussion itself, but also a “client’s-8ye" evaluation of the process. Accordingly, students were engaged in part ofthe intial clarification of the study's purposes and procedures, produced the bulk of the discussion which constituted the content for the method being examined, and provided ppost-discussion evaluation which could be compared to earlier comments. A simple qualita- tive analysis of written comments provided by the students and of my own notes and reactions to the discussion allowed me to define more clearly some observations about the dynamics of discussion and to select some ditactions for further investigation. It would be premature to

    You might also like