Scribd Logo
Upload

Welcome to Scribd!

  • Chapter 20 PDF

    Uploaded by

    PRIYA TARISINI
    1 views24 pages

    Original Title

    chapter 20.pdf

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    1 views24 pages

    Chapter 20 PDF

    Uploaded by

    PRIYA TARISINI

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    of 24
    chapter 2 0 | mperiay 1 S | 10. 2, Programs typically include curriculum materials, instructional methods, uM. Evaluations of programs are difficult, because programs seldom are uniformly effective, and different stakeholders define effectiveness differently assessment procedures, and perhaps a professional development component. In addition, each program functions within a distinct culture. Evaluation research is value-laden and political at every stage of the process. 4, The identification of a program's stakeholders and their concerns is important | in most evaluation studies. The model of objectives-based evaluation focuses on how well a program helps students achieve specified learning objectives. The model of needs assessment is designed to identify discrepancies between an existing condition and a desired condition, so that educators can decide whether to improve a current program or develoo a new one. . The CIPP model of program evaluation is comprehensive, because itis designed to assess all aspects of a program: stakeholders’ needs and problems, competing alternatives, work plans and budgets, program activities, and program effectiveness. ‘The model of responsive evaluation involves the use of qualitative methods to identify and describe the issues and concerns of a program's stakeholders. Open, safe dialogue among stakeholders is critical to this evaluation model, Research and development (R & D) is systematic model for developing programs and products that are evidence-based. Formative and summative evaluation are integral components of this model. Reports of evaluation studies generally are organized in the format of quantitative research reports, qualitative research reports, or mixed-methods reports. ‘The Joint Committee on Standards for Educational Evaluation, which includes representatives from 12 major educational organizations, has created a set of criteria for judging the quality of educational programs and products. i su 512 Part VI_ Using Other Research Methodologies to Study Problems of Practice Key TERMS IPP model context evaluation effectiveness evaluation emergent design evaluation research formative evaluation impact evaluation input evaluation National Assessment of responsive evaluation Educational Progress ray needs assessment ‘summative evaluation cobjactives-based evaluation sustainability evaluation performance objective transportability evaluation process evaluation product evaluation rogram Joint Committee on Standards for Program culture Educational Evaluation research and development (R & D) The Use of Evaluation Research in Educational Decision Making yalnation research in shucation is the process of using quantitative or qualitative meth- ‘ds, or both, to arrive at judgments about the effectiveness of particular aspects of educa- tion. The process of arriving at these judgments is complicated, because “effectiveness” is @ ‘multifaceted concept, meaning different things to different stakeholders. Also, educational programms and processes seldom are uniformly effective. Typically they are found to have benefits and drawbacks, and they might be effective only under certain conditions. Despite its complexity, evaluation research has become increasingly important in edu- cation, Public schooling is essential in a democratic society, but itis also expensive. For this reason, the general public and policy makers want to know whether their money is ‘well spent. In practice, this means that they want to fund programs that work and eliminate programs that do not work. ‘The prominence of evaluation’s role in governmental funding and decision making is il lstrated by a recent evaluation conducted by the Academic Competitiveness Council, which ‘was commissioned by the U.S. Congress. The following is an excerpt from an ERIC abstract (Document Reference No. ED496649) ofthe Council's report on its evaluation findings: ‘The Academic Competitiveness Council (ACC) is responsible fr reviewing the effectiveness of existing federally funded Science, Technology, Engineering, and Math (STEM) programs, and for improving the state of STEM education in the United States. To this end, it has conducted a review of program evaluations submitted by 115 STEM programs. The ACC's review revealed that, despite decades of significant federal investment in science and math education, there is general dearth of evidence of effective practices and activities in STEM education, and thee is evidence of ineffective duplication of efforts If educators take no action in response to this kind of evaluation, there is a isk that funding for science education and other educational programs gradually wll diminish. This scenario could unfold if policy makers and other groups came to believe that educational programs in genera, not just the ones evaluated by the Academic Competitiveness Council, ate ineffective and not capable of improvement by education professionals ur illustration involves evaluation at a macro level. Closer to home, consider how evalu ation research might affect you as an educator. During your career, you might be asked to learn Chapter 20. Evaluation Research ‘about a new program and teach in itor administer it, For example, suppose you are teaching in ‘regular school and are asked to teach in a new charter school instead. Undoubtedly, you will be interested in what is known about the effectiveness of this new school. Who is it intended for? Is there evidence that it produces better outcomes than traditional schools? What are the risks and drawbacks of this type of schoo!” Would you be comfortable teaching in it? Evaluation studies in the literature might help you answer these important questions Af none are available, you and your colleagues will need to learn about the program as you implement it. Ifthe program is effective, all is well. If not, you might find yourself asking. why the program was not researched beforehand to discover its pitfall. ‘The purpose of this chapter is to help you learn about the effectiveness of programs not only from your personal experience with them but also from the findings of evaluation, research, Examples of Evaluation Research Researchers do evaluation studies on many aspects of the education enterprise. The titles of the following journal articles suggest how wide the range is. Arancibia, V., Lissi, M. R., & Narea, M. (2008). Impact in the school system of a strat- gy for identifying and selecting academically talented students: The experience of Program PENTA-UC. High Ability Studies, 19(1), 53-65. Black, E. W., Fertig, R. E., & DiPietro, M. (2008). An overview of evaluative instru- mentation for virtual high schools. American Journal of Distance Education, 221), 24-45, Eteokleous, N. (2008). Evaluating computer technology integration in a centralized school system. Computers & Education, 51(2), 669-686. Gaudet, C. H., Annulis, H. M,, & Kmiec, J.J Jr (2008). Building an evaluation frame- ‘work for a competency-based graduate program at the University of Southern Mis- sissippi. Performance Improvement, 47(1), 26-36. Pence, H.M., & Macgillivay I K. (2008). The impact of an international field experi- tence on preservice teachers, Teaching and Teacher Education, 24(1), 14-25. Schull, C.P., & Anderson, E. A. (2008). The effect of home visiting and home safety on children’s school readiness. European Early Childhood Education Research Jour- ral, 16(3), 313-324 ‘You can find many more examples of evaluation research at the website for the What Works Clearinghouse (http:/ies.ed.govincee/wew), described in Chapter 5. The U.S. Depart- ment of Education sponsors and funds this agency, reflecting its increasing emphasis on evaluation research to guide policy making and influence the direction of school reform initiatives. ‘Programs as a Focus of Evaluation Research ‘A moment's reflection will help you realize that evaluation permeates the entire education enterprise, Teachers evaluate students’ academic achievement and behavior. Administra- tors evaluate teachers. School boards evaluate school districts. School districts evaluate individual schools. Textbook adoption committees evaluate new textbooks. ‘Our focus in this chapter, though, is on methods for evaluating programs. We define a program as a systematic sequence of materials and activities designed to achieve explic- itly stated goals and be used in many different settings. In education, a program typically includes curriculum expressed in text materials and other media, instructional methods, 513 si4 Part VI_ Using Other Research Methodologies to Study Problems of Practice assessment procedures, and pethaps a staf development component to help educators im- plement it properly. Al of these components are packaged so that they can be implemented in many different school sites. For example, elementary textbook series in reading produced bby major publishers qualify as programs by this definition, Saville Kushner and Clem Adelman (2006) claim that programs include more than these components: {Phograms typically exhibit a “program culture." This isto say that they have rules, rituals, roles atension between the individual andthe collective, a recognizable exture to thir social ‘processes, and a broadly agreed boundary that demarcates thoughts and action that are or ae not of tis program... People fit into programs, but programs also fit nto people's ives, (9.712) Based on this statement, we define program culture as the rules, rituals, and roles that accompany the technical specifications of a program and also the tensions that come into play when the needs and desires of individuals in the program conflict with these specifications Program culture becomes an important consideration when educators are asked to adopt a new program that will replace the one they are already using. They are likely to wonder how much work the adoption of the new program will add to their already busy schedules. They might wonder, too, about the motivations of those promoting the new pro- gram, especialy if they like the existing program. These concerns can be mitigated to an. extent if evaluation evidence attesting tothe program's effectiveness is available. ‘Evaluation Research as a Political Activity All research involves values, For example, the fact that researchers choose to study one ‘educational problem rather than another reflects a value judgment that it is more impor- tant, worthwhile, or intresting, or that itis more likely to attract funding and advance the researchers’ careers, Evaluation research i even more value-laden, because the researchers’ explicit goal is to make value judgments about a program based on their empirical findings. Furthermore, individuals who are affected by the program will make value judgments about whether evaluations of it were fair and relevant. If everyone affected by a program shares the same values, an evaluation study can probably proceed smoothly. However, this is seldom the case when educational programs are involved. An educational program has many features, and some might appeal to certain individuals and repel others. Therefore, individuals and organized groups are likely to com- pete with each other for power to make decisions about the program, such as whether to adopt or reject it, or how to evaluate itto highligh its strengths, or perhaps how to evaluate it in order to minimize or conceal its weaknesses. ‘This competition for power and infiuence in evaluating programs, especially when the stakes are high, makes evaluation research an inherently political activity. The intensity of, the politics can increase if the program has prominent cultural features—for example, a program for ethnic-minority groups designed and evaluated by ethnic-majority educators and researchers. Because program evaluation is inherently political, researchers have found it helpful to identify all relevant stakeholders atthe outset of an evaluation study. A stakeholder is an individual who is involved in the phenomenon that is being evaluated or who may be affected by the findings of an evaluation. Identifying all the stakeholders affected by an educational program is not always easy. However, itis essential to be as inclusive as pos- sible. Otherwise, some stakeholders’ voices might be left unheard, thereby compr the integrity of the evaluation study. Chapter 20 Evaluation Research (Models of Evaluation Research In the course of doing evaluation studies over a period of decades, researchers have re- flected on their work and crested formal models of how evaluations should be done. Daniel ‘Stufflebeam and Anthony Shinkfield (2007) identified and compared 26 of these evaluation models. In the following sections, we describe five ofthese models, because they have achieved some prominence in the literature and because they illustrate the range of variation among models. They vary primarily in whether they focus on stakeholder perspectives and pro- ‘gram culture or on the program’ effectiveness in achieving measurable objectives. If you plan to do a program evaluation, knowing about these models can help you in deciding which one best suits your purpose. If you are a stakeholder in an evaluation study, knowing about these models will give you a better sense of the study's goals and procedures, Objectives-Based Evaluation Objectives-based evaluation focuses on the extent to which an educational program helps students achieve the intended learning objectives associated with it. The National Assess- ment of Educational Progress (NAEP), which we describe in Chapter 10, is an example of this evaluation model. NAEP isa federally funded project that annually determines how well students are performing on measures of learning objectives in reading, mathematics, science, writing, history, geography, and other school subjects. [NAEP is based on the model of curriculum, instruction, and evaluation developed by Ralph Tyler (1949), who claimed that school instruction should be organized around spe- cific curriculum objectives and thatthe success of such instruction should be judged on the basis of how well students reach those objectives ‘The U.S. school system largely follows the principles of Tyler's model to this day. Classroom, district, state, and federal assessments are, for the most part, evaluations of hhow well students perform on measures of curriculum objectives, Programs designed to tie teachers’ incentive pay to student learning outcomes and programs to identify and remedi ate “failing” schools also exemplify the Tyler model. Educational experiments reflect objectives-based evaluation if they compare the learn- ing gains of students in an innovative program with those of students receiving a traditional program or no program. Some of these studies are collected and reviewed by the What ‘Works Clearinghouse in an effort to provide educators with evaluation evidence about par- ticular educational programs, ‘Needs Assessment [Needs assessment isa set of procedures for identifying and prioritizing needs related to so- etal, organizational, and human performance (McKillip, 1987).A need usually is defined as a discrepancy between a desired and an existing state or condition. For example, Richard Mihans (2008) analyzed a previously published survey of school staffing (National Center for Education Statistics, 2006). He found that 64,954 elementary and secondary public schools had teaching vacancies inthe 2003-2004 school year. Teach- ing vacancies are projected to soar, even while student enrollments are also projected to increase for decades to come. Thus, there is a discrepancy between the exiting number of teachers andthe desired number of teachers. This discrepancy consttues a need, or in other words, a problem of practice. Mihans suggests that a key approach to solving it sto retain the teachers who are already in the profession, He identifies five conditions that are likely to accomplish this goal: (1) higher salaries, (2) more support from administrators, (3) more opportunities to be mentored, (4) better working conditions, and (5) more professional autonomy. 315 516 Part VI_ Using Other Research Methodologies to Study Problems of Practice Mihane's analysis of particular ned inthe education system and proposal for solu- tions can be used as a basis for developing programs to improve teacher retention. These programs, of course, should be evaluated to determine ther ellecliveuess "Needs assessment can be thought of as tho first stage in program development. By evaluating existing conditions, researchers can determine whether they are satisfactory or in need of improvement. The availability of research data to establish a need should make it easier to argue for resources to improve existing programs or develop new ones. The Context-Input-Process-Product (CIPP) Model ‘Stuffiebeam (2003) developed the CIPP model to help educators evaluate programs, al- though the model also can be used to evaluate projects, personnel, institutions, and other entities. The acronym CIPP refers to four types of evaluation (context, input, process, prod- uct) that should be performed if one wishes to conduct a truly comprehensive assessment of a program as it unfolds over time, + Context evaluation assesses needs, assets, and problems ofthe stakeholders, staff, and beneficiaries ofthe program. + Input evaluation assesses competing alternatives, work plans, and budgets for the program under consideration, + Process evaluation documents and assesses program activities, + Product evaluation assesses whether the program succeeded, Product evaluation has four subparts: + Impact evaluation assesses whether the program reached the right target audience, + Effectiveness evaluation assesses the quality and significance of the program's outcomes + Sustainability evaluation assesses whether the program is institutionalized suocess- fully inthe short term and long term. + Transportability evaluation assesses whether the program can be adapted and insti- tutionalized in other settings. Stuffiebeam (2007) developed sets of checklist items to guide evaluators and stake holders through each of these types of evaluation as wel as other aspects of evaluation, such as making contractual agreements for the evaluation and writing a final report. To illustrate, ‘we show the checklist items for effectiveness evaluation in Table 20.1 ‘Most program developers and administrators probably do not have the resources to perform a program evaluation that encompasses the entire CIPP model. However, they can review all the items and select those thet are most important and feasible for ther particular situation. Responsive Evaluation Stake (2004) developed one of the first qualitative approaches to evaluation. Responsive evaluation focuses on identifying and describing stakeholders” issues (i., points of con tention among different stakeholders) and concerns (i.e, matters about which stakeholders feel threatened or that they want to substantiate), Concems and issues tend to provide a wider and different focus for an evaluation study than the program objectives that are cen- tral to objectives-based evaluation. ‘The four phases of responsive evaluation are (1) initiating and organizing the evalua- tion, (2) identifying key issues and concerns, (3) gathering useful information, and (4) re- Porting results and making recommendations. During the first phase, stakeholders are identified; also, the evaluator and client negotiate a contract to specify such matters as the phenomena to be evaluated, the purpose ofthe evaluation, rights of access to records, and ‘guarantees of confidentiality and anonymity. TABLE 20.1 _ Effectiveness Evaluation Chapter 20. Evaluation Research S17 Effectiveness evaluation documents and assesses the quality and significance of outcomes. Evaluator Activities Client/Stakeholder Activities—Assessment/Reporting Outcomes TD Inteniew key stakeholders, euch as communtiy leaders, beneficiaries, program leaders and staff, and other interested parties, to determine their assessments of the program's positive and negative outcomes. 1 Use effectiveness evaluation findings to gauge the program's positive and negative effects on beneficiaries 1 As relevant, use the effectiveness evaluation findings to gauge the program's positive and negative effects on the community/pertinent environment OAs feasible and aporopriate, conduct in-depth case studies of selected beneficiaries. Use the effectiveness evaluation findings to sort out and judge important side effects TD Engage an evaluation team member and program staff to supply documentation needed to identity and confirm the range, depth, quality, and significance of the program's effects on beneficiaries 0 Use the effectiveness evaluation findings to exarnine whether program plans and activities need to be changed, TAs appropriate, engage an evaluation team member to compile and assess information on the pprogram’s effects on the community 1 Use the effectiveness evaluation findings to prepare and issue program accountability reports, TD Engage a goal-free evaluator” to ascertain what the program actually did and to identify its full range of effects—positive and negative, intended and unintended, 1D Use the effectiveness evaluation findings to make a bottom-line assessment of the program's success, TD Obtain information on the nature, cost, and success of similar programs conducted elsewhere and judge the subject program's effectiveness in contrast to the identified “critical competitors.” TD Use needs assessment data {from the context, evaluation findings), effectiveness evaluation findings, and contrasts with similar programs elsewhere to make a bottom-line assessment of the program's significance 1 Compile effectiveness evaluation findings in a draft report (that may be incorporated in a larger report) and present it to the client and agreed-on stakeholders, 1 Discuss effectiveness evaluation findings in a feedback session, C Finalize the effectiveness evaluation report and present it to the client and agreed-on stakeholders Incorporate the effectiveness evaluation findings in an updated progrem profile and ultimately in the final evaluation report. “A gosh fee evaluators a contracted evaluator who, by agreement is prevented from learning a program's goals ands charged to assess lahat the program i actually doing andl achieving imespecive of ts sms, This technique is powerful or identifying side elects or unintended ‘outcomes, bath positive and negative, and for describing what the program is actually doing, respective of stated procedures, Source: Adapted rom The Evaluation Center, Evaluation Checklists Website: www wmich edulevalet/checksts/cippchecklst_mar07 pf Retrieved November 7, 2008. S18 Part VI_ Using Other Research Methodologies to Study Problems of Practice In the second phase, Key issues and concerns are identified through direct involve- ‘ment with a variety of stakeholders. The evaluators seek to clarify the values of different stakeholders that underlie the issues and concerns expressed. For example, in examining a particular school system's governance structure, the evaluators might discover that some stakeholders value a high-quality curriculum and accountability, whereas others place greater value on equality of representation in decision making and a rational decision- ‘making process. In the third phase of the evaluation, the evaluators collect more information about the concerns, issues, and values identified by the stakeholders, descriptive information about the phenomena being evaluated, and standards to be used in making judgments about it. The final phase of a responsive evaluation involves preparing reports of results and recommendations. Frequently a case study reporting format (see Chapter 14) is used to describe the concerns and issues identified by stakeholders. The evaluators, in negotiation with stakeholders, then make judgments and recommendations based on the information that has been collected. In doing a responsive evaluation, evaluators do not specify a research design at the outset oftheir work. Instead, they use an emergent design, meaning thatthe design of the evaluation changes as evaluators gain insights into stakeholders’ primary issues and con- cers. Consistent with the analytic methods associated with grounded theory (see Chapter 14), responsive evaluators continue obtaining information from stakeholders until the infor- ‘mation they are receiving becomes redundant with information already collected. Dialogue among stakeholders is central to responsive evaluation. However, dialogue right or might not resolve issues and concems about the program being evaluated. Tineke ‘Abma (2006) claims that the actual goal of dialogue is not necessarily agreement. “Dia- logue may lead to consensus, but it is also considered successful if personal and mutual understanding has increased or ifthe understanding of differences is enhanced” (p. 31). Abuu's eight guidelines for responsive evaluators to follow in order to facilitate open dia- logue among diverse stakeholders are listed in Figure 20.1 All types of research methodology require researchers to be sensitive to the needs of their participants. The participants in program evaluation are stakeholders who are embed ded in a political process, and therefore, evaluators need to use guidelines, such as those presented in Figure 20.1, to ensure that the stakeholders feel safe in providing evaluative data, They ate more likely to feel safe if they see that the evaluator respects them and if they ‘know that there will be no political reprisals for anything they say. Responsive evaluation is grounded in qualitative methodology, but it is not alone in this respect. Stuffiebeam and Shinkfield (2007) describe other evaluation models that use qualitative methodology. Educational Research and Development valuation plays a key role in educational research and development (R & D). R & Dis systematic process for developing, improving, and assessing educational programs and materials (refered to hereafter as products). A term sometimes used to describe R & D, research-based product development, convey’ the fact that (1) the goal is to develop a prod- uct based as much as possible on research findings and that (2) the development process will be research-based. Ifyou want to develop your own product (e.., software ora set of curticulum guides), you might be able to accomplish this goal as a thesis or dissertation by developing the product while also doing research oni Dick, Carey, and Carey (2005) advocate the systems approach model of educational R&D. The 10 steps of this model are shown in Figure 20.2. Step 1 involves needs assess- ‘ment, which we have described ina previous section In cis model, a needs assessment is carried out in order to identify the goals ofthe product to be developed. Step 2, instructional analysis, involves identification of the specific skills, procedures, and learning tasks that are seen as desirable or necessary to reach the instructional goals. Chapter 20 Evaluation Research $19 FIGURE 20.1 Eight Guidelines for Creating Effective Dialogues among Stakeholders in a Responsive Evaluation Study 1. Identity and include everyone who is a stakeholder in the program being evaluated. Treat stakeholders as partners and collaborators in the evaluation Pay particular attention to individuals who might feel silenced by other stakeholders 2. Show respect to all stakeholders. Show respect to stakeholders, especially those \who feel silenced or without power, by conducting in-depth, informal interviews with them on a one-to-one basis 3. Build trust. Build trust by interviewing stakeholders and by participating, as equals, with them in activities on the stakeholders’ own “tur.” 4, Examine the stakeholders’ environment to determine their need for privacy. Find out whether stakeholders have concerns that their comments about the program might put them in jeopardy. Respect their rights to privacy and anonymity, 5. Form homogeneous discussion groups. If certain stakeholders feel vulnerable, put them in their own discussion group. Seeing others who are ike themselves might help stakeholders feel comfortable about expressing their concerns about the program 6, Use stories to create an open dialogue. Ask stakeholders to share stories about experiences that convey their concerns about the program. Individuals often become more comfortable in a group if they share stories with each other. 7. Avoid subtle mechanisms of exclusion. Look for signs that certain stakeholders, especially those who have a lower status, are being excluded from participating ina group dialogue. Some members of a group might try to silence other members by nonverbal expressions or critical remarks; be sure to foster expression by everyone, 8. Interact with all stakeholder groups. If stakeholders form groups that have different status with respect to the program (e.g., administrators, staf, clients, community members), be impartial. A responsive evaluator should be @ spokesperson for all the groups and convey each group's distinctive perspective to the other groups. Source: Adapted from Alma, T. A. (2006) The practice and politics of responsive evaluation. ‘American Journal of Evalustion, 271) 31-23 Step 3 is designed to identify the level of entry behaviors (sometimes called enabling objectives) that learners bring to the learning task. It also involves identification of other characteristics ofthe learners that might affect learning (e., specific personality traits such as test anxiety) and the settings in which the instruction will occur and in which the learned skills will ultimately be used. ‘During Step 4, the developers write performance objectives, which are descriptions of the behaviors that the learners will be able to demonstrate after instruction, Then assess- ‘ment instruments to test achievement ofthe objectives are developed (Step 5); the appropri- ate instructional strategy is formulated (Step 6); and instructional materials are developed or possibly selected from available materials (Step 7). Steps 8, 9, and 10 of the R & D systems model involve the distinction between forma- tive and summative evaluation, which was formulated by Scriven (1967). Scriven found 520 Part VI_ Using Other Research Methodologies to Study Problems of Practice FIGURE 20,2 __ Steps of Systems Approach Model of Educational Research and Development 2 s Conduct revise lnstuctonat 7 é i ington } 7 + 3 ‘ 7 wie nat | | coon | | Dain, pertrmance |-+! assessment») istotonal |») SES*RCL | 5 ornate nieces. | "| Istumens, |") seat. inate |") gration | | oinstcton Le Design ant condict aecnceenen oi | summative svalton Source: Adapted fom figure on pp. x=! in Dick, W, Carey, L, & Carey, J, 0. (2008). The systematic design cof insrition th ed). Boston: Aly & Bacon. Published by Alyn and Bacon, Boston, MA Copyright © 2005, by Pearson Education. Adapted with permission from the publisher. that, in practice, evaluation serves two different functions, which he called formative evalu- ation and summative evaluation. Step 8, formative evaluation, involves collecting data about an educational product while itis under development. This is done to help the developers and evaluators decide ‘whether the product needs revision before release and, if so, what revisions are necessary Formative evaluation is also used to determine whether the prospects for an eventually effective product are low, and if so, to reach a decision that further product development should be terminated and the product should not be released, ‘As shown in Figure 20.2, formative evaluation can occur at all the earlier stages of the development process. For example, the developers might carry out a formative evaluation of the product's objectives during Step 4, examining such issues as the clarity and comprehen- siveness ofthe objectives. Based on the results, they might eliminate some objectives, re- ‘rite others, or add new objectives. Once they have developed instructional materials (Step 7), they might do more formative evaluation and further revise the performance objectives, pethaps so that the objectives better match the content of the instructional materials. ‘A mote thorough formative evaluation is also conducted when a prototype of the prod uct (that i, a relatively complete set of the necessary elements) is available. This formative evaluation involves a field test of the product. Compared to implementation of the com- pleted product, a field testis a trial of the product (1) with a smaller number of research participants, (2) more hands-on involvement of the developers, and (3) a more controlled cvironment than the real-life conditions in which the final product is meant to be used, ‘At some point in your career, you might find yourself involved in a formative evalu ation of a program or product. Your knowledge about evaluation research will help you to be a good participant in the formative evaluation process. Your expert feedback can serve as one basis for improving the program or product, o that itis effective for many other educators later on Step 10, summative evaluation, is conducted once the development process (i.e. Steps 1 through 9) has been completed. The purpose of summative evaluation is to determine Whether the completed product achieves its objectives under real-life conditions. Summa- Chapter 20 Evaluation Research tive evaluation also might involve comparing the effectiveness of the completed product with that of competing products. ‘Summative evaluation usually is carried out by someone other than the developers, but it ‘also can be done by members of the development team if appropriate controls are used to min- ‘mize researcher bias. Ifa summative evaluation demonstrates thatthe products effective, we can characterize it as being evidence-based, In Chapter 1, we explained that evidence-based practice is becoming increasingly important in education and other professional fields ‘Most evaluation studies of programs that appear in the education literature are summa- tive evaluations. As an educator examining the evaluation research literature, you probably ’be most interested in summative evaluations of instructional programs, methods, and materials that have been tested under conditions similar to your work environment. These ‘evaluations will help you determine whether these products will be effective under condi- tions similar to your own situation, (How to Read an Evaluation Research Report Many evaluation studies are done under contract fora school system, governmental agency, or other organization. These reports serve local purposes and might contain sensitive infor- ration. Therefore, they are rarely available in the published literature, Some of the intended readers of these reports might have little knowledge of research methods and terms. Therefore, the evaluation reports are likely to be nontechnical. The ‘emphasis is on the implications ofthe findings rather than on the methods used to generate them. Other evaluation studies are conducted to assess programs that are widely used or that show promise of solving an important problem of practice. These studies generally are con- ducted with the intent that they will eventually be reported in wide-circulation educational Journals. These are the kinds of reports that we consider here. ‘We start by noting that most evaluation studies use quantitative, qualitative, or mixed methods. If an evaluation study uses quantitative methods, the report will be organized like the reports of quantitative methods described in Part Ill. Many of these reports are experiments that are designed to test the effectiveness of a program. An example is the experiment by Gerald Knezek and Rhonda Christensen (2008) whose purpose was to evalu- ate technology-intensive programs at the primary grade level. The abstract for the article, downloaded from ERIC, is shown in Figure 20.3. As you read te abstract, you will see that the study uses quantitative methodology. Specifically, itis an experiment (see Chapter 13). If a program evaluation uses qualitative methods, on the other hand, it most likely will be a case study. Therefore, the report probably will be organized like the reports FIGURE 20.3 Abstract of Evaluation Study Using Quantitative Methods * moet nmamo in esaea s a tae 7HON an et ieanese sarees i aac ee ca an: te Do masa: al 21 522 Part VI_ Using Other Research Methodologies to Study Problems of Practice described in Chapter 14, An example is an evaluation study by Patrick McQuillan and ‘Yves Salomon-Fernandez (2008) whose purpose was to determine how the staff of under- performing schools responded to stete-Led interventions to improve the students’ academic achievement. The schools were underperforming as defined by the No Child Veft Behind ‘Act (NCLB), which mandates that schools make adequate yearly progress toward profi- ciency forall students in Englishvlanguage arts and mathematics by 2014, The researchers collected qualitative data from staf at two underperforming middle schools and one underperforming high school. They interviewed 16 teachers and administra- tors at these schools, made observations at each school on two occasions, and analyzed vari- ‘ous documents. Te teachers and administrators constituted the stakeholders fortis study ‘MeQuillan and Salomon-Fernandez (2008) used the method of constant comparison based on grounded theory (see Chapter 14) to identify themes and paters in the data, Their report lists 11 themes, each of which reflects the stakeholders’ perspective on state-level interventions. Each theme is illustrated by interview comments made by the teachers and administrators. Figure 20.4 presents two of the themes, one postive and one negative, to illustrate the results ofthis evaluation study. FIGURE 20.4. Themes and Quotes from a Report of a Qualitative Evaluation of a State-Level intervention on Underperforming Schools Theme 1: Benefits of State Intervention Administrator Statement “Ican't tell you how much support, cooperation I've gotten from teachers. ... Most people here spend a great deal of time after school, they go the extra mile. This has forced [teachers] to examine what they do. ... thank [Massachusetts Comprehensive System] for bringing my staff together and making them even more Unified, ... This has been my most rewarding experience in education." (p. 17) Teacher Statement “ithas had a positive impact in terms of the lower end of the teaching staff who didn’t necessarily give a lot of thought and consideration to what they were doing They've had to raise the bar. lin fact] we've all stepped up to the [challenge [We've been forced to look at ourselves and reflect more. And that’s been the positive part.” (p. 18) Theme 2: Negative Impacts of interventions on Schools Teacher Statement "We have not been able to operate as a real school because all our attention has been focused on doing what the DOE [Department of Education] wants... I's not like we're doing things because i's best for our kids... [Its ahvays around the parameters set by the DOE. You do something and you have to think, 'ls this going to fit within the guidelines of DOE? Will it meet DOE expectations?” (op. 20-21) Administrator Statement “I'm constantly preparing information for the state and... itreally takes me away ‘from instruction and working in classrooms. ... It takes a lot of energy. ... I's good thing to have oversight, but there are limits... t's the most frustrating part of my job." (p. 21) Sourea: McQuillan P., &Salomon-Femander, ¥, (2008). The impact of state intervention on “underperforming” schools in Massachusetts: Implications for polcy and practice. Education Policy Acalysis Archives, 148), 1-49, Chapter 20 Evaluation Research ‘These findings can help other stakeholders (c.g, federal and state officials, educators’ professional organizations) understand the impact of external mandates on those who must implement them. Better solutions to problems of educational practice are more likely if all stakeholders can have their concerns and aspirations heard and respected. Evaluating an Evaluation Research Study ‘You can judge the quality of most evaluation research reports by answering the questions listed in Appendix 2 (for a quantitative study), Appendix 3 (for a qualitative study), and Appendix 4 (for specific research designs). In addition, you can refer tothe authoritative standards for program evaluation devel- oped by the Joint Committee on Standards for Educational Evaluation (1994), The committee consisted of representatives from 12 major educational organizations, including the American Association of School Administrators the American Federation of Teachers, and the American Educational Research Associ ‘The standards are shown in Figure 20.5. You will see that there are 30 standards, which are grouped under four criteria: utility, feasibility, propriety, and accuracy. You can use these standards to judge the adequacy of evaluation research studies that you read or to design your own evaluation research. ‘The Joint Committee excluded the evaluation of educators from their consideration of program evaluation, because they previously had developed a separate set of person- nel evaluation standards (Joint Committee, 1988). They also developed a separate set of standards for assessing evaluation practices in elementary and secondary classrooms (Joint Committee, 2002). An example of HOW PROGRAM EVALUATIONS CAN HELP IN SOLVING PROBLEMS OF PRACTICE ‘A major problem in education is that some students do not adjust well to the curricu- lum and regulations of mainstream schools. For this reason, educators have devel- oped alternative models of schooling, such as described in the following brief news item, Four LA. teens say they were on the verge of becoming dropouts, but the individual- ized attention they got from the staff at Hollywood's Alternative Education and Work Center helped them set a new direction, and now all four plan to attend college. ASCD SmartBrief July 18, 2008, summarizing an article in the Los Angeles Times, July 17,2008. School and government officials might wonder whether this alternative model is worth expanding regionally or perhaps nationally, so that other potential dropouts might benefit. However, the second-hand testimonials in the news item are slim evi- dence of the program's effectiveness. Evaluation research cen provide better evidence, ‘as we explain below. Experts in program evaluation can present various evaluation models to educators, admin- istrators, and other stakeholders. Objectives-based cvaluation, needs assessment, CIPP, and responsive evaluation are some of the available options. $23 524 Part VI_ Using Other Research Methodologies to Study Problems of Practice FIGURE 20.5 Criteria and Related Standards for Effective Evaluation Utility: The extent to which the evaluation is informative, timely, and useful to the affected persons. 1, Stakeholder identification. All the groups affected by the evaluation should be identified. 2. Evaluator credibility. The evaluator should be competent and trustworthy. 3. Information scope and selection. The information to be collected should pertain directly to the evaluation questions and stakeholder concems. 4, Values identification. The evaluators’ basis for raking value judgments from the obtained results should be made clear. 5, Report clarity. The evaluators’ report should be comprehensive and easily understood. 6. Report timelines and dissemination. Evaluation reports, including interim reports, should be disseminated to users in a timely manner. 7. Evaluation impact. The evaluation should be conducted so as to encourage appropriate action by the stakeholders Feasibility: The extent to which the evaluation design is (1) appropriate to the setting in which the study is to be conducted and (2) cost-effective. 1. Prectical procedures. The evaluation procedures should be practical and minimally distuptive to participants 2. Political viability. The evaluators should obtain the cooperation of affected interest groups and keep any of them from subverting the evaluation process. 3. Cost effectiveness. The benefits produced by the evaluation should justify the resources expended onit Propriety: The extent to which the evaluation is conducted legally and ethically, 1. Service orientation. The evaluation should help stakeholders meet the needs of all their clients and the larger society as well. 2. Formal agreements. The formal parties to the evaluation should state their obligations and agreements in 3 written contract. 3. Rights of human subjects. The rights and welfere of persons involved in the evaluation should be protected. 4, Human interactions. Evaluators should show respect in their interactions with persons involved in the study. 5. Complete and fair assessment. The strengths and weaknesses of the entity being evaluated should be explored completely and fairy 6. Disclosure of findings. Individuals with a legal right to know and those affected by the results should be informed about the evaluation results. 7. Conflict of interest. If @ conflict of interest should aise, it should be treated openly and honestly 8, Fiscal responsibilty. Expenditure of resources for the evaluation should be prudent and ethically responsible. Accuracy: The extent to which the evaluation produces valid, reliable, and comprehensive information for making judgments of the evaluated program's worth 1. Program documentation. All pertinent aspects of the program being evaluated should be described in detail 2. Context analysis. Aspects of the program's context that affect the evaluation should be described in detail 3. Described purposes and procedures. The evaluation’s purposes and procedures should be described in detail 4, Defensible information sources. Sources of data should be described in sufficient detail that their adequacy can be judged. 5. Valid information. The data-collection procedures should yield valid interpretations. 6. Reliable information. The data-collection procedures should yield reliable findings. 7. Systematic information. The evaluation data should be reviewed and corrected, if necessary. 8. Analysis of quantitative information. Analysis of quantitative data in an evaluation study should be thorough and should yield clear interpretations. 9. Analysis of qualitative information. Analysis of qualitative data in an evaluation study should be thorough end should yield clear interpretations. 10, Justified conclusions. Evaluators should provide an explicit justification for their conclusions. 11, Impartial reporting. Evaluation reports should be free of bias and of the personal feelings of any of those connected to the evaluation 12, Meta-evaluation. The evaluation should be subjected to formative and summative evaluation Using this lst of standards. Source: Adapted from Joint Committee on Standards fr Educational Evaluation (J. R Sanders, Chai). (1994). The program evaluation standards 2nd ed). Thousand Oaks, CA Sage. Chapter 20 Evaluation Research Itis likely that these stakeholders have a variety of questions, ll driven in part by concerns abou the costs and benefits of funding this particular model of schooling for potential drop- outs. Is the Alternative Education and Work Center a good environment for these students? Are students acquiring worthwhile knowledge, skills, and values? Is the school being managed ef- fectively? Are the schools curriculum, schedules, and other organizational features suficiently documented that they canbe incorporated into other new or existing schools? If the stakeholders want all these questions answered, the CIPP model of program evaluation might be a good option. The stakeholders can go tothe CIPP website and review the checklist items listed there. They can choose those parts ofthe model impact, effective- ness, sustainability, transportability) thet are most appropriate to their concerns. ‘The evidence provided by this typeof program evaluation can be helpful tothe stake- holders in deciding what to do next. For example, they might find that the Alternative Education and Work Center is helping some potential dropouts, but not others. This finding possibly could lead them to provide funding forthe school’s administrators to expand cur- riculum and resources to reach students whose needs are not currently 525 heck 1. Program culture refers to the 4. explicit and tacit views of those involved in a program evaluation study. i. the roles and relations between program developers and educators, . the sitals, rules, and roles that come into play as individuals become involved with a program. 4. a model of evaluation research that focuses on the study ofthe culture into which program needs to fit stakeholder is the individual who initiates the request for an evaluation anyone who will be affected by the evaluation findings. ‘an evaluator who assesses a program by analyzing costs relative to benefits. 4. an evaluator who uses personal interpretation to evaluate | phenomenon, 3. The work of the National Assessment of Educational Progress and the What Works Clearinghouse is based primarily on a. needs assessment i. the CIPP model . objectves-based evaluation. 4d. Allof the above, 4, Needs assessment typically involves ‘4, measurement ofthe discrepancy between an existing condition and a desired condition, >. interviews of stakeholders to identify what they require fora program to function efficiently ¢. estimation ofthe costs and benefits of a proposed intervention. 44. the determination of which program, in ase of possible programs, should have priority for funding. 5. The CIPP model of evaluation focuses on 1. stakeholder needs and problems. b. a programs work plans and budgets. ¢. a program's impact and transportbilty 4. Allof the above. Sel oee> 6. A central feature of responsive evaluation is its 1. focus on identifying te issues and concems of stakeholders. b. specification of the evaluation design prior to data collection, ‘©. concern withthe goals and objectives ofthe program being evaluated. 4. specification of procedures for reconciling the different perspectives of various stakeholders. 7. The primary purpose of formative evaluation in ‘educational R & Dis to ‘2, demonstrate the effectiveness of a program under operational conditions. ». evaluate the program once the development process has ‘been completed. ¢. obtain information to guide revision and further development of the program. 4. satisfy the oversight mandates of the agency funding the program's development. 8, Unlike formative evaluation, summative evaluation of a program generally ‘8 occurs throughout the R & D process. », isconducted to determine whether development of the product should be discontinued. «. is conducted to determine why stakeholder groups have different views of the program's effectiveness. 4. is conducted to determine the effectiveness of the ‘completed program, 9. Reports of program evaluation studies 4 are rarely published, because the evaluation process reveals sensitive political matter that stakeholders wish to keep private. ', are only published in professional journals if te study involves the objectives-based model of evaluation. generally are organized more like reports of historical studies than like reports of case studies, 4. generally are organized like reports of quantitative research studies and case studies. S26 Part VI_ Using Other Research Methodologies to Study Problems of Practice 10. The program evaluation standards of the Joint Committee ‘on Standards for Educational Evaluation ‘are suitable forthe evaluation of teachers and schoo! administrators are suitable for use by teachers in constructing their own tests. ©. were developed by a commiltee representing major educational organizations in the United States. 4. does not specify ethial standards for conducting program evaluation, because standards of this type differ greatly across school districts and other agencies Chapter Referenc ‘Abma,T. (2006). The practice and polities of responsive evalu tion. American Journal of Evaluation, 27(1), 31-43, Bolla, B. (2007), Federal programs to improve science education are not well reviewed, pane nds. Chronicle of Higher Education, 5367), A19, Dick, W, Carey, L., & Carey, 1.0. (2005). The sstemati design of instruction (6th ed). Boston: Alyn & Bacon, Joint Commitee on Standards for Educational Evaluton 1988) The personnel evaluation standards: How to assess systems for

    You might also like