This action might not be possible to undo. Are you sure you want to continue?
Journal of MultiDisciplinary Evaluation
Number 2, February 2005
Editors E. Jane Davidson & Michael Scriven
Associate Editors Chris L. S. Coryn & Daniela C. Schröter
Assistant Editors Thomaz Chianca Nadini Persaud John S. Risley Regina Switalski Schinker Lori Wingate Brandon W. Youker
Webmaster Dale Farland
Mission —The news and thinking of the profession and discipline of evaluation in the world, for the world—
A peer-reviewed journal published in association with The Interdisciplinary Doctoral Program in Evaluation The Evaluation Center, Western Michigan University
Editorial Board Katrina Bledsoe Nicole Bowman Robert Brinkerhoff Tina Christie J. Bradley Cousins Lois-Ellen Datta Stewart Donaldson Gene Glass Richard Hake John Hattie Rodney Hopson Iraj Imam Shawn Kana'iaupuni Ana Carolina Letichevsky Mel Mark Masafumi Nagao Michael Quinn Patton Patricia Rogers Nick Smith Robert Stake James Stronge Dan Stufflebeam Helen Timperley Bob Williams
Table of Contents PART I Editorial In this Issue: JMDE(2)..........................................................................................1 Marketing Evaluation as a Profession and a Discipline .......................................3 E. J. Davidson Articles Monitoring and Evaluation for Cost-Effectiveness in Development Management........................................................................................................11 Paul Clements Network Evaluation as a Complex Learning Process ........................................39 Susanne Weber Practical Ethics for Program Evaluation Client Impropriety...............................................................................................72 Chris L. S. Coryn, Daniela C. Schröter, & Pamela A. Zeller Ideas to Consider Managing Extreme Evaluation Anxiety Though Nonverbal Communication...76 Regina Switalski Schinker Is Cost Analysis Underutilized in Decision Making? ........................................81 Nadini Persaud Is E-Learning Up to the Mark?. ..........................................................................83 Oliver Haas The Problem of Free Will in Program Evaluation........................................... 102 Michael Scriven PART II: Global Review—Regions & Events Japan Evaluation Society: Pilot Test of an Accreditation Scheme for Evaluation Training............................................................................................................ 105 Masafumi Nagao Aotearoa/New Zealand Starting National Evaluation Conferences ................ 107 Pam Oliver, Maggie Jakob-Hoff, & Chris Mullins
........... Coryn Evaluation: The International Journal of Theory............................... Volume 10(4)...........................................................................Conference Explores the Intersection of Evaluation and Research with Practice and Native Hawaiian Culture......................... 2003 .......... Volume 19(2)......... DC: Evaluation of Driver Education.......... 164 Daniela C................. ..................... 137 Thomaz Chianca PART III: Global Review—Publications Summary of American Journal of Evaluation........ Mark New Directions for Evaluation............. 157 Brandon W. Risley Education Update........ S............................................................ October 2004 .................... Youker An Update on Evaluation in Canada ...................... 130 Chris L....................................... Coryn ......... S............................................ 111 Northport Associates African Evaluation Association . Coryn An Update on Evaluation in Europe.......... Schröter Measurement: Interdisciplinary Research and Perspectives.................... S....................................................... Volume 25(4)....................... 109 Matthew Corry Washington.......... 2004.............. 161 Chris L............... 125 Brandon W.............. 123 AfrEA International Association for Impact Assessment ............................................................. Youker Canadian Journal of Program Evaluation....... Research and Practice.... Schröter An Update on Evaluation in the Latin American and Caribbean Region ................ 132 Daniela C.................................... .... 145 Melvin M...... 150 John S........................................ Volume 1(1).... 168 Chris L........ Fall 2004 ... 153 Nadina Persaud The Evaluation Exchange—Harvard Family Research Project ................
Our list of 932 people who want to be notified of new issues now includes residents from more than 100 countries. and his proposed solution. That article and the other German contribution (on the evaluation of online education) are interesting not only for Journal of MultiDisciplinary Evaluation: JMDE(2) 1 . • There is an editorial by Jane Davidson on the perception of evaluation by others. have obvious generalizations to other areas of public and private investment. and there have been around 2. The current issue is a bit longer: it runs over 170pp.000 hits. Here are some highlights. who raises serious concerns about the crucial matter of how the big (U. Her approach also bears on systems theory and organization learning. in which Susanne Weber sets out an approach to monitoring and evaluation based on current abstract sociological theorizing. and (ii) he suggests a way to raise the standards considerably. in case those are interests of yours. but you can download just the parts that interest you.500 downloads of all or part of the first issue. He’s unlike most critics in two respects: (i) he went to Africa to check things out on the ground for himself. and other) agencies are evaluating their vast expenditures on development programs overseas.Editorial In this Issue: JMDE(2) Michael Scriven The journal homepage has had over 6. • One of the major articles is by Paul Clements. • The other major article is from Germany.S. You will no doubt realize that both the problem he writes about. and what we can and should do about it.
that canvas ideas we think deserve attention by evaluators. • We introduce a new feature—“Ideas to Consider”—for short pieces. The review of evaluation in Latin America and the Caribbean in the last issue has already been reprinted in translation.Editorial their content. The sixteen articles in this section tell a remarkable story: evaluation is changing the world and the world is changing evaluation! MS Journal of MultiDisciplinary Evaluation: JMDE(2) 2 . selected by the editors and ideally just of memo length. and one on the tricky problem of how to evaluate programs (or drugs) which depend on the motivation of users for their success (should attrition rate count as program failure or subject failure?). one on the role of body language in creating and countering evaluation anxiety. • Our strong interest in international and cross-cultural evaluation continues with an update on several of our previous articles covering evaluations in regions and publications around the world. but for the sense they provide of how evaluation is seen by scholars in Europe. as it well deserved. one on approaches to evaluating online education. There’s a quartet of these to kick the feature off: one on the still-persisting shortage of cost analysis in published articles and reports on evaluation. and its sequel here tells an impressive story of activity in that region.
Editorial Marketing Evaluation as a Profession and a Discipline E. After all. Right across the social sciences and in many other disciplines where evaluation is relevant in more than just its intradisciplinary application. Aotearoa/New Zealand It can be a bit like pushing sand uphill with a pointy stick. it seems that the vast majority of practitioners consider it to be part of their own toolkit already. Meanwhile. It’s all very well for us to come together in our evaluation communities around the world and talk to each other about our unique profession. I am hopeful we can persuade enough of a critical mass to call it a reasonable consensus. But with a little more application. recount a story where a group of practitioners from outside the discipline actually sat up and took notice. we are still working on building a shared understanding of what it is exactly that makes evaluation distinct from related activities such as applied research and organisational development. it seems to me that a more difficult yet equally important task is to articulate clearly to the outside world—to clients and to other disciplines—what it is that makes evaluation unique. I offer a few reflections on the challenges with this. and propose some possible solutions for moving us forward. albeit Journal of MultiDisciplinary Evaluation: JMDE(2) 3 . Jane Davidson Davidson Consulting Limited. In this piece. One of the great challenges in developing evaluation as a discipline is getting it recognised as being distinct from the various other disciplines to which it applies. Not that there isn’t a lot to talk about. as they say here in New Zealand.
1 For the full article. founded more often than not on sound theory. in this case from an industrial/organisational psychologist: [A] discipline evaluation is not. Evaluation is a helter-skelter mishmash. theory-driven. and offered—oh perhaps with an exception here and there—as a program of study at institutions of higher learning. and offered as programs in accredited colleges. notwithstanding the fact that it is a stew that has produced useful studies and results in a variety of fields. Here’s a fairly typical response. including education. The first doctoral degrees in industrial psychology did not emerge until about the 1920s. and community development enterprises.Editorial often under a different name. Most of these practitioners consider evaluators delusional when we suggest that evaluation is sufficiently distinct to call a profession. but obviously not quite young enough for its practitioners to recall the struggles they must have had in the late 1800s and early 1900s. without detracting in the least from its multitude of contributions and creative authors and practitioners. is not systematic. coherent.1 Industrial and organisational psychology is a relatively young discipline itself. let alone an autonomous discipline. grew out of a blend of industrial engineering and experimental psychology. universities. Perloff’s (1993) “A potpourri of cursory thoughts on evaluation.” Journal of MultiDisciplinary Evaluation: JMDE(2) 4 . Disciplines are systematic. see R. Industrial psychology. coherent. a stew of hit-ormiss procedures. Evaluation. which focuses primarily on personnel selection and ergonomics/human factors. and professional schools. mental health.
and a host of other disciplines. underlying logic. experimental psychology. The challenge here is convincing non-evaluators (such as the I/O psychologist quoted earlier) of this. I think it’s fair to say we have a critical mass who are quite clear that evaluation is at least a professional practice with a unique skill set that is honed with reflective practice and other forms of learning. Was it (or would it have been) reasonable to declare industrial psychology a discipline even though most insiders didn’t agree on its definition. These things take time.” There was probably a lot of dissent in the ranks as to whether it really was any different from industrial engineering. open minds. Journal of MultiDisciplinary Evaluation: JMDE(2) 5 . Whether or not we have the courage and conviction to declare ourselves a discipline at this point.Editorial It seems likely to me that the fledgling discipline of industrial psychology had its share of critics in those days. Sure. but there’s no point being put off by those who like to throw their hands in the air and declare the whole exercise impossible. It seems to me that little progress in theory or practice can be made beyond a certain point without first declaring evaluation to be a discipline and then seeing what develops. measurement. the almost universally held perception is that all one needs to do evaluation is some content expertise and perhaps a few measurement skills (and accounting skills). not everyone will buy the idea initially. Often trained to the graduate level in business and/or the social sciences. thinking and rethinking. Consultants are a particularly hard nut to crack. or the soundness of its theories? How much shared understanding constitutes a critical mass? There’s something of a “chicken and egg” argument here. And I am sure there were furious debates about the definition of industrial psychology itself. Perhaps it was even called a “helter-skelter mishmash.
Interestingly. especially underrepresented groups—don’t just use the regular channels that yield the same old candidate pools. go to where you know the right people are and personally encourage them to apply.g. they also sent out direct emails to evaluators who had been recommended by other evaluators and had the notice posted on an evaluation listserv. the RFP specifically stated that the client was looking for an evaluation expert with content expertise rather a content expert (e. [It is interesting to note that the process used by the client to specifically target evaluators closely mirrors best practice for the recruitment of top-notch job candidates.Editorial What could possibly make seasoned professionals such as management consultants sit up and take notice of evaluation? Let me set the scene. used a creative and unusual process to select the contractor. just two (yes. 2!) of these were from people who identified as evaluators and participated actively as members of the [national and international] evaluation community.] The client in this case. a management consulting or industrial/organisational psychologist) with evaluation experience. as distinct from the applied research skills a well-qualified management consultant or organisational psychologist might possess. Most clients are unaware that there is such a thing as evaluation expertise. Rather than asking Journal of MultiDisciplinary Evaluation: JMDE(2) 6 . This was despite unusual efforts on the part of the client to attract expressions of interest from evaluators.. which they had heard good evaluators do not usually respond to. This is very unusual in the evaluation of leadership initiatives. A client organisation had put out an RFP asking for an independent evaluation of a leadership initiative. under the guidance of an evaluator not bidding on the job. Of 22 initial expressions of interest in the contract. Rather than simply posting the RFP on the usual electronic bulletin boards.
larger consulting firms who do not have evaluation as their primary function are far more likely to have an extensive library of proposal templates and a number of junior staff trained in writing proposals. In contrast. In a small community such as New Zealand. After all. multinational business consulting firm (“Firm X”) whose quality assurance procedure consisted of appointing one of their independent auditors to oversee the evaluation. the selection team invited them to a face-to-face meeting where they could present their thoughts on the evaluation. Therefore. An added benefit of the face-to-face interview approach was that it increased the odds of both attracting and identifying a “real” evaluator. what better way to understand an evaluator’s grasp of his or her profession than to ask how his or her work should itself be evaluated? One case in point was a large. which the client felt couldn’t accurately be gauged without meeting the evaluator face to face. including an outline of their “quality assurance procedures.Editorial shortlisted bidders to submit the usual 20-page proposal. This was because credibility was a key element of the evaluation. they have inadequate resources to devote to compiling lengthy. As such. the standard written proposal solicitation process is far more likely to yield bids from content experts than from evaluators. slickly presented proposals that have less than an even chance of being successful.” The proposed quality assurance procedures turned out to be one of the more telling pieces of information. Prospective contractors were asked to submit a number of supporting documents for the interview. the vast majority of evaluators are solo practitioners who often partner with others for particular pieces of work. Journal of MultiDisciplinary Evaluation: JMDE(2) 7 .
And no. The client quietly pointed out that. by all accounts. The consultants from Firm X were flummoxed! Firm X asked who had been awarded the contract to evaluate the leadership initiative. I think we’ve all tried convincing the colleagues in our content disciplines that what we do is unique. And every now and then we get a breakthrough with our evaluation evangelising. When the final decision was made. they did send a junior employee to see the client to get feedback about why their bid had been unsuccessful. the runner-up was told the background and qualifications of the successful bidder—and immediately recognised who the competitor was (New Zealand being a small evaluation community). There is a wonderful lesson here for evaluation as it strives for recognition as a distinct profession and as a discipline. and something worth paying attention to. if they really were evaluators (as they claimed to be). They were even more surprised to be told that the main reason was because they were not evaluators.Editorial In the final round. To their credit. only the two “actual” evaluators passed the interview process and made it onto the final short-shortlist for being awarded the contract. But the reality is that evaluation-savvy clients will likely sell us more Journal of MultiDisciplinary Evaluation: JMDE(2) 8 . more than just measuring a couple of variables of interest. They then asked who the runner-up was. In contrast. By chance. complex. the two met up a few days later and had a chuckle when they finally connected the dots. and were told. they would already have found that out through their extensive evaluation networks—in the same way as the top two contenders had found out about each other. “audits” and “reviews” of the type they were well versed in were not the same as high-quality evaluations. extremely surprised not to make even the final short-shortlist of two. Firm X was.
the selection criteria. or the use of frameworks and models that have been developed specifically for evaluation. the quality of the selection process. A second client education strategy is to seek opportunities to help with the development of evaluation RFPs. SIOP has developed an extremely simple and straightforward leaflet. Like us. we should be sure to highlight it in a way that makes it easy for a client to tell a “real” evaluation from the rest. I/O psychologists also have trouble getting the general public (especially managers in organisations) to understand what it is they are particularly skilled to do. the application of evaluation-specific methodologies not known to our nonevaluator colleagues. Whatever it is. which it sends to members for distribution to managers they know. In response to this need. This was the case in the organisation I described. Although the organisation was constrained by regulations about how an RFP process could be managed. There is nothing quite like being denied a contract for not being an actual evaluator! What are some of the strategies we can use to educate clients? The simplest one that comes to mind is to highlight in our work what it is we are doing that is unique to evaluation. The third strategy for spreading the word about evaluation would be to follow the example of the Society for Industrial and Organizational Psychology (SIOP) in the States. This might be serious and systematic attention to utilisation issues. The goal was to have each member distribute the leaflet to five Journal of MultiDisciplinary Evaluation: JMDE(2) 9 . and client satisfaction with the outcome.Editorial converts among this audience than we could possibly manage for ourselves. and it made a very substantial difference to how well the task was outlined. good evaluative thinking allowed individuals within the organisation to generate a creative solution that led to the right result.
org/visibilitybrochure/siopbrochure.htm It is likely that by directing our educational efforts outwards toward clients. A copy of the leaflet may be viewed online at http://siop. which will in turn let us make better sense to the outside world. we will have the side effect of creating some better clarity within the evaluation profession. Journal of MultiDisciplinary Evaluation: JMDE(2) 10 .Editorial managers.
Kalamazoo. Ebenezer Aikins-Afful. MI 49006. September 1992. the World Bank planned to rehabilitate 1500 rural boreholes at a cost of $4. but the reasons for the reduction were not clear.Articles Monitoring and Evaluation for Cost-Effectiveness in Development Management Paul Clements2 1. Western Michigan University. and Kenya’s total fertility rate fell from 6. Department of Political Science. and 56% from incremental water consumed. Malawi: The World Bank. The plan had anticipated that 85% of project benefits would come from the value of time the villagers saved that they would have spent collecting water.” Lilongwe. The World Bank’s Fourth Population Project in Kenya aimed to decrease Kenya’s total fertility rate to six births per woman by improving family planning services. e-mail: clements@wmich. the rate of return was reduced to 14%. “Malawi Infrastructure Project: Mid-Term Review Report. 1992. 3 Carlos Alvarez.4 million. and 10% from the incremental water consumed. Appendix B. Development Assistance Requires a High Analytic Standard In the Malawi Infrastructure Project. Journal of MultiDisciplinary Evaluation: JMDE(2) 11 .4 in 2 Corresponding author: Paul Clements. two years later.3 No reason was given for reducing the estimate for time savings or for increasing the value for water consumption. with an estimated economic rate of return of 20%. Peter Pohland and Ashok Chakravarti. however. I was given it upon agreeing not to reference it. The economic analysis from the project plan comes from the Malawi Infrastructure Project’s Staff Appraisal Report. The project was approved in 1990. The Midterm Review estimated 31% of benefits from time savings.edu. At the project’s Midterm Review.
p. the capital city. DC: The World Bank. The completion report estimated actual returns at 18% because water production in 1991 was 10-20% below expectations. but its efforts to strengthen the National Water and Sewerage Corporation (NWSC) had been undermined by the government’s failure to raise water rates amidst hyperinflation and late payments on its water bills. “Form 590. and 4 World Bank.4 in 1993. 30.” (unpublished project implementation summary for Third and Fourth Kenya Population Projects).” Washington. 5 World Bank. The World Bank’s Water Supply and Sanitation Rehabilitation Project in Uganda aimed to rehabilitate the water and sewerage system in Kampala. 1991. DC: The World Bank.”4 Project activities. Washington. Its plan calculated a 20% economic rate of return based on incremental annual water sales of $5. “Project Completion Report: Uganda Water Supply and Sanitation Rehabilitation Project (credit 1510-UG).5 The project had indeed achieved its construction goals. Documents for the Fourth Population Project do not explain how its development objectives were related to the activities it funded. The NWSC would have been unable to maintain the system without ongoing support. were largely unsuccessful. however. mainly supporting the National Council for Population and Development. 1995.Articles 1989 to 5. There were many other development agencies with family planning projects in Kenya. and in six other major towns.” and a 1995 supervision report asserted that “The project development objectives have been fully met. some with much stronger performance. The project’s Implementation Summary Reports consistently indicated that “All development objectives are expected to be substantially achieved. Journal of MultiDisciplinary Evaluation: JMDE(2) 12 . and in 1994 a large part of the project budget was reallocated to the fight against AIDS.5 million from 1988 to 2014.
Development as if Impact Mattered: A Comparative Organizational Analysis of USAID.Articles indeed by 1993. the World Bank and CARE based on case studies of projects in Africa. an economic rate of return that anticipates 23 additional years of water sales based only on the current state of the infrastructure—is that even though at least the second two are at face value analytically incorrect. The projects were selected based on descriptions of less than a page with no information on results.7 What is remarkable about these inconsistencies—an economic analysis in a midterm review that does not follow from the one in the project plan. even with a major new project supporting the water company. organizational incentives for development agencies. doctoral dissertation for the Woodrow Wilson School of Public and International Affairs. and this common practice reflects a structural problem of accountability. 7 Along with four projects of the US Agency for International Development and four from CARE International. it was once more operating in the red. These incentives are “structural” in that they result from the pattern 6 Paul Clements. Journal of MultiDisciplinary Evaluation: JMDE(2) 13 . I would like to argue that the tasks undertaken by the large multilateral and bilateral donor agencies require a particularly high analytic standard. all located in Uganda. Kenya and Malawi. development objectives that do not reflect project activities. they are presented as routine reporting information. and personal incentives for managers—have led to positive bias and analytic compromise. with no attempt to hide them such as in obfuscating language. Indeed they reflect common analytic practice in the international development community. p.6 These examples come from a blind selection of four World Bank projects that I studied for my doctoral dissertation. Princeton University. 1996. but several incentives that influence development practice—political incentives for donor and recipient governments. 325.
Unlike private sector investments. but to improve conditions for a beneficiary population—to reduce poverty. and second. first. We can be confident that such an improvement is possible. a conception grounded in a view of its likely impacts. and its relations with its various stakeholders. a less well-educated population. Typically one needs to strengthen existing institutions or to build new ones. Journal of MultiDisciplinary Evaluation: JMDE(2) 14 . and more risk and uncertainty than in the environments facing most for-profit enterprises. and this paper proposes a possible solution involving a dramatic improvement in the quality and consistency of project evaluations. Donor agency officials need a conception of the relative merits of many actual and potential projects. Project managers need to maintain a unified conception of the project. or to contribute to economic growth. The problem therefore requires a structural solution. and no profit incentive to keep managers on task. development projects aim not to make a profit. in places that need development assistance one cannot assume that institutional partners will be competent and mission-oriented. Yet in the project environment there is likely to be weaker infrastructure. because the evaluation problem facing development agencies has determinate features with specific analytic implications. These conditions in combination place particular demands on development managers. Furthermore. and an analysis that turns problems on the horizon for developing countries into programmatic opportunities. and/or to encourage beneficiaries to adopt new behaviors and to take on new challenges. because a similar structural problem has already been addressed in the management of public corporations.Articles of the flow of resources inherent in development assistance. There is no automatic feedback such as in sales figures. development assistance comes down to designing investments and managing projects. its unfolding activities. Sooner or later.
5-6. We know. is primarily its own affair. “Guidelines: Procurement under IBRD Loans and IDA Credits. or so I will argue. however. There are two problems in maintaining the will and the capacity to address this challenge: an incentive problem and one we can call intellectual or cognitive. and substantial influence over project administration—but it is useful to 8 The World Bank. DC: The World Bank. The donor agency makes the loan. “The responsibility for the execution of the project. of course. Some might like to think that development can be achieved by getting governments to liberalize markets or by getting local participation in project management. 1985.” Washington. and therefore for the award and administration of contracts under the project.Articles The central challenge in the management of development assistance is to maintain this kind of consciousness—this analytic perspective—among the corps of professional staff. is strong evaluation. Journal of MultiDisciplinary Evaluation: JMDE(2) 15 . one might imagine. that there can be no formula for successful development. with the donor providing technical assistance upon request. that this image is incorrect—donor agencies typically have the predominant influence over project design. 2. Whether a government manages its projects well or poorly. pp. rests with the Borrower. Each investment presents a unique design and management challenge. But Accountability in Development Assistance is Weak 2.1 Donor agencies are responsible for the success of their projects According to the World Bank’s procurement guidelines. but it is entirely the responsibility of the borrower government to spend the money. Intuition suggests and experience teaches. and these may well be important tactics.”8 One might think that a development loan to a government is like a business loan to an entrepreneur. The key to solving both problems.
10 Many development loans. 7. are for projects at the edge of the borrower’s 9 International Bank for Reconstruction and Development. One reason is parallel to a private bank’s prudential interest in the management of its loans. and Richard Webb. 1989). The Bank shall make arrangements to ensure that the proceeds of any loan are used only for the purposes for which the loan was granted. 1748. p. 1991. John P. Mahn-Je Kim. so it may take pains to see that its loans are well spent. with due attention to considerations of economy and efficiency and without regard to political or other non-economic influences or considerations. Lewis. ed. so the Bank must retain enough control to ensure that the projects it supports are properly administered. As the World Bank’s Articles of Agreement state.” in Devesh Kapur.. Washington. “International Bank for Reconstruction and Development: Articles of Agreement (As amended effective February 16. 1997. Volume Two. By this logic we would expect relationships with bureaucratically stronger governments to be closer to the private sector model. Journal of MultiDisciplinary Evaluation: JMDE(2) 16 . 10 See e. pp. DC: The World Bank.9 The Bank wants to be repaid. however. Many loans are to governments with limited bureaucratic capacity in countries with inconsistent management standards. The World Bank: Its First Half Century. and it also has an interest in promoting economic growth and enhancing well-being in borrower countries. “The Republic of Korea’s Successful Economic Development and the World Bank. and indeed some governments with coherent industrial strategies (consider South Korea in the 1970s) have succeeded in using Bank loans very much for their own purposes. DC: Brokings Institution Press.” Washington.Articles recall why this is so.g.
Donor governments want their funds to contribute to the borrower’s development. and the Bank (like other donor agencies) is a repository of expertise in the sectors it supports. 1997. Tolbert. and World Bank loans to Côte d’Ivoire have been subject to particular influence from France.Articles frontier of technological competence. 109-160. Investing in Development: Lessons of World Bank Experience. and many governments have been unable independently to prepare proposals that the Bank could accept. 353. 12 Paul Clements. and. enmeshed in donor governments’ general promotion of their foreign policy agendas. Journal of MultiDisciplinary Evaluation: JMDE(2) 17 . so they insist that donor agencies take responsibility for project results.11 and the responsibility that comes with authorship cannot be lightly abandoned during implementation.13 Bilateral aid is even more closely linked to donor government interests than aid 11 Warren C. Lewis and Webb. 1999. p. A second reason that donor agencies take an interest in how their funds are spent is that donor funds come from (or are guaranteed by) governments. Baum and Stokes M. governments do not release funds without taking an interest in their disposition. on the other hand. The Bank also has demanding requirements for project proposals. pp. “Informational Standards in Development Agency Management. 1359-1381. the country’s former colonial master. 13 Jacques Pégatiénan and Bakary Ouayogode. particularly in earlier years when patterns of Bank-borrower relations were established. ed. the Bank’s Articles of Agreement notwithstanding.” in Kapur.” World Development 27:8. Therefore the Bank has generally taken primary responsibility for designing the projects it funds. Foreign aid is also. New York: Oxford University Press for the World Bank. On one hand this political logic reinforces the prudential logic discussed above.12 It matters that the United States is not indifferent as to whether and when the World Bank will make loans to Cuba. 1360. “The World Bank and Côte D’Ivoire. 1985. p..
the borrower and often the lender suffer a financial loss if the investment fails. on the individual and corporate interests of their leaders and employees. the parameters of development spending cannot be understood merely in terms of the requirements for maximizing development impacts. With private loans. Donor agencies have control over development monies but they face no financial liability for poor results (and no financial gain when impacts are strong). Not only from the donor side but from that of recipient governments too. In this context their orientation to their task will depend largely on the demands and constraints routinely placed on them by other agents in their organizational environment.Articles through multilateral institutions. of course. With most development projects. when we consider the incentives governing a donor agency’s management of its portfolio. create an accountability problem. But the treasury seldom has control over individual development projects. Journal of MultiDisciplinary Evaluation: JMDE(2) 18 . donor agencies are required to take more responsibility than private banks for managing the loans they make. that is responsible for payments. typically the treasury. The main cause for different incentive structures between donor agencies and private banks. neither the donor nor the implementing agency faces a financial risk if impacts are disappointing. however. and on the mechanisms of accountability that are institutionally (“artificially”) established.2 The usual watchdogs are not there to hold donor agencies accountable The structural conditions of development assistance. by contrast. arises from differential exposure to financial risk. For projects funded by loans it is the borrower government. As intermediaries between donor and recipient governments. 2. The analogy with the private sector breaks down even further. therefore.
15 Some promote humanitarian and progressive agendas. and the period between when monies are spent and when their results transpire is typically so substantial that effective oversight would require major bureaucratic capacity.14 The electorates in donor countries want to believe that aid is helping poor people. Wenar notes that there has been a “historical deficiency in external accountability” for donor agencies. but democratic politics also leads to pressures on donor agencies to support the agendas of well-organized interest groups. but external oversight of project effectiveness faces major practical hurdles. Department of Agriculture. 2003. since the intended beneficiaries of aid cannot vote in donor country elections. Aid organizations have evolved to a great extent unchecked by the four major checking mechanisms on bureaucratic organizations. 296. These four mechanisms are democratic politics.S. p. American farmers have influenced U. press scrutiny. and it is the aim of this paper to suggest how these conflicts could be. in any case.S. however. if not removed. This clearly leads to conflicts of interest. Philosophy & Economics.Articles In regard to external agents. “What we owe to distant others. the reliability of democratic politics as a source of accountability is limited. regulatory oversight. Aid projects are so widely dispersed. 15 For example. There has been significant regulatory oversight aiming to ensure that aid funds are not fraudulently spent. Donor agencies have not.” Politics. Journal of MultiDisciplinary Evaluation: JMDE(2) 19 . 283-304. which are overseen by the U. has normally rested with the donor agencies themselves. at least substantially ameliorated. Responsibility for project evaluation. 2:3. food aid programs. been subject to 14 Leif Wenar. Generally. but others have aims that create tensions with development goals. and academic review.
and academic studies have contributed to many foreign aid reforms. Few who have spent much time with development agency personnel can doubt their generally admirable commitment to development goals. they are largely dependent. however. have been significant sources of accountability. in contrast. Second. for information on aid operations on the donor agencies themselves. while institutional norms require donor agencies to maintain the appearance of a coherent system of responsibility for results. however. Also. and these interests take shape in the specific task environments that they face in their home offices and in the field. Given the strength of the political and bureaucratic interests that drive the programming of aid. scholars and journalists can only be expected to hold aid agencies accountable in a limited and inconsistent manner. Since donor agencies have generally controlled their own evaluation systems. and the reforms this paper will propose depend heavily on the personnel’s sustained interest in professionalism and effectiveness. by their individual and corporate interests. Press scrutiny and particularly academic review. First and most obviously. internal personnel evaluations have tended to focus on variables only loosely Journal of MultiDisciplinary Evaluation: JMDE(2) 20 . There are two aspects of the way their interests come to be constructed that are particularly relevant to the problem of accountability. due in part to the long time span between the commitment of funds and the evaluation of results. they have had the opportunity to design these systems in such a way that they would tend to reflect positively on the agencies themselves. They must evaluate. Their behavior is also influenced.Articles significant external accountability by way of regulatory oversight. but it serves their individual and corporate interests if evaluation results are generally positive (or at least not often terribly negative). their institutional relationships require them to maintain the appearance that their operations are generally successful. and the above-noted dispersal of aid projects.
p.” The classic account of the “money-moving syndrome” is Tendler’s Inside Foreign Aid.Articles correlated with good results. Ibid. 88-96. reinforcing the present argument about evaluation. and sometimes on variables that conflict with good practice. other less relevant criteria inform resource allocation decisions We will later consider some of the approaches donor agencies have taken to evaluation below. Perhaps the most longstanding and sustained critique of donor agencies’ internal operations involves the imperative to “move money..”17 In the context of her organizational analysis. Journal of MultiDisciplinary Evaluation: JMDE(2) 21 . it is enough for now to note that donor agencies have controlled their own evaluation systems.18 Tendler also finds. 2. Inside Foreign Aid.S. p. For the purposes of understanding the accountability problem in development assistance. she gives several examples of aid officials knowingly supporting weak projects in order to reach spending targets. Baltimore: Johns Hopkins University Press. 88. Ibid. 1975. Tendler identifies a “pressure to commit resources that is exerted on a donor organization from within and without. 16 17 18 Judith Tendler.3 Lacking secure accountability for results.16 Focusing on the U. Agency for International Development (USAID) and the World Bank. In the context of the general deficiency in external accountability.” and finds that “standards of individual employee performance … place high priority on the ability to move money. the priorities that have been enforced within donor agencies take on particular significance..
and arguments were left out.”21 While Tendler offers several political and organizational reasons to explain the money-moving imperative. but Tendler found that many ostensibly economic projects were selected by noneconomic criteria. an AID technician might be remonstrated with. all in order to alleviate the uncomfortable feeling of responsibility for possible betrayal. for all communication through language. to act like policy 19 20 21 Ibid.19 The World Bank typically required economic analysis of proposed projects. 95. p. p. once they win some organizational power. In such an environment we might expect well-intentioned professionals. aid officials learned to self-censor reports that could provide ammunition for critics. Journal of MultiDisciplinary Evaluation: JMDE(2) 22 .Articles that in a political environment often hostile to foreign assistance. p.. inevitably. We do not find a sustained effort to consider how development funds can be employed to maximize their contribution to development. For writing what he considered a straightforward description of a problem or a balanced evaluation of a project. 51. thoughts were twisted. I would like to emphasize what is absent from the organizational culture she describes. Ibid. 93... … Such a situation must have resulted in a certain atrophy of the capacity for written communication – and. Ibid. “What would Congress or the GAO [General Accounting Office] say if they got hold of that!?” … Words were toned down.20 Much of the economic analysis that was carried out amounted to a “post hoc rationalization of decisions already taken.
”23 noting also that “[t]he methodology for project performance rating is deficient. Journal of MultiDisciplinary Evaluation: JMDE(2) 23 .”22 in part because “signals from senior management are consistently seen by staff to focus on lending targets rather than results on the ground.” Washington.. promoting their individual conception of a good development agenda in large measure despite the prevailing incentives. iii. iv.Articles entrepreneurs. p. The World Bank’s “project approval culture” was recognized in its internal 1992 study. Ibid. 1992. the Bank has moved increasingly to spending modalities that further dilute accountability for results. p. is that organizational decisions will routinely be taken on the basis of expected impacts. What we cannot expect. as I believe reproductive health professionals at USAID have done. “Effective Implementation: Key to Development Impact. DC: The World Bank. 23. The report cites a “pervasive preoccupation with new lending. Ibid.”25 Since the appearance of the Wapenhans Report. sectoral) and 22 Portfolio Management Task Force. 23 24 25 Ibid. The two kinds of programs that have become most central to Bank strategies particularly in lower income countries are adjustment loans of various kinds (structural. “Effective Implementation: Key to Development Impact” (popularly called the Wapenhans Report). it lacks objective criteria and transparency.”24 Although the report describes the Bank’s evaluation system as “independent and robust. p. We might expect segments of a donor agency that have strong external allies to develop coherent agendas that they can implement themselves..” it finds that “[l]ittle is done to ascertain the actual flow of benefits or to evaluate the sustainability of projects during their operational phase. however.
53-69. Journal of MultiDisciplinary Evaluation: JMDE(2) 24 . to identify a set of indicators for measuring the strategy’s impacts. it is questionable whether governments of low-income countries will be able to do much better. There is often a feeling that they are imposed. They ask the government. If the World Bank has had such a hard time ascertaining the level and sustainability of impacts from its own portfolio. what is at stake. “Poverty Reduction Strategy Papers: A New Convergence.1 The basic idea of the proposed evaluation approach The problems discussed above present formidable obstacles to maintaining accountability in foreign assistance on the basis of program and project results. as the government receives the loan for policy changes it presumably otherwise would not have made. They tend to operate on a wider scale than traditional projects. Independent and Consistent Evaluation Can Improve Accountability and Learning in Development Assistance 3. and it seems that their more participatory approach to policy formation and implementation is intended to substitute. We should recall. Poverty Reduction Strategy Papers typically push a larger part of the responsibility for evaluation onto the borrower government.26 Adjustment loans require borrowers to adopt free market reforms in order to better align economic incentives with development goals. 3. for rigorous agency evaluation. as part of the process of generating a poverty reduction strategy. however. and they are often implemented only partially and inconsistently. to some extent. 2003. These factors make them harder to evaluate. In the absence of meaningful accountability there is little to counter-balance the pressures for aid resources to 26 David Craig and Doug Porter. with more diffuse impacts.Articles Poverty Reduction Strategy Papers (PRSPs).” World Development 31:1. however.
616-641.27 One way to address the historical deficit of external accountability.Articles support political interests of donor and recipient governments. there have been persistent doubts about the basic effectiveness of development assistance at improving economic and/or social conditions in recipient countries. http://www. second edition. however. Journal of MultiDisciplinary Evaluation: JMDE(2) 25 . and personal interests of management stakeholders. policies and growth. Radelet and Bhavnani find positive country-level economic impacts from aid based on cross-country econometric studies focusing on the approximately 53% of aid that one would expect to yield short term economic impacts. John Hudson. 289-329.” The American Economic Review.org/Publications/?PubID=130. and to improve the incentive and the capacity to manage for impacts. “Aid. have failed to find evidence of positive impacts from foreign aid. 1987. 847-868. A series of cross-country econometric studies. “Politics and the effectiveness of foreign aid. These include Paul Mosley. in a recent paper by Michael Clemens. Boone. Does Aid Work? Report to an Intergovernmental Task Force. 40:2. for example to push the focus of management attention forward from moving money to achieving results. and Sara Horrell.cgdev. Clemens. 2000.” The Economic Journal. In their comprehensive 1994 review of foreign aid on the basis of donor agency documents. The inconsistency and mixed reliability of evaluations have also undermined learning from experience. 97:387.” European Economic Review. These results are reviewed and contested. the public sector and the market in less developed countries. UK: Oxford University Press. “Aid. and Craig Burnside and David Dollar. organizational interests of donor and implementing agencies. Steven Radelet and Rikhil Bhavnani. so the aid community has been slower than it would otherwise have been to identify successful strategies and to modify or abandon weak ones. Robert Cassen and associates find that most project achieve most of their objectives and/or achieve respectable economic rates of return. P. “Counting chickens when they hatch: The short-term effect of aid on growth. 2004. however. is to 27 Indeed despite development agencies consistently reporting positive results from their overall operations. 1996. 90:4. Oxford.” Center for Global Development Working Paper 44.
In this way rewards and sanctions can be allocated based on contributions to impacts and managers can gain a feel for what is likely to work in a new situation. the appropriate frame of reference is not the individual project but the donor agency’s overall portfolio. 28 Impacts are defined as changes in conditions of the beneficiary population due to the project. the world-wide distribution of similar and near-similar projects.e.Articles institute independent and consistent evaluations of the impacts and costeffectiveness of donor-funded projects. and finally consider how it compares to established evaluation approaches.28 This would mark a significant departure from existing practice. For both accountability and learning. for learning. Journal of MultiDisciplinary Evaluation: JMDE(2) 26 . so I first explain the concept. They start. and. Both accountability and learning.) how to configure the project design so as to maximize impacts. start from impacts and then work backwards in causality. In both cases the relevant conception of impacts is one that supports comparisons among projects. compared to the situation one would expect in the project’s absence (compared to the counterfactual). in light of how the project is unfolding. for managers. then suggest how it could be implemented. for example. for donor agencies. The project planner or manager’s question is. i. in light of relevant features of the beneficiary population and the project environment. learning uses it to construct lessons based on schemes of similarities (so the lessons can be applied to other contexts). The donor agency’s question is how to allocate its resources so as to maximize the impacts of its overall portfolio. (and. with strong or weak results. and while accountability uses the discovery of causes to allocate responsibility.
across countries and donor agencies. and the contexts in which they work are so often difficult and demanding. but in terms of impacts expressed in consistent and comparable units. (or.) evaluations should be conducted in consistent units. Cost-effectiveness aims to achieve the greatest development impacts from the available resources.29 and the multilateral development banks (such as the World Bank) have historically evaluated most of their projects in terms of their economic rates of return. 3. Accountability and learning bear a heavier burden than in other contexts. therefore. Bilateral donor agencies have typically evaluated their projects in terms of how far they have achieved their stated objectives. idiosyncratic objectives. We can compare evaluation in terms of cost-effectiveness with two other approaches that donor agencies have often used. for a given type of project. Accountability requires consistent units precisely because the appropriate frame of reference for accountability is the donor agency’s overall portfolio. It applies with particular force to large donor agencies because their other sources of accountability are so sparse. An evaluation’s units of analysis establish a perspective or orientation in terms of which the project and its activities come to be understood. 29 This is the 'logical framework' approach.Articles Now this logic may sound quite general. the tasks they undertake are so costly and complex. Journal of MultiDisciplinary Evaluation: JMDE(2) 27 . In order to establish a consistent orientation across a donor agency’s portfolio.2 Comparing projects in terms of cost-effectiveness I would like to suggest that the unit that provides the appropriate frame of reference for donor agency evaluations is cost-effectiveness. This is why projects should be evaluated not by the extent to which they achieve their individual.
the criterion “to achieve objectives” bears no clear relation to costs. A project’s economic rate of return (ERR) expresses the relation between the sum of the economic value of its benefits and its costs. 31 J. Baltimore. If this criterion is taken as the basis for accountability.Articles When projects are evaluated in terms of their objectives. the ERR is the discount rate at which the discounted sum of benefits minus costs is equal to zero. and the World Bank has typically expected an ERR of 10% or higher from its projects in the economic sectors. comparisons among projects are likely to be misleading.31 For an evaluation system based on costeffectiveness. The difference between cost-effectiveness as I am defining it and an ERR is that an ERR measures benefits in terms of their economic values (ideally at competitive market prices) while costeffectiveness measures benefits in terms of the donor’s willingness to pay for them. 1982. including those that may be expressed in qualitative as well as in 30 Specifically. Also. Journal of MultiDisciplinary Evaluation: JMDE(2) 28 .30 It can also be described as the return on the investment. Some projects have ambitious objectives while others are more modest. Economic Analysis of Agricultural Projects. and benefits to the poor generally count the same as benefits to households that are already well off. Price Gittinger. second edition. so a project that achieves a minority of its objectives can clearly be superior to another that achieves most of its own aims. it establishes an incentive to set easy targets and/or to over-budget. a donor would need to establish a table of values specifying how much it is willing to pay for each of the various benefits that it expects from its projects. MD: Johns Hopkins University Press. The economic analysis of projects typically does not include improvements in health or education.
for example.33 and carries out the appropriate quantitative and/or qualitative analysis of the project’s activities and their results. The project’s total impacts are compared to its costs based on the donor’s table of values. 577-592. indicates the range of cost-effectiveness scores in which she is confident that the true value of likely impacts lies. an evaluator estimates the sum of impacts up to the present point in time and the magnitude of impacts that can be expected in the future that can be attributed to the project. At the completion of a project.32 3. Journal of MultiDisciplinary Evaluation: JMDE(2) 29 . 1995. on a scale from one to six.” World Development. 33 These may be found in the project plan. In this way a basis would be established for comparing. and on this basis the evaluator estimates the project’s likely costeffectiveness. The impacts are summed together with appropriate weights from the table of values and compared to costs. To estimate impacts the evaluator lists the relevant impacts from a project of the present type and design. for example. the proposed evaluation approach would work like this.3 The proposed evaluation approach in practice In practice. 23:4. The evaluator assigns each form of impact a numeric value and/or a qualitative rating based on his/her judgment of the project’s likely effects in the respective areas over the lifetime of the project’s influence. The evaluator also notes her degree of confidence in the cost-effectiveness score. if her confidence is moderate or low. primary health care and agricultural extension projects.Articles quantitative terms. with one representing failure and six indicating excellence (see Table 1). “A Poverty Oriented Cost-Benefit Approach to the Analysis of Development Projects. and. and on this basis the evaluator estimates the project’s cost-effectiveness. In this case she also specifies the additional 32 This approach to establishing the value of project impacts is described in Paul Clements.
First it does not address the bias arising from donor agency control of the evaluation process.4 An evaluation association to address bias and inconsistency While evaluations in terms of cost-effectiveness may support learning and accountability. Third. the results are often highly sensitive to minor changes in assumptions.19. The estimate of the project’s cost-effectiveness anchors the evaluator’s analysis of the project’s design and implementation. even where accepted methodologies are available (such as economic cost-benefit analysis.g. for economic projects).9% Below 0% Degree of Cost-Effectiveness 6 5 4 3 2 1 Interpretation Excellent Very good Good Acceptable Disappointing Failure 3.4. present methodological challenges.9% 10% . There are no widely accepted methodologies for some of the required impact estimates (e. there are (at least) three problems with the proposed evaluation approach. different Journal of MultiDisciplinary Evaluation: JMDE(2) 30 . All four components—the analyses of impacts.29. cost-effectiveness. for reproductive health and AIDS education projects). When evaluations are contracted out on a project by project basis. design and implementation—serve as a basic unit to support accountability and learning within the project and the donor agency and across the development community. Table 1: Scale of Cost-Effectiveness Economic Rate of Return 30% and above 20% . including impacts in the future.9% 0% .9% 5% .9. Second the estimates of impacts that it requires.Articles information that could plausibly be collected that would allow a more precise estimate.
accountants and auditors are employed by the very managers whom they are expected to hold accountable. Society depends for its operations on some basic presumption of trust. however.Articles assumptions are likely to be applied to different evaluations. the consequences of major lapses can be quite severe. It is these rules that are the source of accountants’ and auditors’ independence from corporate management. but managers face incentives to use the resources for their private purposes. especially in relations involving large and complex organizations. which depends in turn on guarantees of disclosure and lucidity. These associations establish qualifications that their members must achieve and rules that they must follow in order to retain professional membership. As with evaluators in foreign aid. as evidenced by the collapse of the international accounting firm Arthur Anderson after the accounts it managed for the Enron Corporation were found to be unreliable. and to ensure that they have mastered the relevant techniques. accountants and auditors have established professional associations. Although independence is not maintained perfectly. In order to protect their independence from management. and it is the task of accountants and auditors to ensure that these rules are followed. Amartya Sen lists transparency guarantees as one of five sets of instrumental freedoms that contribute to people’s overall freedom to live the way they would like to live. undermining the validity of comparing and aggregating their results. There are strong parallels between the conditions for the problem of bias and inconsistency in the evaluation of foreign aid and conditions facing public corporations in the management of their internal finances. Sen points out that where Journal of MultiDisciplinary Evaluation: JMDE(2) 31 . Stockholders want corporation managers to employ the corporation’s resources in such a way as to maximize profits. There are elaborate rules governing how managers may appropriately use a corporation’s resources.
In order to ensure that impacts are estimated and then compared in common units.34 A professional association of development project evaluators could play a role guaranteeing disclosure and lucidity in the management of international development assistance similar to that of associations of accountants in the management of public corporations. The technical difficulties in estimating project impacts are objective problems. as they appear to be in foreign aid. financial irresponsibility. Journal of MultiDisciplinary Evaluation: JMDE(2) 32 . One task in establishing the association would be to work out impact assessment approaches for different kinds of projects. the association would establish a constitutional principle asserting that each endof-project evaluation conducted by its members would estimate the project’s impacts and cost-effectiveness to the best of the evaluator’s ability. and procedures for expelling members who fail to uphold the standards. these approaches would be refined. 38-40. Over time. such an association would need the same structural features as associations of accountants— qualifications for membership. 34 Amartya Sen. 1999. Such an association could also address the problems of estimating project impacts and of comparing impacts in common units. so it is possible to identify principles and practices for addressing them. pp. In order to address the problems of bias and inconsistency. society is vulnerable to corruption. as its members gained experience. New York: Knopf Publishers. An evaluation association would provide a forum for identifying better evaluation approaches and for ensuring consistency in their application. and underhand dealings. a set of rules and standards governing how evaluations are to be carried out. Development As Freedom.Articles these guarantees are weak.
2000. Thousand Oaks. 1980. 1994. CA: Sage Publications. it would be a simple operation for someone planning. “Community Organization and Rural Development: A Learning Process Approach.36 whom 35 See e. 4. to assist them in taking responsibility for improving their own conditions and to incorporate them in more democratic processes of development decision making.g. E. B. an urban water project.1 Monitoring and evaluation for empowerment To explain M&E for cost-effectiveness it is useful to compare it with other evaluation approaches. Cracknell. 36 David C.” Public Administration Review. say.Articles There are dozens of donor agencies and many thousands of implementing agencies in the development assistance community. 480-511. which would be accessible to the entire development community. The universe for the evaluation association’s operations would be the development assistance community overall. Korten. 40:5. to review the approaches of the five to ten most cost-effective water projects in similar environments. Robert Chambers. Evaluating Development Aid: Issues. Since each evaluation estimates the project’s cost-effectiveness. The strongest challenge to standard approaches to aid evaluation in the last two decades has involved the elaboration and application of participatory approaches.35 These have aimed to involve beneficiary populations in project management. Monitoring and Evaluation (M&E) for Cost-Effectiveness Compared to Other M&E Approaches 4. and each agency has its own management culture and approaches. Problems and Solutions. Authors such as Korten and Chambers. Each evaluation completed by a member of the association would be indexed and saved in an online repository. “The 33 Journal of MultiDisciplinary Evaluation: JMDE(2) . and it would support learning about better practices on the basis of the type of project throughout this community.
Robert Chambers. 27:8. Journal of MultiDisciplinary Evaluation: JMDE(2) 34 . Origins and Practice of Participatory Rural Appraisal.” World Development. 1994. that participatory approaches are right for all projects. It turns out. 22:10. While M&E for cost-effectiveness appreciates that empowerment is an important development goal. They have promoted an approach I call “M&E for empowerment” because it emphasizes learning at the local level.Articles Bond and Hulme describe as “purists.”37 have sought to reorient the development enterprise to support the goal of empowerment. 1999. Evaluation and empowerment goals overlap in their management implications. Donor agency officials are the primary audience for aid evaluation because they exercise primary control over these resources. “Process Approach to Development: Theory and Sri Lankan Practice. so M&E for cost-effectiveness would promote participation in these cases. Bond and D Hulme. 953-969. M&E for cost-effectiveness does not assume. 1994. the legislatures that appropriate aid budgets. with some additional synthesis.” World Development 22:9. “Participatory Rural Appraisal (PRA): Challenges. In many instances participatory strategies are more cost-effective than projects based on so-called blueprint approaches. The empowerment of project beneficiaries is interesting from an analytic viewpoint. Robert Chambers. seeking to empower project beneficiaries by involving them in the evaluation process. 1253-1268. however. that the form of evaluation that can best inform these officials will also best inform officials of developing country governments. 1437-1454. 1339-1358. 37 R. and the overall development community. however.” World Development. project managers. p. 1340.” World Development. and empowerment was certainly neglected by the development community prior to the mid-1970s. it identifies the locus for the primary learning that evaluation should support among those who are responsible for resource allocation decisions. Potentials and Paradigm. as well as. “Participatory Rural Appraisal (PRA): Analysis of Experience. 22:7.
Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners. As an end. and that impacts should be understood as changes in the conditions 38 E. 2000. Judy L. DC: The World Bank. Baltimore. Lury. so that through appropriate comparisons changes attributable to the intervention can be identified in a statistically rigorous manner. more or less imperfect approximations to randomized and controlled double-blind experiments.g. M&E for cost-effectiveness considers empowerment like any other possible means to be considered in program design. 1982. As a means. Dennis J. Monitoring and Evaluation of Agriculture and Rural Development Projects. I call it “M&E for truth” because it emphasizes making statistically defensible measurements of project impacts. M&E for cost-effectiveness considers successful empowerment to be a benefit which must be valued and counted along with other benefits in the assessment of a project’s cost-effectiveness. Journal of MultiDisciplinary Evaluation: JMDE(2) 35 .2 Monitoring and evaluation for truth It is possible that a great practical barrier to useful evaluation arises from some of those most knowledgeable of and committed to evaluation as a science. in effect. Baker.Articles because it can be seen both as a means to improving project designs and as an end in itself. This approach often uses household surveys that measure conditions that a project seeks to influence. Washington. 4. Casley and Denis A. For this reason M&E for cost-effectiveness views empowerment in a dual light. It has been common practice to begin discussions of aid evaluation methodology with the experimental method of the natural sciences. MD: The Johns Hopkins University Press. This approach is right to insist that projects should be assessed primarily on the basis of their impacts. Under M&E for cost-effectiveness both more and less participatory projects are considered within the same evaluation framework.38 and to present the various evaluation methods as.
baseline surveys often provide critical information for estimating impacts. Evaluators locate the present project along the continuum established by other similar projects based on how its design hypothesis unfolded as compared to theirs. evaluations that adhere rigorously to the maxims of statistical rigor seldom estimate the future impacts that can be attributed to a project. It is the evaluator’s task to analyze how this hypothesis has unfolded. M&E for cost-effectiveness employs participatory methodologies in many instances to elicit beneficiaries’ judgments of the significance of project outputs. but in the causal models of change inherent in project designs. and evaluations that do not often use the demanding requirements of statistical rigor as an excuse not to address the question of impacts at all. so impact estimates for other similar projects serve as a first approximation for the benefits that may be anticipated from the present project. and statistical methodology of course provides central criteria for their analysis.Articles of the population compared to what would be expected in the project’s absence (in evaluation jargon. Also. however. Monitoring and evaluation for cost-effectiveness is methodologically eclectic in its effort to reach reliable judgments of cost-effectiveness. It is arguable. as compared to the counterfactual). that in its orientation to statistical rigor it has established a “gold standard” that many evaluators are all too quick to disavow. A given project is taken as an instance of a project of its type. and on this basis to estimate the quantity of benefits that beneficiaries are likely to realize. As suggested above. The evaluator’s final estimate of a Journal of MultiDisciplinary Evaluation: JMDE(2) 36 . Only a very small proportion of project evaluations present statistically rigorous impact estimates. Each project design presents a hypothesis as to the changes in beneficiary conditions that can be expected from the actions the project undertakes. Clearly. It is grounded not in the first instance in the scientific method.
and to establish monitoring systems that would track the relevant impacts (or their main contributing factors) through the life of the project. the discussion up to this point has focused on evaluation only. however. Development professionals would become Journal of MultiDisciplinary Evaluation: JMDE(2) 37 . but one could anticipate.Articles project’s impacts and cost-effectiveness. For the proposed evaluation approach to address the accountability problem in foreign aid. if the proposed evaluation approach were implemented. A More Analytic Development Assistance Community Although I have described the proposed approach as monitoring and evaluation for cost-effectiveness. that its effects would gradually suffuse through all stages of project planning and implementation. is based on triangulation taking into account all the forms of information we have so far considered. The development assistance community as we know it has evolved under conditions of inconsistent and often limited and biased evaluation. The development assistance community would soon learn what outcomes need to be tracked for different kinds of projects to inform subsequent impact estimates. it is essential that planners and managers should know in advance that upon its completion there will be an independent evaluation of their project’s impacts and cost-effectiveness. 5. It would soon be taken for granted that when targets or systems for estimating impacts are altered during project implementation. While many individuals and groups in the contemporary development community are engaged in promoting development agendas of their own conception. Project planners would soon learn to include an estimate of cost-effectiveness in their project designs. however. the proposed reforms would enhance the experience of development work as a cooperative venture with shared goals. the reasons for these changes should be clearly documented.
The development community would be quicker to identify and to adopt more successful strategies. The general public has tended to be fairly skeptical of foreign aid.Articles more confident that others would endorse their sound justifications for their management strategies. and their shared concern for efficiency would be enhanced. for example. and management strategies would be more rigorously grounded in expected impacts. by country or by agency. The tax-paying public would receive better information on the consequences of foreign aid. 39 I found large scale corruption in only one of the 12 projects in the sample for my dissertation. In due course this could be expected to increase the generosity of citizens in the wealthier countries towards people in need. Although I believe that outright corruption on any significant scale is uncommon in the development community. and they would have better grounds for confidence in its integrity. The proposed reforms would make it straightforward to aggregate project impacts. and the management standards described in this article provide good reasons for skepticism.39 the higher analytic standard that the proposed reforms would bring about would reduce corruption even further. Members of the development community generally would become more conscious of the pathways by which their actions contribute to improvements in beneficiary conditions. Journal of MultiDisciplinary Evaluation: JMDE(2) 38 .
2004 Journal of MultiDisciplinary Evaluation: JMDE(2) 39 . From these can be generated questions for decisions in monitoring and evaluation within complex actor settings as well as criteria for concepts of evaluation and monitoring in a networking context. 35.fh-fulda. and applications serve numerous possible 40 Corresponding author: Susanne Weber. one set of criteria for the development of evaluation designs in a networking context. Department of Social Studies.de or susanne. four dimensions of network evaluation and monitoring are suggested and they are embedded in the multi-dimensional design approach of the “learning network” which puts collective competence development and future. Paper prepared for European Evaluation Society Sixth Conference.Articles Network Evaluation as a Complex Learning Process Susanne Weber40 The following contribution will explicate. telephone: 0049-6619640-224. Needs for evaluation and monitoring that is action.and futureoriented lead to other needs already established by social-ecological planning theory. Networks: Between myth. Fulda. Marquardstr. D-36039 Fulda.weber@sw. Berlin. On this basis.D. PH. economic or social.de. management and “muddling through” Networks are by now being discussed in all disciplines of the social sciences as the new paradigmatic form of organization and pattern for action. their application contexts can be political. based on an understanding of networking as a reflexive process and on an approach working from a theory of regulation. e-mail: firstname.lastname@example.org effect-orientation at the center of the developmental process. Germany. There are divergent assumptions about their status and range of applicability. University of Applied Sciences. 1.uni-marburg.
rationalization. Wolf 1999). society for instance is now once more increasingly explained in terms of network theory (Castells 2000. a ‘warmth metaphor’ including an accentuated demand for initiative: regions shall be guided out of their passive role. a line of orientation. it is often very nearly mythologized (Hellmer/Friese/Kollros/Krumbein 1999). and as a term employed almost as universally as the term “system” (Grunow 2000:314). e. a factor conducive of democratization and successful cooperation. with “network” as one of the basic social categories. market presence. Networking is also the point of departure for more or less close forms of cooperation in a regional context. Sydow suggests a tighter definition. maintaining and integrating patterns (ibid. often initiated by support programs and generating research interest in practice and action.g. intermediary agencies for regional learning networks are created which are supposed to tie different social fields together. As part of regional networking processes. and thus a higher degree of intensity for networking. For these reasons the term “network” is defined as a “compressed term” (Kappelhoff 2000:29): networking represents a perspective of hope. but with a “school of thought.Articles networking goals and purposes. characterizing it from a micro-economic perspective on company networks as “a form of organization of economic activity by enterprises which are independent by law but more or less interdependent economically”. the “Learning Region” program. It is used to represent a variety of possible meanings and forms of cooperation with different degrees of intensity: following Simmel (1908). taking on an active part in dealing with their concerns” (Gnahs 2003:100). The relations Journal of MultiDisciplinary Evaluation: JMDE(2) 40 . giving orientation. Messmer 1995. to give creative support (Jutzi/Wöllert 2003:130) and to serve as bridges for the initialization of regional processes by defining needs.:135). professional optimization. Here we are dealing not only with a clear accentuation of the term “network”.
and stable versus dynamic networks (ibid. by their type of control and the stability of their relations (stable-dynamic) (ibid.:81). Due to their structural complexity. is to be expected.: 13f). In this paper. For applications in competence development Duschek and Rometsch (2004) suggested grouping the various network types into three main types: explorative versus exploitative. which will be further discussed below since it can be made productive for the analysis and design of monitoring and evaluation in network cooperations. They can be categorized e. they are created endogenously or induced exogenously and represent more or less “polycentric systems” (Sydow 2001:80). the commitment that becomes relevant in a networking context reduces the autonomy of the individual network partners.g. etc. network cooperation is discussed on a social-scientific basis. and tensions. hierarchic versus heterarchic. Risk and conflict are inherent to the structure of such institutional and organizational cooperations (Messner 1995). The theoretical framing of the network and network arrangements proves to be of decisive importance for the design of network cooperation as well as for the evaluation-theoretical and conceptual position of monitoring and evaluation in a network. They are relatively stable.Articles that are introduced here are reciprocally complex and rather cooperative than competitive. a chance for creating common space for development curtails the flexibility of individual network actors. (ibid. risks. Theoretical perspectives that have complexity (Kappelhoff 2000) and structuring (Sydow 1999.:2). Sydow (1999) presented the model of structural tension in network cooperation. as a social process in which the surfacing of specific conflict potentials. Windeler 2001) as Journal of MultiDisciplinary Evaluation: JMDE(2) 41 . they are not always at an advantage over other forms of organization: Steger (2003) identifies contradictions like self-interest versus collective interest: in building common structures of action.
Articles their starting point are capable of representing and analyzing this topic in a way that is adequate for design practice and network management at the same time. The different levels of actors become relevant for the employment of complex monitoring and evaluation in networking Journal of MultiDisciplinary Evaluation: JMDE(2) 42 . The five characteristics of network regulation show us consequences for the design of monitoring and evaluation. In the context of a regulatory system monitoring and evaluation are also regulated and constitute themselves during that process. The approach of network regulation as a theoretical foundation for monitoring and evaluation in networks The network regulation approach offers criteria for the conceptual level of monitoring and evaluation in networking contexts. Multi-dimensional regulation means that divergent interests on the different levels are regarded as structurally unavoidable. It regulates itself systemically and contextually (Windeler 2001:203f). 2. as a collective social setting. they are part of the system and are systemically generated by it. From the perspective of a theory of structuring. the group. Multi-dimensional regulation The existence of different levels of actors is characteristic for networks: that of the individual. which transcends the static look at organizational networks. monitoring and evaluation are not outside the networking activity. the network itself and society as a whole. Constitution One aspect of the five characteristics that belong to an approach following a theory of structuring is a procedural understanding of constitution. the organization. The network constitutes itself in time and space via social practices.
and what consequences are intended. that the embedding in institutional contexts and relevant environments Journal of MultiDisciplinary Evaluation: JMDE(2) 43 . or toward summative legitimization. which may e. Co-evolution means that context relevancy cannot be ignored.g. One has to deal conceptually with the question which levels should be included for the generation of knowledge and how. be confronted with a need to legitimize their activities because they receive public funds. When evaluation concepts are dealt with according to sectors. Networks are embedded in specific contexts and environments that play an important role for conditions and cultures of action. Network monitoring and evaluation will be designed. this then includes practice oriented toward planning and resources. according to the respective network culture. Thus we can distinguish sectorspecific evaluation cultures: In the profit field we find a strong orientation toward planning while networks close to the administration. Contextualization The third assumption of a regulatory approach is that the constitution of organizational networks is a coordination of activities in time and space. Co-evolution From a theoretical perspective of structuring—and this is the fourth aspect—the development of organizational networks can be seen as a process of co-evolution with the relevant environment. toward process and correction. Every network will develop its own contextspecific culture and specific social memory (Windeler 2001:325). rely on summative and ex-post evaluation. One has to ask what different goals and goal achievements in a multilevel context are to be analyzed and how multiple goal structures figure in monitoring and evaluation designs.Articles processes. and will have to be designed.
That is why procedures and programs need to be designed that analyze the contributions and potentials of the individual actors. have to be developed reflexively. subgroups. Subcultures. and for sanctioning. for monitoring. Networking in terrains structured by dominance The fifth aspect of an approach according to a theory of structuring is recognition that organizations as collective actors interact competently and powerfully on a terrain structured by dominance (ibid.:30ff). so network monitoring and evaluation are capable of fulfilling a learning and development function for the inner core as well as for its environment. for compliance with these criteria.:251). discursive. and in antagonistic settings it can be contested between stakeholders. but also potential power- Journal of MultiDisciplinary Evaluation: JMDE(2) 44 . the practices and activities as well as the networking context as a whole. Network monitoring and evaluation face the challenge to analyze not only factual dimensions like the management of business activities. subunits will describe and interpret themselves and their respective situations differently. Network membership is very intentional.Articles has to be considered. It remains an open question in each case to what degree the collective actors are able to reproduce their system reflexively and to establish reflexive monitoring. strategically important and available (ibid. System monitoring can establish practices which throw a light on the experiences and expectations of the network partners (Windeler 2001:326) and which co-evolutionarily reconnect the environment to the system’s inner workings. Not only the inner core of the network which is to change. but the participating organizations as well are exposed to change. Evaluation shows a very sensitive relation between contribution and use. General criteria for evaluation and responsibilities for their design.
driven roadblocks in dominance-structured fields of action, e.g. veto and blocking positions, minimal consensus in goal definition, the curtailing of autonomy of network partners, refusal to learn, and the shifting of risks onto third parties (Sydow 1999:298). The evaluative function (ibid.) of network management is supposed to put the whole range of social, factual, and procedural aspects of network management to the test. Sydow bases the relationship between network management and network development on a theory of structuring (2001:82): network development is seen as observed change over time within a social system that is reproduced by relevant practices. Change takes place in a planned way through intervention and also in an unplanned way, through evolution. This perspective relates to the process by which network actors refer to network structures in their actions and attempts at guidance, reconstructing those structures by their actions (ibid.:83). Incorporated in this are structures, ways of development, and the possibilities of trans-organizational development—but also the possibility of failure, of unintended results, of alternative actions, of coincidence. Network development (and the effects and feedback effects it has on the organizations involved) can be described as the result of reflexive as well as non-reflexive structuring (Windeler 2001). To make networks more successful—and this is the procedural and future-oriented function of monitoring and evaluation that will be the center of our attention here—it makes sense to analyze and to design network management as reflexive network development. Monitoring and evaluation then gain central importance for network development: They facilitate the understanding of network development as a field of learning and of the collective development of competence (Weber 2002, 2003, 2004), and they suggest the importance of analyzing empirical networking projects (Weber 2001a).
Journal of MultiDisciplinary Evaluation: JMDE(2) 45
What then should concepts and designs for network evaluation and monitoring look like? This question leads to others, common in evaluation settings, e.g.: What information shall be generated, and how? What knowledge is needed and functional? What function should reflexivity have, what should it achieve? Who should generate knowledge and what should it be used for? If we take the program seriously that was elaborated at the 2003 DeGeVal convention—evaluation should lead to organizational development (Hanft 2003) and the focus should be shifted from summative and ex-post analysis towards process monitoring and future development (Weber 2001b)—then it makes sense to follow the incremental theory of planning. The social-ecological theory of planning and the 1970s’ criticism of classical theories of planning give us criteria that can be used for the reflection and conceptualization of evaluation designs. 3. Selection decisions for the generation of knowledge in networks Uncertainty about the individual actors’ judgement, the comprehension of the original situation, the actors’ collective action, future developments and strategies under a perspective of transformation (Schäffter 2001) can be made productive, if tied to monitoring and evaluation in a view supported by a defensible theory of science. Following an objectivist or constructivist understanding of reality we can distinguish an “objectivist” from a “constructivist” evaluation paradigm. These different understandings will now be considered in their polarity, and afterwards their functionality for monitoring and evaluation in a networking context will be discussed. Their polarity brings monitoring and evaluation into focus as not just instruments, but as networking practices.
Journal of MultiDisciplinary Evaluation: JMDE(2)
While in classical concepts (Rossi/Freeman 1989:18) evaluative approaches were regarded as analytical instruments without the ambition of serving as theoryguided science (Kuhlmann 1998:92), we here consider concepts and approaches to monitoring and evaluation as active practice which is part of and generates specific network cultures. We assume that settings for communicative evaluation are not “just instruments”, without preconditions and “objective”, but that in reality they have a generative quality, organizing observation and knowledge production according to underlying explicit or implicit criteria and models of evaluation. Working on organizational transformation processes, Roehl and Willke have pointed out the—often substantial—“constructedness” of evaluation settings, which is brought about by the choice of instruments and criteria. Evaluation designs are always subject to leading ideas of change, which include ideas about the validity of changes and which, in a context of complex structures of decisionmaking, predetermine the evaluative direction (Roehl/Willke 2001:29). Drawing on cybernetic, social-ecological or systemic criticism of planning in the 1970s and 1980s (Lau 1975, Atteslander 1976) decisions can be identified that become relevant to the selection of evaluation designs. E.g. in the 1970s’ criticism, dimensions of subjectivity, communication, and system orientation are emphasized in the face of a rationalist, technocratic paradigm. This criticism leads to an alternative planning paradigm that includes choices that are relevant for the design of planning and monitoring, such as the following: o Between “technocratic feasibility” and “systemic irritation” o Between legitimization of the past and planning of the future o Between the reproduction of the old and the generative production of the new
Journal of MultiDisciplinary Evaluation: JMDE(2) 47
Lau (1975) pleads for Journal of MultiDisciplinary Evaluation: JMDE(2) 48 . 3. and their chances for “reality construction” can be analyzed. either of which follows from basic questions of a theory of planning. The following presentation of decisions and questions relevant to evaluation in network settings does not pretend to cover all aspects comprehensively. instead it treats them by means of examples. skeptical of a teleological regulative approach to social processes that presupposes predictable results. On the other hand there is the contrary view. In the 1970s. their deficits. in today’s evaluation practice in different social contexts. that social processes can be rationally planned and influenced.1 Evaluation knowledge between “systemic irritation” and “technocratic feasibility” Within the evaluation community there is a tension between two contradictory approaches. Early on. models that take an optimistic view of regulation are increasingly opposed by regulation-skeptical models calling for more open and dynamic approaches to planning and evaluation. This view assumes that even the most advanced and differentiated instruments of planning eventually cannot “handle” social reality.Articles o Between “expert objectivity” and subject participation o Between the completeness of what is known and the processing of what is not known/uncertain o Between result measurement and the development of competence These selective decisions can be found. in different manifestations. and their range. A “technocratic” approach builds on the assumption that existing knowledge can be used to give an intentional design to social conditions (Herrmann 2001:1365).
a technocratic and a cybernetic or systemic type (Atteslander 1976:20). An “evaluation culture” in the sense of a summative evaluation emphasizes the thorough analysis of the past.2 Evaluation knowledge between legitimization of the past and planning of the future Another dimension pertinent for today’s evaluation debate is the directedness towards past. The focus is set on the summative evaluation of individual measures and programs without strong references to organizational visions and goals. Journal of MultiDisciplinary Evaluation: JMDE(2) 49 .Articles management of complexity through a participative concept of planning that retains a sense of flexibility. and the evaluation of goal attainment. and evaluation is rather geared towards a bureaucratic model of control. Here the aim is often legitimization. present or future is today influenced by sectors and organizational cultures. The systemic-constructivist assumption of the self-organization of institutional systems leads to a concept of planning and thus of measuring effectiveness and evaluation which is based rather on “irritation” than on technocratic “feasibility”. The directedness of evaluation designs towards past. and the activities are relatively little strategically synchronized or planning-oriented. to varying degrees. to create legitimacy or change and complex transformation. transparency. the evaluation of previous projects. Evaluation or monitoring designs aim. present or future. Reflexivity is encouraged and facilitated in order to partially produce uncertainty (Herrmann 2001:1365). 3. Atteslander presents a typology of different planning models and defines a dogmatic.
A tendency towards monitoring with self-evaluation classically corresponds with the evaluation concepts preferred by the non-profit sector. motivation instead of control. etc. Possible risks lie in conducting many parallel activities on all levels (supervision. formative evaluation and self-evaluation. “anticipation” was to be employed systematically. Evaluation knowledge between reproduction of the old and generative production of the new There is also a tension in monitoring and evaluation between the reproduction of the old and the generative production of the new. Evaluation designs which are more strongly embedded in a “planning culture” emphasize diagnosis. in this case the network. they do not rely very much on summative evaluation. the generative production of the new was to be facilitated. and conditions for success. 3.) which do not receive feedback from each other. which are not directed toward the organizational or networking goal. The most effective interventions harmonize with visions and strategies of the system of reference. and which see themselves as strategically oriented. numbers and control. Their focus is on future orientation. processorientation.3. Goals connected to monitoring and evaluation are endogenous development. financial aspects of a cost-benefit relation. This tension is already implicit in the demands made during the planning debate of the 1970s: instead of mechanistic models for planning. feasibility studies. The inclusion of prophecies and projections of all kinds Journal of MultiDisciplinary Evaluation: JMDE(2) 50 .Articles A “monitoring culture” on the other hand emphasizes a process-accompanying. and improvement on the level of professional action. The aim is not the realization of individual activities but the strategic feedback relationship of all measures that is supposed to create an equal directedness of all activities. Instead of prognoses of the future based on the status quo.
3. there are distinct risks of interest-guided dominance and colonization on the one hand. “Expert objectivity” or subject participation The fourth decision in evaluation represents the distance between evaluation by experts and by participants. The efficiency of the solution of material problems depends on the participation of those concerned. Evaluation by experts is often oriented at utilitaristicrationalist models of action and leaves responsibility in the hands of the expert. Knowledge production in networks thus has to rely on the cooperative structures of “participatory research” (Atteslander 1976:53). In a heterarchic decision-making structure.Articles in a context of cybernetic models of planning was seen as more adequate to the challenges and demands of planning than dogmatic or technocratic models of planning (Atteslander 1976:53).4.5 Completeness of what is known or processing of what is not known/uncertain This decision is tied to different accentuations—is knowing or not-knowing the point of reference? Open and dynamic models of planning and of monitoring assume a transformable worldview and a comprehensive definition of goals. democratized expertise is a given and the production of knowledge that becomes relevant for action has to work with network knowledge—if it does not. on openness to criticism. The participants tend to become objects of the evaluation. lack of acceptance and inner emigration by network partners on the other. on horizontal structures of interaction and on democratic procedures for implementation. not systemic partners in collective efficiency measurement and evaluation. 3. They do not presuppose knowledge but rather incomplete knowledge or no knowledge Journal of MultiDisciplinary Evaluation: JMDE(2) 51 .
The focus is not on needs presumed to be objective. Monitoring and evaluation are no longer primarily goal-oriented. which shifts away from a purely concept-oriented evaluation of efficiency towards one that also considers (micro.Articles about the current situation and its structures. In an incremental view. Monitoring and evaluation are more than social technology in this case. but on the needs and perspectives of the network actors. This understanding goes hand in hand with an increase in competence and with self-rationalization of the network partners. 3. it is always preliminary and influenced by a large amount of feedback (Atteslander 1976:55). instead the area of work becomes evaluation-oriented. monitoring and evaluation as “technology”. These approaches are synthesizing— they methodically attempt to integrate ideological. an integrated view is suggested. Measurement of efficiency and evaluation can tune in to a daily networking routine that changes slowly. It systematically needs monitoring and evaluation. The rationality and the kind of prognoses connected to cybernetic efficiency measurement and evaluation can be described as an operational rationality working with a combination of deductive and normative prognoses. they are reflexive practice and the creation of communicative contexts where the constitution of social meaning takes center stage. planning can never be final. Contrary to placing planning. an understanding of planning and monitoring as something to be negotiated is directed towards the development of competence. By taking into view the social aspects of the production of Journal of MultiDisciplinary Evaluation: JMDE(2) 52 .6 Result measurement or the development of competence Contrary to a basic view of processes of planning. technological and social aspects of networking contexts.and meso-) political structures. monitoring and evaluation before or after actual practice.
not schematic. 2001:1378). E. These short sketches of considerations based on a theory of planning can be used for the design of instruments and concepts of evaluation and monitoring. Communicative planning practice further documents that such models represent adequate concepts for action within the complex and contradictory conditions and processes in areas of planning (Herrmann 1998. functionality in poly-centric and heterarchic structures. they follow the principle of negotiation. an open model aims at the development of competence. conflicting and Journal of MultiDisciplinary Evaluation: JMDE(2) 53 .7 Consequences for the design of monitoring and evaluation in networking contexts It has become evident that classical evaluative approaches reach their limits in networking contexts. and the internal democratization of expertise (Atteslander 1976).g. and they furnish a pattern for metaevaluation. They do not pretend to be neutral in terms of values but facilitate working through the topic of value. 3. Communicative planning concepts are process-oriented. implicitly also about instruments and procedures. the equivocal connection between ends and means in social contexts. in complex program evaluation it could be shown that the fact has been neglected that programs follow “multiple. in so far as evaluation concepts themselves become objects of evaluation with the help of certain criteria. Communicative approaches are the only planning approaches that attempt to bridge the gap between conceptual planning and practical action by conceptually integrating the problem of the implementation of planning results. about the status of evaluation in networking contexts. the direction and starting points of analysis. They address central questions about the basic assumptions.Articles knowledge that is relevant for implementation.
:98). quasi-evolutionary learning processes as well as orchestrated reflexive interventions can generate learning and reflexivity in a “learning network”. continued in participative approaches to evaluation (Ulrich/Wenzel 2004) and implemented empirically (Uhl/Ulrich/Wenzel 2004). Intended and unintended qualities of learning will find their space here.. an incremental communicative practice of planning and action that is more adequate to the necessities of the field than classical evaluation designs (Zipp 1976:77). Networks are exposed to structural uncertainty about the future. process-analytical and planning practice with a perspective of collective development of competence. Informal.:85). that the views of those who are responsible for the program are taken into consideration but not the interests of those concerned (ibid. as a field of pedagogical rationality (among others) (Helsper/Hörster/Kade 2003). groups of actors. up to the social body as a whole) is contingent and uncertain. As learning from experience it is Journal of MultiDisciplinary Evaluation: JMDE(2) 54 .e. Under a perspective of reflexivity in a networking context. that evaluation is used as a “killer”. and in their intended and unintended reflexive practice. the network structure and its relevant environment. In the context of a multi-layer concept it is neither possible nor sensible to measure “objective results” exactly. A social-ecological planning paradigm becomes manifest. that the context of their conception is often not sufficiently understood. in the systematic form of evaluative. These demands can be tied to the social-ecological approach to evaluation developed by Guba and Lincoln (1989). they can be reconstructed as “learning” networks (Weber 2002). demanding a mainly communicatively oriented validation. in the sense of eternal truths (ibid. communicative validation. i. process monitoring and evaluation become integral parts of network regulation as a design approach.Articles evolving purposes” (Kuhlmann 1998:97). Learning (on the different levels of individual actors.
systemized.Articles intertwined with everyday working activity. If the knowledge-generating practice of making experiences “on the job” gets established. Depending on to which dimensions the reflexive generation of knowledge within the network can be designed around. and structurally put into a feedback relation with the system’s practice. 4. they will be the focus of the following section. Journal of MultiDisciplinary Evaluation: JMDE(2) 55 . Dimensions of evaluative and planning-oriented learning within a network On the basis of empirical networking projects (Weber 2001a) and literature on networking theory we can determine four dimensions for a strategy of collective learning. 2001) and Windeler (2001). personal and social closeness is regarded as a necessary condition for successful networking processes (Winkler 2002:37). While these two aspects have already been objects of network regulation as well as reflexive approaches. the dimension of the social process and that of learning have not yet been considered systematically under a design point of view. then orderly procedures for institutional and network learning are created. so the social dimension is thus structurally included. Weber 2003). In our analysis of network functions and structural tensions we follow the works of Sydow (1999. that of structural tensions created by networking processes and that of learning and learning arrangements (Weber 2002. These design dimensions of system monitoring and evaluation are the social dimension. the dimension of network functions.1 Social regulating and social monitoring The regulatory approach presupposes structuring by social actors. Learning on the subject level and on different system levels also becomes systemized. Cooperative inter-organizational relations are seen as based on social processes. and monitoring and evaluation are put into a context of a development-oriented strategy of collective learning within the network. 4.
As a fourth phase we see the “performing group”. His definitions of Forming. Trust is also seen as a sine qua non for successful projects (Windeler 2001). while the second phase of the process is characterized by struggle for social status and power within the social structure. capable of working and performing. In a third phase the group then has to come together to a functional whole. but so far it has not been addressed in its quality as a group context. and positions in social space have been negotiated. with its individual and collective elements. into the future.Articles Network knowledge is always social. Supported by group-dynamics and team-development approaches. Tuckman extends the group. Theoretical positions based on structuring and complexity describe networks as coevolutionary entities (Kappelhoff 2000b:382) that do not show a linear development.2 Functional network guidance—monitoring of network functions Journal of MultiDisciplinary Evaluation: JMDE(2) 56 . This shows that the social dimension is indeed recognized as relevant. based on stable expectations and a norm of reciprocity. we can here refer to categories of the group process that Tuckman uses to describe different group and team qualities (1965). Tuckman’s model assumes that the first phase of groups’ encounters is friendly and noncommittal. 4. Tuckman’s four-phase approach is useful for its qualitative criteria for the analysis and design of group-dynamical aspects. in turn. Norming and Performing in a social context can be used for monitoring and for evaluation since they provide criteria for the analysis and design of the social context which can be found empirically with the help of indicators. Still. network relationships are based on exchange. it is created by and embedded in social practice. As a whole. which is. Storming.
they do not just develop their relevance along the stages of the process but also across them: selection. “selection” includes the question of “who?”—who shall be included? This question becomes important at an early networking stage. Measures of system integration influence the selection of actors. While Sydow describes them as procedural. constitution of borders—can be objects of evaluation: the selection of the actors belonging to the system. allocation. border management and system integration are necessary and have to be repeated perpetually and circularly.Articles Another dimension of monitoring and evaluation is the functional dimension of network guidance introduced by Sydow (Sydow 1999). the practice of configuration of positions and of the constitution of borders pose particular challenges to potential newcomers etc. the allocation of resources. Windeler adds two others to these four functional aspects: “system integration” and “border management” (Windeler 2001). “Evaluation” of network organizations can concern the network as a whole or just selected rules of cooperation (Sydow 1999:295f). After that the focus shifts to the “allocation” of tasks and resources. allocation. These objects of network regulation are interconnected in a recursive relation. In a design approach. the evaluation of the process and the specifics of system integration. regulation. The six aspects of network guidance are open to analysis and elaboration under the focus of a functional dimension. system integration. evaluation. configuration of positions. (ibid. evaluation. and of the configuration of positions and the constitution of borders (Windeler 2001:249). All elements of network regulation—selection. the distribution of responsibility among the partners. They Journal of MultiDisciplinary Evaluation: JMDE(2) 57 . “Regulation” of cooperation within the network provides the development and implementation of rules between the organizations.:251).
of the conditions and effects of actions and of the actions of others (Sydow 2001:90). at the quality of the network relations that have been developed or at the “network effect”. Messner. They furnish the informational basis for a (more) reflexive network development by network management. “reflexive monitoring” is designed as a tool for supervision of one’s own actions. They provide analytical potential and differentiating criteria for the evaluation and design of network cooperations. coming from political science. All in all it becomes evident that network monitoring and evaluation have to be integral parts of a complexity-oriented reconstruction of networking processes. has also identified structural dilemmas of Journal of MultiDisciplinary Evaluation: JMDE(2) 58 . In Sydow’s approach to network functions (1999:298) monitoring and evaluation have systematic value. criteria and indicators for network monitoring and evaluation along the emergence of design necessities. 4. monitoring and evaluation facilitate the systematic regulation of networking risks and the increase of networking success. While “evaluation” aims at the contributions of individual network organizations. Sydow has introduced eight lines of tension that have to be regulated in networking processes—or if lacking regulation can cause a networking process to fail (Sydow 1999). Sydow assumes that monitoring and evaluation become important factors in the design of paths of development within reflexive network development. The characteristics of reflexive network regulation provide a concrete basis to the function and design of monitoring and evaluation. From a design perspective. and while as a function of management it is concerned with the practice of evaluating.Articles offer a catalogue of questions.3 Structural tension—monitoring of tension A third focus of complexity-oriented network monitoring has to be the dimension of structural tension.
The following section is based on Sydow’s presentation (1999) of the lines of tension between “autonomy and dependency”. what is regulated by control mechanisms. “formality and informality”. Variety—unity: How can a balance be reached between the variety of participating actors and their integration to some kind of unity? Flexibility—specificity: How flexible is the network in terms of its goals and selfimage. 1994). change).g. “cooperation and competition”. “economic rationality and preservation of power” (Sydow 1999:300).Articles networking that have to be worked on within networking processes (Messner 1995. how specific is it? Autonomy—dependency: How much autonomy is possible and what does it consist of. “trust and control”. what relationship do they have? Economy—power: What relationship is there between arrangements of functionality and power? How are power patterns generated? Journal of MultiDisciplinary Evaluation: JMDE(2) 59 . and how? Cooperation—competition: What role do cooperation and competition play? What relationship is created between them? Stability—fragility: What role do stability and fragility play? How are they created? What regulating mechanisms exist? Formality—informality: How is the relationship between formality and informality regulated. “stability and fragility” (e. “flexibility and specificity”. “variety and unity”. how much dependency exists and what does it consist of? Trust—control: How much trust and what kind of trust is there.
re-entering into the circle of active planning within the network. communication and system reflexivity: networking as a learning process Since networks represent dynamic rather than static arrangements of relations and cooperations. networking has to be read as a learning process. 4.Articles Windeler (2001) also refers to these lines of tension in his approach based on a theory of structuring. aspects of a learning biography and the estimation of one’s own competence can also be used for a kind of monitoring that is oriented to competence development. They are tied to system reflexivity and communication. In network contexts as informal learning contexts. thus for clarification and localization within the discursive context and the network’s path of development. Monitoring and evaluation have the function to generate knowledge from practical experience and to reflect on it. So their primary objective is to provide chances for learning and optimization on the system level. Within a monitoring approach they can be regarded as analytical dimensions and as design parameters. the functions of network guidance. in order to deduce knowledge from it that may guide future actions (Uhl/Ulrich/Wenzel 2004:11). Discursive reflection produces awareness of change in the first place—data gathering procedures not only reconstruct their subject in different ways. the structural lines of tension. the subject of reflection itself is changed by it (Hendrich 2003:157). They are useful for the incorporation of reflexivity in discursive and qualitative processes of analysis. and the dimension of learning within the networking process. The social dimension.4 Knowledge. have been Journal of MultiDisciplinary Evaluation: JMDE(2) 60 . The explicit directedness of monitoring towards the design of learning contexts makes it possible to identify future-oriented developmental potentials of networking projects.
Articles suggested as dimensions for the monitoring of efficiency and for the evaluation of complex transformations (Weber 2003).g. the focus may not be narrowed to a few efficiency indicators since this includes the risk of distortions. A perspective: instruments for evaluative and planning-oriented network development As the criticism of under-complex evaluation designs has shown. e. functional. procedural self-evaluation. In this way they become accessible to process evaluation and optimization. they are part of the structural characteristics of networking and have to be dealt with productively. It will be part of everyday action and has to be functional in terms of the necessities that come with this. Sydow thinks that a continuous employment Journal of MultiDisciplinary Evaluation: JMDE(2) 61 . Under-complex designs for monitoring and evaluation have counterproductive effects when the truth production of the system generates faulty attributions and labeling or unintended effects. in the sense of social dynamics. On this level networking needs “cooperative core competence” to balance existing tensions. What instruments and learning arrangements can support complexity-oriented monitoring and evaluation which reply to demands on the social. coordinators and moderators. Especially quantified data is often endowed with a status of objectivity that makes it difficult to question the results. These tensions cannot be dissolved. structural and learning dimensions in a pragmatic and manageable fashion? 5. brokers. In complex social systems reflexive network monitoring will not exclusively be left to process counselors. Below the level of external evaluation by experts it is recommended that there be developed a discursive. This means that monitoring and evaluation in a network have to be geared towards communication and complex reconstruction.
and identity-orientation as well as an orientation toward a multi-layered approach. Network monitoring and evaluation which are geared to future-oriented learning and collective development of competence will be designed in a rather decentralized. Weber 2003. Depending on a given context of economic sectors or institutions. Monitoring and networking in networks are instruments for the construction of reality. or self-evaluation or ex-post evaluation by experts can be seen as practical. Guba and Lincoln (1989) suggested an approach of “stakeholder-based evaluation”. in publication. Evaluative learning arrangements combine qualitative and quantitative methods. one that is participant-oriented and allows the collective definition of criteria and indicators of successful cooperation. future. procedural and temporal. Weber/Benthin. For an integrated design of monitoring and planning it seems to be practical to generate open evaluation designs (Lynen von Berg/Hirseland 2004:15). To deal adequately with complex relations of cause and effect they should be represented in a complex fashion (Bangel 1999:354). wrapped up in a heterarchic and polyvalent structure of interests and in complex transformation processes. Participatory effect monitoring Journal of MultiDisciplinary Evaluation: JMDE(2) 62 . although the employment of quantitative methods is not excluded. These designs should take the form of participative evaluation (Oels 2003. and can thus fulfill the different demands of a networking context. Design criteria for network evaluation should be a multitude of perspectives.Articles and practice of reflexive monitoring can render more formal evaluative methods unnecessary (Sydow 2001:97). in publication) which should be multi-layered. instruments of quality management can be employed. process-. methods that generate knowledge and those that “measure” success in a methodical mix. dynamic and open fashion.
and if needed legitimization as well (Ulrich/Wenzel 2004:28). planning arrangement of network learning.Articles follows a central evaluative objective of increasing the collective capability to act. monitoring and evaluation functions and in this way fulfill the evaluative functions of understanding. in publication). and of doing things differently and possibly better than in the past (Oels 2003). The approach to evaluation. legitimization and optimization. the social dimension in networks can be analyzed with the help of indicators: the Balanced Scorecard is an instrument for the analysis of network functions as objects of evaluation. functional. dialogical settings. the objects of evaluation can be regarded as dimensions of social. Its goal is to expand the repertoire of action (Benthin/Baumert 2001). of breaking out of old ruts. contexts and procedures of complex (self-) evaluation are created that simultaneously cover the functions of understanding and optimization. instruments of network management can be put to use which can take over planning. monitoring and planning described as “stakeholder-based evaluation“ is based on a constructivist paradigm and aims at addressing a large variety of perspectives— which can also be contradictory—in order to create a complex picture of the whole.g. Structural tensions can be analyzed e. Especially in open. while the dialogical arrangements of Large Group Interventions can provide an evaluative. Journal of MultiDisciplinary Evaluation: JMDE(2) 63 . Indicators and criteria for monitoring and evaluation are generated interactively with the actors concerned. Special emphasis is placed on the definition of learning objectives. structural and learning evaluation. In this way. to increase autonomy and to minimize the degree of manipulation and passivity (Oels. On the basis of a participant-oriented approach. with an appreciative evaluation approach. For example.
10-71. in publication).).Articles Large Group Interventions provide strategic agility and risk minimization in fast transformation processes with a high degree of network activity. open approaches like Large Group Interventions make it possible to regulate the network tensions brought about by system monitoring and evaluation.: De Gruyter. They create a mode of “pedagogical organizing”. chain reactions and “reflexively” influenced causal connections. So participant-oriented effect monitoring in networks will always have to try and strike a balance with that which is not known (Kade 2003). Atteslander (Hrsg. (1976): Sozialwissenschaftliche Aspekte von Raumordnung und Raumplanung. Journal of MultiDisciplinary Evaluation: JMDE(2) 64 . Soziologie und Raumplanung. spiral-shaped model of evaluation.a. For this reason it will escape the myth of technocratic feasibility – and embark on a journey of collective procedural learning. in which unrecognized conditions and unintended results of actions emerge as well as “blind spots”. pp. P. with its quality of experimental practice (Weber 2004. As procedures of transformation they follow the systemic paradigm (Bunker/Alban 1997:5). Berlin u. working with the iterative practice of producing systemic rationality. A practice that is oriented towards reflexivity and knowledge generation closes the circle of knowledge provided by monitoring. In P. because they make use of collective intelligence (Königswieser/Keil 2000). References Atteslander. But it will never produce “complete” results and will always have rational and irrational parts (Windeler 2001:220). This practice of system reflexivity produces a discursive arrangement of ulterior and self-guidance in which a lot escapes the grip of reflexivity. evaluation and planning in the sense of an incremental. Systemic.
Verlag. Dahme & N. 37-54. Theoretische und empirische Implikationen. 1-7. Manuel (2000): Elemente einer Theorie der Netzwerkgesellschaft. Brödel. (Hrsg. Die Brandenburger Konzeption der adressatenorientierten Evaluation. In. Wohlfahrt (Hrsg. Grunow. 303-336. Engaging the Whole System for Rapid Change. Anke. In: WEBER. Heft 3/2004. Nicole. Heft Nr.-J. 2/2000. Bremer. Helmut. Pp.und Vermittlungsdienste der Bundesanstalt für Arbeit. Hagemann. Alban. (2001): Selbstevaluation als Methode der Qualitätsentwicklung.). 41. 45. Bettina (1999): Evaluierung der Arbeitsmarktpolitik aus Ländersicht. Billie. Netzwerkökonomie im Wohlfahrtsstaat. Duschek. Gnahs. In: Sozialwissenschaftliche Literaturrundschau. Markus (2004): Netzwerktypologien im Anwendungsbereich Kompetenzentwicklung.Articles Bangel. Benthin. Waxmann. Erfahrungen mit institutioneller Vernetzung im ländlichen Raum. pp. Chollet. In H. Berlin. Prozesssteuerung und summativen Evaluation. In: Informationen für die Beratungs. pp. BAUMERT. pp. Rometsch. Bunker. pp. Münster / New York / München / Berlin. Castells. Wettbewerb im Sozial. (1997): Large Group Interventions. 261-280.und Gesundheitssektor. 3745-3754. Rainer. Berlin: Edition Sigma.): Netzwerkentwicklung in der Jugendberufshilfe. Barbara Benedict. Dieter (2000): Netzwerkanalyse. Dieter (2003): Indikatoren. Martina. T. Nr. Stephan. 92-106. In: QUEM Bulletin Berufliche Kompetenzentwicklung.und Messprobleme bei der Bestimmung der Lernhaltigkeit von Regionen. Jossey Bass.): Begleitforschung in Lernkulturen. pp. Opladen. San Francisco. Ina-Marie (Hrsg. Journal of MultiDisciplinary Evaluation: JMDE(2) 65 . S.
Jochen (2003): Pädagogische Felder im Modernisierungsprozess. Hanft. Hendrich. Hörster. Ina-Marie (Hrsg.11. Anke. Rainer. (1989): Fourth Generation Evaluation. W.V. In: Otto. Hagemann. Kriftel. Karin. Konflikten und begrenzter Rationalität. Edition Sigma.his. Egon G. Sozialpädagogik. Helmut.. Werner. F. http://evanet. Münster / New York / München / Berlin. Neuwied.149-161. Chollet. Umgang mit Widersprüchen. Franz (1998): Jugendhilfeplanung als Balanceakt. Regionale Innovationsprozesse zwischen Kontinuität und Wandel. Neuwied.2003 in Hamburg. Verlag. H.de Hellmer.. Wolfgang (2003): Ansätze interaktiver Evaluationsmethoden in der beruflichen Weiterbildung am Beispiel informell erworbener Kompetenzen. (1999): Mythos Netzwerke.. Reinhard.): Begleitforschung in Lernkulturen. Jahrestagung der Deutschen Gesellschaft für Evaluation e. Katrin am (2003): Beispiel Erfahrungen der und Probleme mit Handlungsforschung wisenschaftlichen Begleitung Journal of MultiDisciplinary Evaluation: JMDE(2) 66 . Weilerswist. Krumbein. Yvonna S. Wöllert. Hans (2001): Handbuch der Sozialarbeit. Friese. (DeGeVal).-10. Helsper.. Kade. Thiersch. Anke (2003): Evaluation und Organisationsentwicklung. Lincoln. In: Brödel. 1375-1382. Chr. Hans-Uwe. Kollros. Berlin. Auflage. Bremer. pp. Waxmann. 8.Articles Guba. Eröffnungsvortrag zur 6. Jutzi. London. EvaNet-Positionen 10/2003. Herrmann. Franz (2001): Planungstheorie. pp. Herrmann. 2.
Marion (Hrsg. Kuhlmann. In: Gogolin. Rainer. pp. Opladen: Westdeutscher Verlag.Articles intermediärer Agenturen. Praxisbeispiele für Großgruppenveranstaltungen. Tippelt. Sydow & A. Baden Baden. Kappelhoff.html (am Journal of MultiDisciplinary Evaluation: JMDE(2) 67 . Münster / New York / München / Berlin. Chollet.Umgang mit Wissen – Nichtwissen. 2557. Peter (2000b): Komplexitätstheorie und die Steuerung von Netzwerken. pp. Opladen: Westdeutscher Verlag. In: Brödel.deming. pp.2003) Königswieser.). Roswita. Windeler (Hrsg. Peter (2000a): Der Netzwerkansatz als konzeptueller Rahmen für eine Theorie interorganisationaler Netzwerke. 1. Hagemann.de/efqm/balanscore-1. Steuerung von Netzwerken. Rudolf (Hrsg. Ingrid. Über die Zukunft pädagogischer Kommunikation.). Verlag.): Begleitforschung in Lernkulturen.10. pp. Evaluationsverfahren in der Forschungs. Windeler (Hrsg. Bremer. Stefan (1998): Politikmoderation. Helmut.): Innovation durch Bildung. Konzepte.J. Jochen (2003): Wissen . Anke. 347-389. (o. Kirstein. Ina-Marie (Hrsg. NomosVerlag. Keil. Waxmann.und Technologiepolitik. In: J. Kade. H.): Was ich schon immer über die Balanced Scorecard wissen wollte. Designs. 89-108. Beiträge zu 18. Opladen. In J. Kappelhoff. Unter: http://www.) (2000): Das Feuer großer Gruppen. 129-143. Steuerung von Netzwerken. Sydow & A. Stuttgart: Klett-Cotta. Kongress der Deutschen Gesellschaft für Erziehungswissenschaft.
Lau, Christoph (1975): Theorien gesellschaftlicher Planung. Eine Einführung. Stuttgart, u.a. Lynen von Berg, Heinz; Hirseland, Andreas (2004): Zivilgesellschaft und politische Bildung – Zur Evaluation von Programmen und Projekten. In: Uhl, Katrin; Ulrich, Susanne; Wenzel, Florian M. (Hrsg.): Evaluation politischer Bildung. Ist Wirkung messbar? Verlag Bertelsmann Stiftung. Gütersloh. pp. 15-26. Messner, Dirk (1995): Die Netzwerkgesellschaft. Wirtschaftliche Entwicklung und internationale Wettbewerbsfähigkeit als Probleme gesellschaftlicher Steuerung. Köln: Weltforum-Verlag. Messner, Dirk (1994): Fallstricke und Grenzen der Netzwerksteuerung. In: PROKLA-Zeitschrift für kritische Sozialwissenschaft. Nr. 97.pp. 563-596. Oels, Angela (2000): “Let’s get together and feel alright!” Eine kritische Untersuchung von „Agenda 21“-Prozessen in England und Deutschland. In H. Heinelt & E. Mühlich (Hrsg.), Lokale Agenda 21-Prozesse. Erklärungsansätze, Konzepte und Ergebnisse. Reihe ‚Städte und Regionen in Europa’ Band 7, Opladen: Leske + Budrich. pp. S.182-200. Oels, Angela (i.V.a): Großgruppenevaluation in Netzwerken. In S. Weber (Hrsg.), Netzwerklernen. Methoden, Instrumente. Erfolgsfaktoren. Oels, Angela (2003): The Power of Visioning. An evaluation of community-based Future Search Conferences in England and Germany. Münster. Schäffter, Ortfried (2001): Weiterbildung in der Transformationsgesellschaft. Zur Grundlegung einer Theorie der Institutionalisierung. Hohengehren.
Journal of MultiDisciplinary Evaluation: JMDE(2)
Simmel, Georg (1908): Soziologie. Leipzig: Duncker & Humblot. Steger, Renate (2003): Netzwerkentwicklung im professionellen Bereich dargestellt am Modellprojekt REGINE und dem Beraternetzwerk zetTeam. Materialien aus dem Institut für empirische Soziologie an der FriedrichAlexander-Universität Erlangen-Nürnberg. 6/2003. IfeS. ISSN 1618-6540 (Internet). www://ifes.uni-erlangen.de. Sydow, Jörg (1999): Management von Netzwerkorganisationen. Zum Stand der Forschung. In J. Sydow (Hrsg.), Management von Netzwerkorganisationen. Wiesbaden: Gabler. pp. 279-305. Sydow, Jörg (2001): Management von Unternehmungsnetzwerken – Auf dem Weg zu einer reflexiven Netzwerkentwicklung? In: Howaldt, Jürgen; Kopp, Ralf; Flocken, Peter (Hrsg.): Kooperationsverbünde und regionale Modernisierung. Theorie und Praxis der Netzwerkarbeit. Gabler Verlag. Wiesbaden. pp. 79-102. Tuckman, B.W. (1965): Developmental Sequences in Small Groups. Psychological Bulletin 63, pp. 384-399. Uhl, Katrin; Ulrich, Susanne; Wenzel, Florian W. (2004): Einleitung: Evaluation und politische Bildung – was kann man “messen“? In. Uhl, Katrin; Ulrich, Susanne; Wenzel, Florian M. (Hrsg.): Evaluation politischer Bildung. Ist Wirkung messbar? Verlag Bertelsmann Stiftung. Gütersloh. pp. 9-12. Ulrich, Susanne; Wenzel, Florian M. (2004): Partizipative Evaluation. In: Uhl, Katrin; Ulrich, Susanne; Wenzel, Florian M. (Hrsg.): Evaluation politischer Bildung. Ist Wirkung messbar? Verlag Bertelsmann Stiftung. Gütersloh. pp. 27-48.
Journal of MultiDisciplinary Evaluation: JMDE(2) 69
Weber, Susanne (i.V.) (Hrsg.): Netzwerklernen. Instrumente, Methoden, Erfolgsfaktoren. Weber, Susanne (2001a). Netzwerkentwicklung in der Jugendberufshilfe. Erfahrungen mit institutioneller Vernetzung im ländlichen Raum. Opladen: Leske + Budrich. Weber, Susanne (2001b): Vom „Ist“ zum „Soll“: Partizipative Verfahren und neue Planungsrationalität. In: Keiner, Edwin (Hrsg.): Evaluation (in) der Erziehungswissenschaft. Weinheim und Basel. pp. 255-272. Weber, Susanne (2002): Vernetzungsprozesse gestalten. Erfahrungen aus der Beraterpraxis mit Großgruppen und Organisationen. Wiesbaden: Gabler. Weber, Susanne (2003): Zur Evaluation von Großgruppenverfahren am Beispiel regionaler Vernetzung. In: Dewe, Bernd; Wiesner, Gisela; Wittpoth, Jürgen (2003): REPORT. Literaturund Forschungsbericht Weiterbildung. Erwachsenenbildung und Demokratie. Dokumentation der Jahrestagung 2002 der DGfE Sektion Erwachsenenbildung. pp. 110-119. Weber, Susanne (2004a): Transformation und Improvisation. Universität
Großgruppenverfahren als Technologien des Lernens im Umgewissen. Unveröffentlichte Marburg. Weber, Susanne (2004b): Organisationsnetzwerke In: W. Böttcher & E. und pädagogische (Hrsg.), Habilitationsschrift. Philipps-Universität
Organisationstheorie. Ihr Potential für die Analyse und Entwicklung von pädagogischen Feldern. pp. 253-269.
Journal of MultiDisciplinary Evaluation: JMDE(2)
Berlin. In: Atteslander. kapitalistischer Produktion. Wolf. Benthin. Windeler.): Evaluation politischer Bildung. Florian M. Methoden. Koordination und Kompetenzen. Winkler. Matthias. Susanne. Winkler.): Soziologie und Raumplanung. Florian M. (Hrsg.Articles Weber.V. In: Freitag. Erfolgsfaktoren. Gisela (1976): Planungsziele und Planungswirklichkeit. 177-196. pp.b): Innovation. In: Uhl. 72-93.Westfälisches Journal of MultiDisciplinary Evaluation: JMDE(2) 71 . Nicole (i. Zipp. Arnold (2001): Unternehmensnetzwerke. Ingo (2002): Steuerung zwischenbetrieblicher Netzwerke – Koordinations. pp. Peter (Hrsg. Wenzel.und Integrationsmechanismen.): Kooperationsentwicklung in zwischenbetrieblichen Netzwerken. Münster . Instrumente. pp. Würzburg. Susanne. Katrin. Susanne: Netzwerklernen. Ein Versuch über Widersprüche und Metamorphosen Dampfboot. Strukturierung. Konstitution und Strukturation. Wiesbaden: Westdeutscher Verlag. (2004): Selbstevaluation wertschätzend gestalten – methodisches Vorgehen in 6 Schritten. Gütersloh. In: Weber. Ingo (Hrsg. Ulrich. Boston. Harald (1999): Arbeit und Autonomie. Ist Wirkung messbar? Verlag Bertelsmann Stiftung. 31-55. Wissen. Wenzel. Selbstreflexivität im Netzwerk generieren.
christian. a request for proposals was issued for an adult drug treatment program. S. The Evaluation Center. In extreme cases. Allowing intellectual property to become the property of the entity requesting the proposal directly influences the evaluator’s work and raises several significant ethical issues regarding the contractual statements found in most requests for proposals which give funding agencies property rights to all information and materials submitted to them.000. Schröter.Practical Ethics for Program Evaluation Client Impropriety Chris L. it was never the intention to fund these submissions. Daniela C. MI 49008. and so on. e-mail: University. The RFP was of the usual sort. design. and Pamela A. S. knowing full well that potential funding entities have the legal right to implement these ideas without the submitter’s approval.coryn@wmich. Zeller Requests for proposals (RFPs) often include statements transferring ownership of the content of proposals to the requestor.edu. only to use their ideas. budget. Western Michigan 1903 West Michigan Avenue. that is. the second a poorly designed effort 41 Corresponding author: Chris L. Journal of MultiDisciplinary Evaluation: JMDE(2) 72 . Recently. funding entities have even requested proposals for the purpose of idea-generation only. Thus. Coryn. a well planned. This kind of ethical abuse is neither new nor unique to evaluation. Take the following case. expertise and experience. Kalamazoo. evaluators are frequently faced with the problem of responding to a RFP in an unprotected manner. for example. Two proposals were ultimately selected as the final candidates: the first. systematic evaluation with a proposed budget of just under $100. Coryn41.
000. The client may save as a result of funding the low-cost evaluator if the evaluator is able to implement and fulfill the contract as proposed by the high-cost competitor. The following questions arise as a result of the client's decision: 1. that the contracted low-cost competitor has neither the means nor the competencies necessary to effectively implement the competitor’s plan and design. Ultimately. Does the low-cost competitor have the expertise and competency to implement the costlier competitor’s plan and design? It may be reasonable to infer. Moreover. As such. the second proposal neither included expenses for travel nor the indirect costs associated with university-based research units. 2. The funding entity decided that $10. but what the evaluand saves monetarily may be lost in validity and credibility. However.Practical Ethics for Program Evaluation budgeted at slightly more than $10. the low-cost competitor was funded. the independent consultant indicated within his proposal that all work would be conducted by his students as part of a class project and that these students would not be reimbursed for their work. the issues surrounding the contractual statements that allow all submitted Journal of MultiDisciplinary Evaluation: JMDE(2) 73 .000. in some case.000 was more attractive than $100. local to the city within which the program was based. Can the costlier competitor’s plan and design be comparably implemented by the low-cost competitor at 1/10 of the price? Perhaps costs can be cut dramatically by hiring a local evaluator with access to free labor and university resources. Why the substantial budget differences? The first proposal was submitted by a university-based evaluation unit and the second was submitted by a university professor acting as an independent consultant. but under the premise of utilizing the costlier competitor’s plan and design.
This could be accomplished in several ways: (i) a fee could be provided for the use of plans and designs. Given the current climate of the competitive evaluation market. or (iii) the evaluator could be contracted as a metaevaluator. Second. the professor discussed in the case example is more than likely not a member of any organized evaluation organization and therefore not accountable to professional standards of conduct.Practical Ethics for Program Evaluation materials to become the property of the funding agency are indeed troubling. proposal writers are faced with several poignant questions: How detailed and precise should evaluation proposals be if they become the intellectual property of the entity requesting them? Should the funding entity return rejected proposals? How can we as evaluators protect our intellectual property given that funders have the rights to use all proposals they receive? If the funding entity uses any or all of the rejected proposals. First. yet he had obviously violated the unwritten standards of conduct expected of a researcher by accepting the contract and using another’s work without the consent of the proposal writer. In addition to the impropriety of the client/funder. it should—for the sake of integrity—compensate the originator. what can be done to alleviate these problems in the future? Some writers of proposals have attempted to take matters into their own hands by explicitly indicating in their proposals that no portion of the submission may be used without their express Journal of MultiDisciplinary Evaluation: JMDE(2) 74 . (ii) the evaluator could collaborate for a consulting fee to help execute the evaluation. other relevant ethical concerns are raised. in full or part. The aforementioned example of client/funder impropriety is utterly unacceptable and the repercussions for the evaluation profession are profound.
It has been suggested here at the Evaluation Center that approaching AEA might be appropriate. and perhaps some defensive strategies such as blacklisting abusers. Yet if potential clients/funders willingly and knowingly use these materials. the implications of an epidemic of this kind of client behavior are frightening. We might suggest developing a code of conduct for evaluation clients. What do you think? Journal of MultiDisciplinary Evaluation: JMDE(2) 75 .Practical Ethics for Program Evaluation consent. unbeknownst to the proposal writer. what can be done? As can be seen.
How can evaluators through unspoken messages impact stakeholders? A gaze broken too soon. & Scriven. evaluators should strive to make evaluates less fearful of the evaluation process. Gooler. getting information from participants should be more fruitful. Thus. or who attack the evaluation without regards to how well conceived it might be” (ibid). Journal of MultiDisciplinary Evaluation: JMDE(2) 76 . if XEA can be quelled. defining our interpersonal relationships. the likelihood of implementing recommendations should be increased if impactees of the evaluation are comfortable with the evaluation process. 2004). a forced smile. 261). Donaldson. (2002) use the acronym XEA to describe excessive anxiety and explain that “…there are people who are very upset by. The use of communication research may be a unique approach to relieving XEA. a conversation conducted across the barrier of an executive desk—together such nonverbal strands form the fabric of our communicative world. First. p. A common technique or ‘magic bullet’ to prevent excessive anxiety would not exist in program evaluation. et al. 2002. October 26. Scriven stated that evaluators should care about excessive evaluation anxiety for two reasons. In his EVAL 600 class (Western Michigan University. a flat voice. and sometimes rendered virtually dysfunctional by.Ideas to Consider Managing Extreme Evaluation Anxiety Though Nonverbal Communication Regina Switalski Schinker “Many evaluative situations cause people to fear that they will be found to be deficient or inadequate by others…” (Donaldson. any prospect of evaluation. Secondly. an unreturned phone call. one aspect being nonverbal communication.
Coker. he/she should mimic the dress of those being Journal of MultiDisciplinary Evaluation: JMDE(2) 77 . Alternately. voice pitch and speed. and facial expression. governing the flow of our social encounters. Fatt. Nolen. such gestures as negative head nodding and foot movement in space signify a tense environment (Keiser & Altman. Self. 346). If an evaluator greets a stakeholder with a calm. shifting eyes. Averting eyes. 1995) all convey comfort and confidence. At times. Credibility. a stakeholder will make an impression regarding the professional evaluator. Gestures. In an instant. include such factors as volume. Nolen (1995) states. they are conveying confidence and believability. Paralanguage. This impression will be based on a number of nonverbal cues. dress. and reinforcing our attempts to influence others (Ebesu & Burgoon. revealing our emotions. or vocal cues. 1986. 1976). open arms. & Coker. consistent gaze. vocal cues are more important than words (Nolen. casually crossed legs. and blinking excessively all signal untrustworthiness. and expertise are often determined through nonverbal communication channels (Ebesu & Burgoon. posture. 1995). 1996). 1999. 1995). 1996. 1999). Eye contact. Gestures like smiling with head nodding. 1996. Appearance.Ideas to Consider declaring our personal identities. and leaning towards the person of focus (Nolen. For example. p. trustworthiness. insecurity. Paralanguage. eye contact. pitch and pronunciation (Fatt. rate. looking at notes for an extended period of time. if the evaluator wants to imply receptiveness of others. that the objective of communication may determine the choice of clothing. and/or lack of credibility (Burgoon.
or face-to-face. communication leads often the most fruitful communication (Burgoon. 1999). 2001) they must approach their stakeholders with friendliness.Ideas to Consider evaluated. However. While nonverbal cues occur almost automatically (Palmer & Simmons. If evaluators are to reduce anxiety and gain a helpful reputation (Donaldson. & deTurck. If the evaluator wishes to promote an image of status and expertise (ibid). An environmental factor such as seating arrangement says a lot about the evaluator and their intentions. p. 1995). 1995). It is through an evaluator’s communicative style that the image of evaluation will be formed. For a more in-depth understanding. Hale. sociability. it would be wise and worthwhile for every evaluator to take a graduate level course in nonverbal communication to better understand the person and attitude they are portraying and how their communication cues may affect XEA. and ease. 350). 1984. “A person expecting to exercise leadership typically sits at the head of a table…” (Ebesu & Burgoon. “Similarity implies receptiveness” (Nolen. close proximity. Journal of MultiDisciplinary Evaluation: JMDE(2) 78 . My vision for evaluation concerns communicative style. the evaluator can use the above nonverbal cues to exhibit dominance or cultural similarity. Fatt. I would like to see evaluators become conscious of the nonverbal messages they are sending to stakeholders and peers. they should dress more formally than those they are evaluating. Buller. we must try to be cognizant of them. Environmental factors. 1996. In summary. closedness or openness.
com/p/articles/mi_m4422/is_6_16/ai_55580031/print Nolen. Nonverbal Communication. 2004 from http://findarticles. S. Internal Auditor.nonverbal communication.. & Burgoon. B. J. W. E. (1986). Overcoming our negative reputation: Evaluation becomes known as a helping profession [Electronic version]. In M.). An integrated approach to communication theory and research (pp. Donaldson. D.nonverbal communication in internal auditing. D. A. 2004 from http://findarticles. A. Gooler. 12 (4. Salwen & D. 495-524. W. A. J. (1999.I. American Journal of Evaluation. P..I. (2001). Ebesu. It’s not what you say. American Journal of Evaluation. M. S. 355-361. R. Strategies for managing evaluation anxiety: Toward a psychology of program evaluation [Electronic version]. 261-272. S. Donaldson. Human Communication Research. (2002). 345-358).. K. B. (1984)... Hale. Burgoon.Ideas to Consider References Burgoon. T. M. K. Retrieved November 9. 22..E. 351-378. June 1).com/p/articles/mi_m4153/is_n2_v52/ai_17003168/print Journal of MultiDisciplinary Evaluation: JMDE(2) 79 . (1996). Coker. Relational messages associated with nonverbal behaviors [Electronic version]. A. p. & deTurck.. p. K.. Spring). Summer). Fatt. Human Communication Research. Jerold L. J. April 1) Reading people . Retrieved November 9. & Scriven. J. L. NJ: Lawrence Erlbaum Associates. Stacks (Eds. 23(3). Mahwah. & Coker. Buller. 10 (3. it’s how you say it . Communicative effects of gaze behavior: A test of two contrasting explanations [Electronic version]. Communication World. (1995.
M. (1995). B. Human Communication Research. B. September). J. Conscious and nonconscious encoding of liking [Electronic version].Ideas to Consider Keiser. 128-160. 147-161.). Relationship of nonverbal behavior to the social penetration process [Electronic version]. G. Self. NJ: Lawrence Erlbaum Associates. An integrated approach to communication theory and research (pp. C. Communicating intentions through nonverbal behaviors. Palmer. & Simmons. T. Salwen & D. 345-358). W. In M. 22 (1. & Altman. 2 (2. Stacks (Eds. (1976). Mahwah.. K. Winter). C. (1996) Credibility. I. Human Communication Research. Journal of MultiDisciplinary Evaluation: JMDE(2) 80 .
the use of cost analysis is still infrequent. Informed decisions require information on both costs and effects. cost analysis can play an important role in national planning. (2) political or moral controversies in assigning values to input/outcome measures (e. Andrieu. (6) inability to quantify all costs and benefits. 2004. Given that the ultimate societal goal is to optimize the use of scarce resources. policy makers and society at large. Rossi et al. Fitzpatrick et al. The current underutilization of cost analysis should seriously concern evaluators. These include: (1) unfamiliarity with the necessary analytical procedures. (8) lack of data. Berk & Rossi.Ideas to Consider Is Cost Analysis Underutilized in Decision Making? Nadini Persaud Is cost analysis underutilized in decision making? Research suggests it is. Further. 1990. (5) determining when benefits and costs occur. (4) determining who incurs the benefits and costs. and (10) difficulties with separating program developmental costs from operating costs (Alkin & Solomon. 2004. The question is “Can anything be done to raise awareness on this issue?” Yes! Leading Journal of MultiDisciplinary Evaluation: JMDE(2) 81 . 1983. (3) determining the extent to which benefits identified and quantified have been caused by the program)..g. Sewell & Marczak. (7) lack of resources to conduct long-term follow up studies. (9) data in a form incomprehensible to the evaluator. 2001). it is often poorly done because many evaluators lack the necessary technical skills (Levin & McEwan. where cost analysis is conducted. determining the appropriate discount rate).. Some reasons for the underutilization of cost analysis center on difficulties associated with its use. 1977. According to several authors. 1997).
CA: Sage.. A. Thinking about program evaluation. W.) Thousand Oaks. B. Evaluation research methods: A basic guide. H. & McEwan. (1983).). P. P.arizona. J..Ideas to Consider evaluation textbooks and journals must take a more active role in promoting cost analysis. Beverly Hill. M. Andrieu. Fitzpatrick. R. J.Y: Longman Publishers. CA: Sage. (2nd Ed. L. M. 2004. graduate programs and certificate programs in evaluation need to incorporate cost analysis in their course requirements. The costs of evaluation. 217–232.. H. E. (2004). (1977). CA: Sage. (2004). Thousand Oaks. M. P.). CA: Sage. C.. (7th Ed). Berk. (2001). Program evaluation: Alternative approaches and practical guidelines. Benefit-cost evaluation. Sewell. & Freeman. they will never be confident they are conducting cost analysis competently References Alkin. Cost-effectiveness analysis: Methods and applications. Rossi. Journal of MultiDisciplinary Evaluation: JMDE(2) 82 . In L. R. from http://ag. & Rossi. Tucson. R. C. J. M. M. CA: Sage. N. (1990).htm. In addition. Sanders. & Solomon. (3rd Ed.. Newbury Park. & Marczak.edu/fcs/cyfernet/cyfar/Costben2. Retrieved on October 15. Rutman (Ed. White Plains. L.. p. H. Thousand Oaks. Levin. H. M. Lipsey. AZ: USDA/CSREES and the University of Arizona.. Evaluation: A systematic approach. Using cost analysis in evaluation. & Worthen. If evaluators are not exposed to such techniques and trained to use them. (1997).
Goethe University of Frankfurt/ Germany and the Free University of Berlin/ Germany.Ideas to Consider Is E-Learning Up to the Mark? Fundamentals in Evaluating New and Innovative Learning Approaches Involving Information. 42 Oliver Haas (M. Here he has been involved in Vocational Education and Training projects. The following article deals with two issues: a) Evaluation methods used in web-based learning arrangements that are dependent on standards in evaluation research b) The implicit logic of the communication medium “internet”.and Communication-Technology Oliver Haas42 Abstract With the introduction of training courses or seminars via the internet it becomes relevant to evaluate and assess these new and innovative forms of learning and teaching. Tanzania. SocSc.) conducted university studies in Sociology at the Johann-Wolfgang v. He is currently employed a Technical Advisor by the German Agency for Technical Cooperation (GTZ) and has worked in Russia. Journal of MultiDisciplinary Evaluation: JMDE(2) 83 . The paper concludes by illustrating methods of evaluation and assessment that correspond to both paradigms. and South Africa. Malaysia.
are always under pressure to legitimize their existence and benefits. This becomes even more crucial when their application is highly related to risks and financial expenditure. Journal of MultiDisciplinary Evaluation: JMDE(2) 84 . and “e- learning” are used interchangeably in this paper. Despite the recent enormous interest generated in learning via the web44. to name a few. and innovative teaching and learning procedures specifically. “online learning”. the most recent learning technologies43 only changed learning strategies in human resource development to a minor extent. skills and better qualification through its utilization compared to the “traditional” methods of teaching and learning? Does the internet change teaching and learning concepts or even our perception of learning? After the last big technical innovation —print media—had a significant impact on teaching and learning.g. However. Efficiency and effectiveness. are key criteria under which these procedures are critically assessed. etc. efficacy and suitability has not been answered fully. “internet-based learning”. Computer based training. the most recent development in Information and Communication Technology (ICT)—the world wide web—as a universal medium of exchanging and mediation of information and knowledge has set a new yardstick regarding the accessibility of learners and the dissemination of learning content. The terms “web-based learning”. quality as well as relevance and significance.Ideas to Consider Introduction Does the internet make us more intelligent? Do we obtain more knowledge. language laboratories. 43 44 E. Innovations in general. the question of acceptance.
These are methodological aspects that cannot be neglected when evaluating learning via the internet. the determination of “economies of scale” has proven to be a challenge on its own. theories on how to go about evaluating learning via the net as well as gaining empirical valid data are difficult to find. this is not possible without a methodological introduction. While the determination of “costs” is relatively easy to define. paradigms and procedures. the question of efficacy and efficiency of online-learning has become increasingly relevant for the private sector. So far evaluations of internet-based learning have been mainly applied in the public sector.g.Ideas to Consider Methodology-driven and comprehensible criteria-based assessments of procedures. However. With regard to in-house and on-the-job training the common understanding is that any training is an investment in employees that needs to be justified like any other investment. Assessments of learning software (e. Due to the complexity of e-learning. Journal of MultiDisciplinary Evaluation: JMDE(2) 85 . Yet over the past years. As such. events or actions are called “evaluations”. It is the aim of this paper to provide a selection of possibilities on how to evaluate the efficiency and benefits of online learning. This is where evaluation becomes relevant. evaluations in this field are not just necessary but also possible. Components in evaluating online seminars To evaluate learning via the internet is nothing new. To evaluate ‘learning via the internet’ requires an accurate preparation of empirical research that can only lead to useful data if meaningful criteria under which the evaluation will take place have been assigned in advance. in the field of Computer Based Training) have been conducted widely and provide useful experiences as well as theoretical concepts. Up to now evaluation of learning via the web has been a seriously neglected aspect of impact assessment in education and training.
Ideas to Consider
In recent years, highly acclaimed work has been done in the development of methodical standards and instruments in evaluating the efficacy of education and training. However, these standards—consisting models, methods, and instruments—can also be utilized in other contexts. Evaluation methods and criteria are often explicitly created for a specific evaluation purpose. This suggests that evaluation—or better evaluation research—should be considered as applied science following a specific need and demand. Areas and criteria of evaluation The setting of criteria should be the first step when assessing online training courses. Criteria need to be defined for each and every aspect of the learning scenario. This involves all participants of the training course, the utilized learning material, the pedagogic approach, and technological aspects, as well as guidance in the learning process and technical-administrative support as part of the training course. Criteria are directly related to the quality of online seminars and constitute the foundation of any evaluative approach to online learning. Thus, a clear statement and definition of evaluation criteria is of crucial importance for the whole evaluation process. Often several areas of evaluation are strongly interlinked and have a significant impact on each other.45 However, for a differentiated assessment of online learning courses it is essential to select evaluation criteria for each evaluation area separately. The decision on evaluation areas and its correlating evaluation criteria is to be done at the beginning of the overall evaluation and needs to be specified for each learning program. Yet, there are some criteria that could be defined as
For example, the influence of the pedagogic approach or the technology on the students’
motivation as well as learning success.
Journal of MultiDisciplinary Evaluation: JMDE(2)
Ideas to Consider
typical in evaluating online learning. The following chart provides a selection of these criteria: Chart 1: Evaluation areas und evaluation criteria for online learning Evaluation area Participants/ students Evaluation criteria • Acceptance of training course • Drop-out rate • Degree of collaborative learning46 • Rate and intensity of interaction with learning content • Learning success • Communication among students • Transfer and utilization of learning content at the workplace Pedagogical approach • Learning and teaching methods • “Didactic of activation”47 • “Didactic of enabling”48 • Degree of “blend” in the pedagogic approach49 Learning material • Editing and processing of learning content • Comprehensibility, amplitude, correctness, “time sensitivity” of learning material
46 47 48 49
User of training courses can collaboratively work on tasks independent of time and space. For a detailed explanation of the term please refer to page 4. see above. “Blended learning” is an integrated learning concept that combines Information and
Communication Technology (ICT) with “traditional” learning methods and media in a single learning arrangement.
Journal of MultiDisciplinary Evaluation: JMDE(2)
Ideas to Consider
• Quality and reliability of connectivity • Technical infrastructure at the learning site (e.g. internet accessibility) • Collaboration and communication tools
Support and administration
• Registration and financing • Online-support • Offline-support • Technical support
Participants When evaluating online training courses it is crucial to remember that it is not the learner that is evaluated but the learning content delivery! However, the learner plays an important role as a resource person for evaluating the overall training program. The learner’s behavior, learning success as well as transfer of learning content to the workplace provide important empirical information with regards to the quality of the training course. Pedagogic approach Even though internet-based learning has been—especially in its early years— strongly related to technology, it is still about the provision of learning content and qualifying people in order to create employability. Therefore web-based learning also—or even more—has to be based on a pedagogic foundation, generally provided through the Curriculum. In Education and Training one distinguishes between two didactic models:
Journal of MultiDisciplinary Evaluation: JMDE(2)
The action-oriented learning approach is based on a holistic interpretation of technical. 1998 German: “Ermöglichungsdidaktik” The “action-oriented learning approach” is one relevant learning approach of this model. the “didactic of activation” assumes that successful learning can only take place if it is adequately planned with learning methods having been selected accordingly and the sequence of learning is being followed rigidly. Learners graduating through this approach are expected to have acquired not only skills and knowledge obtained from qualifications. Rolf/ Schüssler. Werner. 2004). Ingeborg. such as problem solving techniques. Programmatic learning and curricular planning are core determinants of this model. individual. communication skills and the ability to work in teams (see Heitmann. methodological and social competence.53 Degree of “blend” in the pedagogic approach The “blended” learning approach treats web-based learning as one way of delivering learning content. Group work.Ideas to Consider “Didactic of activation”50 Coming from engineering science.51 “Didactic of enabling”52 This model focuses on the learning and its success. Journal of MultiDisciplinary Evaluation: JMDE(2) 89 . the provision of several learning paths and methods to acquire knowledge are key features of the “didactic of enabling”. Consequently. other methods of “traditional” learning 50 51 52 53 German: “Erzeugungsdidaktik” See Arnold. but also “key competencies”. It tries to create an enabling environment for the learner to build up on existing knowledge and to expand his skills and competencies according to his need and demand.
it has to be decided which learning content will be delivered via the web and where other forms of learning material are more relevant. The demand for content delivery via the web must result from the pedagogic approach of the overall learning program. Learning material The learning material provided should be a main focus of the assessment in two respects: 1) Quality of content 2) The processing of learning content into learning material incorporating ICT In terms of quality of content it is mainly comprehensibility. The stability of the Web-server as well and its capacity are aspects that must also be assessed. the utilization of text. pictures. technical aspects are of great importance. Support and administration Journal of MultiDisciplinary Evaluation: JMDE(2) 90 . correctness and “time sensitivity” of learning material that needs to be evaluated. Additionally. simulation. Depending on the Curriculum. animation. Quality and reliability of connectivity including time needed for loading frames and content need to be assessed.).Ideas to Consider such as group or individual exercises remain valid and relevant. etc. Concerning its processing of content into learning material the evaluation should focus on the conversion of content to learning material (e.g. Technical System When dealing with web-based learning. coverage. the extent and time of utilization of web-content and services provided by the learning platform can provide useful information on the suitability of the technology.
Ideas to Consider E-learning can take place on an independent level or as an add-on to teaching (the ‘blended’ approach). Summative evaluations however examine the training course after it has been finalized. Here.54 54 See Kromrey. the main focus is on data compilation. 1998. p. The content-based support provided by tutors and technical administration ensures a smooth operation of the e-learning course. Each has its own suitability to assess the quality of projects and programs and make recommendations for improvement. Newsgroups and mailing lists are a very economic way of using portals of this kind. In the following sections. These types of evaluation aim at ensuring quality and provide useful suggestions for further improvement and refinement of the online course. However. where students can interact and exchange views. These data give useful hints regarding acceptance. Formative and summative evaluations Formative evaluations focus on the training course during its development. the focus will be on the suitability of evaluating learning arrangements in the field of web-based learning. Types of evaluation Evaluation research deals with several different types of evaluation. The more complex a learning course the more support it needs. effectiveness and the impact of online training courses.100 Journal of MultiDisciplinary Evaluation: JMDE(2) 91 . support and administration can be reduced if training courses include collaboration tools. ideas and information. However. various types of evaluation will be outlined.
In contrast. On the other hand one can say that without any empirical data. the assessor’s perspective and most importantly the reasons for conducting the evaluation. the acquisition of data can quickly turn into a wild collection of data without correlations. suitability depends very much on the type of training course. its composition. Internal evaluation and external evaluation Those who have been actively involved in the development of the online-course do internal evaluation. These types of evaluation provide a grid that should support the decision-making process. Here. empirical data provide the foundation to validly answer theorydriven questions. to get clarity on assumptions and hypothesizes and to be able to Journal of MultiDisciplinary Evaluation: JMDE(2) 92 . interaction and structure.g. process-evaluations focus on procedures. External evaluation involves external assessors as a main resource to conduct the evaluation.Ideas to Consider Product-evaluations and process-evaluations Product-evaluations consider one specific product when assessing (e. a learning program). the state of its implementation. In an evaluation. Without evaluation criteria. Obviously a “one-size-fitsall” solution is not possible. Evaluation methods and gathering of data Evaluation criteria and empirical data are two central elements of every evaluation. They conduct the assessment and evaluation of its performance. handling and utilization of these products. showing what type of evaluation is most suitable for which specific training course. questions and presumptions remain without an answer.
evaluation research can be divided into “reactive” and “non-reactive” procedures. This is especially relevant for written assessment methods (e. The answers can mingle with the originally intended aim of the assessment in such way that it is difficult to make a distinction of all results obtained afterwards. “Non-reactive” assessment procedures (such as hidden observations) take place without the awareness of the subject of assessment. As a consequence one can say: The assessment with reactive assessment procedures causes a wild mix of (interesting and not interesting) data. questionnaires. internet-technology provides a number of possibilities to collect empirical data. Generally. The reactive element of responding does not exist and therefore also no distinction of data is necessary after the assessment has taken place. anonymity of all gathered data in reports should be guaranteed. When conducting “reactive” assessment procedures (such as interviews) the interviewee is aware of the assessment being conducted.Ideas to Consider make recommendations. etc. the utilization of non-reactive assessment procedures also has its challenges: After the assessment is finalized the subject of assessment should be informed on the objectives of the evaluation as well as the reasons for conducting the assessment in such a way. rating scales. Therefore the person assessed can react to the assessment process in an unpredictable way.g. methods of empirical research are necessary in order to help collect data without interfering with the actual learning process. Nevertheless.) that are applied frequently in evaluations. Like all methods of empirical research. Generally. Journal of MultiDisciplinary Evaluation: JMDE(2) 93 . That is why when evaluating online training courses.
When dealing with online training courses this issue becomes even more relevant and higher standards need to be set. teaching text. It is advisable to make use of programmes designed for text analysis to analyse comprehensive text. It is selfevident that all documents used in the learning context need to be carefully tested. various variations of data assessment exist.) on various levels play a crucial role within learning arrangements involving ICT. necessary and already formulated questions to be asked during the interview as well as (if necessary) hints and tips with regard to the behaviour of the interviewer. • “Structured interviews” are based on a guideline that has been prepared beforehand. etc. Oral interviews are mostly distinguished by their degree of complexity and structure. Journal of MultiDisciplinary Evaluation: JMDE(2) 94 . Interviews Interviews are the most common method of data collection. documentation of communication amongst the participants including the mentor. Analysis of documents Text and documents (e. This guideline contains all relevant. This is because the mentor. is not always directly available. Here. when selecting evaluation methods it must be noted that obtaining empirical information depends on the decision whether to utilize reactive or nonreactive methods of assessment. A substitution is not possible. curricular text.Ideas to Consider Therefore. The most popular distinction is made between oral interviews and written interviews.g. who functions as a corrective element in the learning process.
These are access data that provide useful information for the evaluation process (e. Online-questionnaires should be the most common way of assessing and evaluating online learning processes.). sequence and duration of utilization. • “Freely structured interviews” are completely free of structure. However. Behaviour can be validly recorded through technology-based recording of behaviour via the usage of the learning platform. etc. Besides that. acceptance of specific learning content.g. one should not overestimate the role and function of these files and the information Journal of MultiDisciplinary Evaluation: JMDE(2) 95 . A specific sequence or wording of questions is not part of the guideline. • “Closed questions” suggest answering options to be chosen from. • “Open questions” provide the interviewee with the possibility to formulate the answer in an individual manner. “log files” inform about time. Written interviews or questionnaires can be divided based on the form of question. Recording of behaviour Behavioural expressions in the context of e-learning can be recorded via the socalled “log files” to be found through the server. where all html-documents of an online course are stored. which is used in the interview.Ideas to Consider • “Semi-structured interviews” are based on clusters and groups of questions and topics to be dealt with during the interview. Observation Observations do not play a significant role when assessing online learning.
If other forms of communication like collaboration tools. it will not be captured by “log files”.g. • “Norm-oriented” methods assess an individual test result compared to a control group. This means that the whole range of research methods is of relevance for assessing online learning. Usually assessment via “testing” is done in an ad hoc way and rather informally. intelligence tests). bulletin boards are being used as part of the learning process it will not be captured by these files. methods of empirical social research need to be adopted and adjusted according to the specific need and demand created by the training course. Standardized forms of testing however are highly sophisticated and complex (e.55 On the other hand they only provide clustered quantitative information that only might be of limited value for the overall assessment.Ideas to Consider provided. Journal of MultiDisciplinary Evaluation: JMDE(2) 96 . “log-files” only cover the access to HTML-documents. After all. Testing In evaluation research the term “testing” describes a standardized procedure to assess the occurrence of empirically defined performance characteristics. • “Criterion-oriented” methods are based on a predefined figure to assess individual test results. Just as in any other area of empirical 55 If passwords have been given out. Empirical research When assessing online learning. On the one hand they do not comprise all of the information necessary to conduct an evaluation. In standardized testing one distinguishes between “normoriented” and “criterion-oriented” methods.
they also have risks and challenges. Due to their low-cost implications and easy application throughout the whole evaluation phase as well as transparency. Methods like these are relevant to obtain prompt results and to get a preliminary orientation for the overall implementation process. The first two sections will illustrate aspects of data survey with regards to concepts of evaluation for web-based learning Utilization of criteria indices In the field of learning software. results gained through criteria indices are easy for others to understand. However. “criteria indices” are widespread. In other words. Generally. the following section will provide a “bird’s eye view” of the topic. it has been empirically proven that assessors who utilize criteria indices when assessing Journal of MultiDisciplinary Evaluation: JMDE(2) 97 . These indices are often not based on sound theoretical ground.Ideas to Consider research the decision on the most suitable method depends on the research question as well as the capacity of the respective research method. after dealing with evaluation of online-learning on an operative level. Concepts of evaluation for web-based learning So far we have dealt with evaluation of web-based learning from a methodical point of view. criteria indices are very popular. Evaluation via criteria indices is based on a selection of various relevant and non-relevant criteria that have been pre-determined by experts. Furthermore. This shows in uncertainties when selecting criteria as well as the emphasis on each and every criterion. Each one has aspects relating to data survey and data assessment. Furthermore. criteria indices can be described as checklists. The following sections will apply the outlined methods with the aim of illustrating three types of evaluation concepts.
Evaluations via criteria indices make it possible to allow statements on the theoretical impact of e-learning.g. Just as with criteria indices it might also make sense to analyze coherence relations via surveys involving training participants. If coherence has been determined (e. Despite all criticism. drop-out rate). but also touch on the elaboration of coherence amongst characteristics.g. the finding is almost impossible to predict for other online-learning programmes if no other variables such as motivation of students are rigidly controlled. difference between achieving a learning objective and learning groups). one should not generally object to the method of criteria indices. with which online-seminars and web-based learning programmes can be validly evaluated. that not only deal with the existence or non-existence of characteristics. A continuation of this approach leads to so-called Journal of MultiDisciplinary Evaluation: JMDE(2) 98 . In fact this is a requirement for many criteria and cannot be replaced by expert surveys. Determination of coherence A step beyond the application of criteria indices are those concepts of evaluation.g.. between the amount of participating students and contributions in online discussions).. individual self-assessment on learning progress) provides further possibilities of application that will be illustrated in the following section. When expanding these indices through variables (e. one can gain a proper instrument. To extend this approach with research on students (e.Ideas to Consider learning programmes obtain results that may deviate relatively strongly from each other. or differences (e. However.. They provide a grid for the area of web-based learning and a possibility for empirical research and evaluation.g. this is not free from difficulties.
g. connection between learning success and participation in a specific learning module/ qualification module). data alone only partially gives a statement on the quality of online-learning (including suggestions for improvement).g.g. The following section will provide an insight into various types of data that can be obtained when evaluating e-learning: Data indicating learning success A quantification of learning success only becomes a valid empirical statement through comparison (e. estimated success in learning) will be determined from interrelations with other variables. Journal of MultiDisciplinary Evaluation: JMDE(2) 99 . where a specific variable (e.Ideas to Consider “linear structure equalisation models”. before/after-comparison) or through inclusion of analyzing relations (e. Chart 2: Concepts of evaluation Evaluation via criteria indices Perspective of the expert Perspective of the user • Learning success • Drop-out rate Evaluation via analyzing relations • Relation between learning success and acceptance of a specific learning/ teaching method • Estimated relation between the degree of communication with other participants and own learning success • Self-estimated learning success • Self-estimated degree of communication and interaction with other participants Aspects of assessing data If the data assessment is based on criteria indices or interrelations.
evaluation research in ICT is still very much in its initial steps. Assuming that out of several alternative learning modules only one is accepted by a single learner or small group. Data relating to technical features As long as these data are not utilized in relation with other relevant data of the evaluation (e. it can still be of significance and relevance for a specific learning gproject. However. one can assess online-learning on a practical level as long as an adequate system of classification comprising all relevant and necessary evaluation aspects has been developed. of this kind proves to be very difficult to interpret. Conclusion and discussion Despite remarkable work being done in related fields such as educational software. Data relating to the acceptance of the offered learning programme Except for the case when responses in this regard are not accessible data. Before making changes based on such statements the data need to be examined further in order to ensure reliability and interrelations with other data. criteria-based evaluation that focuses on interrelations and thus serves the purpose of verifying or falsifying hypothesis-based evaluations. Journal of MultiDisciplinary Evaluation: JMDE(2) 100 .Ideas to Consider Decisions of participants/ experts Judgments made by participants or experts are a fundamental empirical finding when evaluating online-learning. these data can be considered as insignificant. learning success). The system presented in the present paper is in conformity with these functional requirements as it provides a pragmatic. drop-out rate.g.
Rolf. Journal of MultiDisciplinary Evaluation: JMDE(2) 101 . Helmut (1998): Empirische Sozialforschung. Werner (2004): The action-oriented learning approach for promoting occupational performance and employability. Pretoria.Ideas to Consider References Arnold. Schüssler. Kromrey. Skills Development Research Series. Erwachsenenpädagogische Grundlagen und Erfahrungen. 8th edition. Leske + Budrich. Book 3. Opladen. Schneider Verlag. South African-German Development Co-operation. Ingeborg (1998): Ermöglichungsdidaktik. Heitmann. Baltimore.
and I think the issue has extremely general applicability. about the extent Journal of MultiDisciplinary Evaluation: JMDE(2) 102 . and the program should not be ‘charged’ with it. and one that the FDA should require them to release. My take is that the key issue is whether the program. Think about the evaluation of: any chemical drug abuse program. Here are two: (i) it applies to standard pharmaceutical drug evaluation because there is a serious problem referred to as the fidelity or adherence problem. though that’s not the label they use. I find myself taking sides with the vendor against the would-be consumer advocate. Failure to stay with the program—that is. The evaluators. attrition—is therefore (largely but not entirely) a failure on the part of the subject not the program. will produce the claimed results. distance or online education.Ideas to Consider The Problem of Free Will in Program Evaluation Michael Scriven A group of hard-nosed scientists who have been studying the major commercial weight-loss programs recently reported their disappointment that the proprietors of these programs refuse to release data on attrition. First. twelve step programs like AA for alcohol and gambling abuse. ones that you might not think of immediately. On this issue (possibly for the first time in my life). Now it also applies in some important cases outside program evaluation. think it’s obvious that this is a—or perhaps the—key ratio needed to appraise the programs. in evaluation overall. here’s why I think this is a very general problem that we need to deal with. if followed. and that following the program is (largely but not entirely) a matter of strength of will. not only in program evaluation. continuing education of any kind— this clearly applies to all of them.
but of course B is also controlled by strength of will. high front-end loading of payments may be excessive. (B) Supportive power.e.. one can’t be very critical of high-pass filtration for weight-loss. equipment. and ease of use are Journal of MultiDisciplinary Evaluation: JMDE(2) 103 . to motivate. continuing costs (including opportunity costs and fees). For short: Appeal. i. Case (ii): in teacher evaluation. if there’s no money-back guarantee. costs. Of course. given the unreliability of such selection tests in the personnel area (pharmacogenomics is the subject devoted to this in the pharmaceutical area. Think of a program (or drug regimen. as well as to teach good content well. air conditioning. but. and controlled by selection. counseling). B is affected by support level including infrastructure (e.. or educational effort) as having three aspects that we need to consider in the evaluation: (A) Attractive power. not whether it’s used. success is clearly limited. A is affected by presentation. If the support. where it’s considerably more successful) and the importance of giving people a chance when they want to try. although we want to say that the teacher has some obligation to inspire interest. not only by natural capacity—as we all agree—but also by dogged disinterest. (C) Transformative power. Here’s the schema I recommend for dealing with this kind of consideration. for all of which the program is largely responsible. for high failure (‘attrition’) rates where the cause is simply refusal to try. The vendor or provider has the responsibility to use selection to weed out cases who are demonstrably unsuitable for the treatment. distance ed. We don’t want to blame teachers for failing to teach inherently capable students who are determinedly recalcitrant.Ideas to Consider to which patients ex-hospital do in fact take the prescribed dosage on a regular basis. and ease of use. and twelve-step programs.g. In these studies we surely want to say that the merit of the drug lies in what it does if it’s used. marketing and perhaps allocation. Grip. and Impact.
and often taken for the rest of your life. Journal of MultiDisciplinary Evaluation: JMDE(2) 104 . and are at fault if they don’t work. You have to take responsibility for doing your part. it’s something where you choose to get some help in doing something to yourself. They only work if you make them work. While the good evaluator absolutely must check to see if the provider has indeed provided what was promised. Which leaves C. not undergo. Under these conditions. Even with drug treatments. the drugs have to be taken. easily checked and often met. the Impact issue. which you do undergo. attrition is your failure. but that doesn’t mean success is a free lunch. doing the homework. and failure is often the fault of the people not the program. and the evaluator must not take that responsibility away and say that the program failed if it didn’t get you through to the Promised Land. This is not surgery. not the vendor’s. when it was you who failed. getting to the meetings? That’s the key issue. The fact is that they are selected by people as something they will undertake. and that what was provided was about as good as can be provided at the cost level in question. the rest is up to the subjects. This is an important issue because it’s important that evaluation not assume that these treatments are done to people. willpower becomes the controlling variable. taking the pill. We have free will. the real kick in the program: will it deliver as promised if we do our part. which is done to you.Ideas to Consider disclosed in advance and are both reasonable and delivered as pictured and promised. Free will is the freedom to start a program: will power is what it takes to complete it.
based on this pilot test. JES believes that its accreditation scheme will.Global Review: Regions Japan Evaluation Society: Pilot Test of an Accreditation Scheme for Evaluation Training Masafumi Nagao The Japan Evaluation Society (JES) has conducted a pilot test of an accreditation scheme for short-term evaluation training programs. The committee has judged that the course cleared the hurdle for accreditation and. The course was given twice—first in July 2003 and then in August 2004. including on-site guidance and evaluation of its results on the basis of a formal agreement with JES. It also hopes that the scheme will lead to multiplication of evaluation training programs by providing qualification incentives to individuals Journal of MultiDisciplinary Evaluation: JMDE(2) 105 . which provided technical assistance for the formulation of the course design. preparation of the textual materials and actual conducting of the course. by virtue of the norms it sets for evaluation training. A special committee established in JES elaborated tentative procedures for accreditation and examined reports submitted by the course organizer and a JES member dispatched to observe the course. The course contents were adapted from the ‘Essential Skills Series (ESS)’ program of the Canadian Evaluation Society (CES). The course was designed to impart functional competence to school teachers for co-ordinating selfevaluation exercises in their schools. have a direct impact on the quality of the short-term evaluation training programs conducted by public sector bodies and specialized training organizations. is making a recommendation to JES Board that an Accreditation Scheme should be formally established. The program chosen for this test was a 4-day school evaluation training course organized by a public teacher training center in cooperation with Hiroshima University.
In addition to school evaluation. Hiroshima University and can be contacted at nagaom@hiroshima-u.Global Review: Regions interested in evaluation training.ac.jp. Journal of MultiDisciplinary Evaluation: JMDE(2) 106 . aid evaluation and government performance evaluation are considered target areas for the accreditation scheme. Masafumi Nagao is a professor at the Center for the Study of International Cooperation in Education (CICE).
in an ideal evaluation environment. It was followed by an engaging discussion which focused on participants’ endeavours to date at building evaluation culture into their everyday evaluation practice. Organised by the Auckland Evaluation Group. and Chris Mullins Aotearoa/New Zealand had its first national evaluation conference in September 2004. and suggestions about what evaluators would like to be able to do. and partly because of the nature of the conference centre itself—a retreat and spiritual centre located in a quiet setting with native bush and overlooking the lake. Associate Professor and Director of Programmes in Victoria University’s School of Government in Wellington. gnarly questions and halfbaked ideas” and the programme included a keynote workshop. Entitled “Really useful stuff”: A new role for evaluators in building evaluative capacity in New Zealand. near Lake Taupo (central North Island). Twenty-five people attended the conference and spent two and a half days engaged in discussions which participants found inspiring and highly worthwhile. the meeting was held at the Tauhara Centre. several other facilitated workshops and discussion groups. This was chosen partly because it is a central location for North Island evaluators to get to. The conference theme in 2004 was “Radical directions. and a number of more informal discussion groups using an Open Space Technology process. the keynote workshop was presented by Bill Ryan. Maggie Jakob-Hoff.Global Review: Regions Aotearoa/New Zealand Starting National Evaluation Conferences Pam Oliver. including examples of both successful attempts and frustrations. Journal of MultiDisciplinary Evaluation: JMDE(2) 107 .
Global Review: Regions In addition to the programmed workshops—ranging from “looking at ways of educating our clients around evaluation tendering”. such as “who evaluates the evaluators?” A second conference is scheduled for June 29th-July 1st. 2005 at the Tauhara Centre and is being organized jointly by members of the Auckland and Wellington evaluation groups. to “building reflective practice through evaluation”—the Open Space events also resulted in some focused discussion on topics of key importance to the development of the profession. and “aspects of ethical decision-making in evaluation”. More information will be available in the next few weeks. Journal of MultiDisciplinary Evaluation: JMDE(2) 108 .
Last October.Global Review: Regions Conference Explores the Intersection of Evaluation and Research with Practice and Native Hawaiian Culture Matthew Corry One of the great potential sources of innovative thinking in evaluation is events where indigenous peoples come together to explore the ways in which evaluation and research can be designed in a way that is culturally relevant and useful to the communities served by these activities. In recent years. leadership. “I am amazed at how far we have come [to be able to] present from a Hawaiian perspective. and spirituality. there have been several hui (gatherings) where Native Hawaiian and New Zealand Māori researchers and evaluators have come together to explore this theme. educators. family. politics. a Māori/Native Hawaiian working group is compiling a series of papers on the topic for possible publication in a monograph series. The event attracted a broad cross-section of researchers. and cultural practitioners from the fields of education. Currently.” said one participant. economics. cultural practices. the Policy Analysis & System Evaluation (PASE) department of Kamehameha Schools organized a conference at their Hawai‘i (Big Island) campus. environmental studies.” Journal of MultiDisciplinary Evaluation: JMDE(2) 109 . health. “The variation of research presented was wonderful. The purpose of the gathering was to gain a better understanding of Hawaiian well-being by bringing together multiple viewpoints from diverse disciplines.
visit http://www.” For more information about PASE. Journal of MultiDisciplinary Evaluation: JMDE(2) 110 .edu/pase/researchproj-ksrschcon. It made tangible the direct connections between our research and the education of Hawaiian children.edu/pase.php.ksbe.ksbe.” says PASE director Shawn Malia Kana‘iaupuni. statistical analyses. “Everything we do—the surveys. To view presentations from this year’s conference. technical reports. and the sharing of findings with other researchers—is to help achieve a better understanding of how to make a difference for the keiki (children) and families we serve. PhD. Presenters also insisted that native voices are necessary to provide a more complete and accurate portrait of känaka maoli (indigenous activities).Global Review: Regions An overarching theme throughout the conference was the charge to rely on küpuna (elders) wisdom and to balance that knowledge with scientific learning. “It was very special to have the conference on our campus and to be talking about the wellbeing of our children while they were among us building their own futures through education. the longitudinal studies. call (808) 541-5372 or visit www. evaluations.
C. Reviews of evaluations typically conclude that young people who complete DE programs crash at about the same rate as those who do not receive formal education.Global Review: Regions Washington. DC: Evaluation of Driver Education Northport Associates February 16-17. Do some types of driver education programs lead to better educational outcomes and safety impacts than others? How can driver education programs be improved in order to yield safer young drivers? Lack of systematic programs of research and methodological weaknesses in previous evaluations have left these questions partially or completely unanswered. Northport Associates is conducting this project in consultation with an advisory group and other experts. and ecological time series studies. random controlled trials. there have been a moderate number of evaluations. Guidelines Project The AAA Foundation for Traffic Safety and BMW of North America are funding a research project to develop guidelines for improving the evaluation of driver education (DE) programs. including quasi-experiments. Journal of MultiDisciplinary Evaluation: JMDE(2) 111 . D. 2005 Project to Develop Guidelines for Improved Evaluation of Driver Education This is intended to provide a brief overview of this project and of the consultative workshop which took place last week in Washington. Over the long history of driver education.
Indeed. and preparing a final report and guidelines for future evaluations of driver education programs. but the learning curve is long. Content covers legal requirements. vehicle handling. inexperienced drivers are at high risk—16 year olds have 10 times the crash rate per mile of experienced adults. Risk declines rapidly over the first few hundred miles of driving.S. lead contractor on the project. Driver education has long been seen as a societal response to the tragic losses of novice drivers. examining methods and theories of driver education programs.Global Review: Regions Northport Associates. make critical decisions quickly. Classroom Journal of MultiDisciplinary Evaluation: JMDE(2) 112 . one of its principal purposes is to prepare beginners for license testing. and leaving inadequate safety margins. Traditional driver education takes place before the driver becomes licensed. detect potential hazards early. DE programs consist of 30 classroom hours and 6 hours in car. Driver Education—A Challenging Evaluand Evaluating and improving DE is highly challenging. but the potential benefits are very high. and efforts to motivate beginners to fear the consequences of crashes. They often raise their risk through overconfidence and choices such as driving too fast. and maintain consistency in critical thought and action. is reviewing the DE evaluation literature. Both skill deficiencies and risky choices contribute to their excess risk. particularly among youth. Young. Typical U. Novice drivers are less able to control attention. Road trauma is a costly public health problem. and data sources. taking up to 10 years to finally level off. The limitations in skills and abilities that contribute to elevated risk are known. measures. identifying and assessing evaluation methods. accepting small gaps in traffic. scan the environment effectively. interacting with traffic.
Global Review: Regions methods most often include teacher-centred lectures. Organizational constraints provide great challenges for DE program development and to their evaluation. demonstrated effectiveness in reducing the risk of drivers of all ages through education alone is rare. there have been major changes in the technological. and regulatory environments of driver education and also in driver licensing. Because the needed comprehensive programs require coordination across bureaucratic boundaries. There appears to have been limited development of new DE content. simulation. Journal of MultiDisciplinary Evaluation: JMDE(2) 113 . In recent years. business. Utilization of evaluation results is also problematical in the DE field. Self-instruction. recent regulatory provisions recognize a formal course delivered by parents. and web-based instruction are increasingly becoming available. computer-based instruction. that carefully designed multifaceted. and sometimes simulators. with some discussion and support with film and video. In some jurisdictions. It is widely believed. Simply increasing knowledge and skill does not make safer drivers—“better” drivers do not necessarily crash less. Disappointing evaluations led to (or justified) reduced support for driver education in the 1980s. organizational interests and behavior become issues as important as individual behavior change. multi-level behavior-change programs are required. if not yet proven. but instructional and delivery methods are rapidly changing. when a more rational response would have been to redouble efforts to make the programs more effective. While more education is always a popular prescription for improving safety. with the move to graduated licensing suggesting graduated training.
Two dozen invited participants included academics.northportassociates. governance. The consultative Workshop took place February 16-17. and market incentives are critical issues for driver education globally. but seems especially so for the highly diverse driver education field. consultants and research staff from: • National Highway Traffic Safety Administration • National Institutes of Health • Insurance Institute for Highway Safety • American Driver and Traffic Safety Education Association • State Departments of Motor Vehicles Journal of MultiDisciplinary Evaluation: JMDE(2) 114 . structure. whose two main goals—independent mobility and safety—are antithetical.C. The discussion board has been active and received contributions from driver education experts and evaluators. D. Guiding the Guidelines—The Consultation Processes Consultation for the development of guidelines consists of an internet discussion board (www. offices. Much of the board discussion has focused on appropriate objectives and success criteria for DE—safety impacts or other outcomes and impacts. Significant further development is needed for DE to fully satisfy the expectations placed upon it by society.Global Review: Regions Content. More comprehensive evaluation and continuous improvement are seen as critical to progress.com/aaafts) and a two day workshop. Developing guidelines for evaluation of complex programs is always challenging. standards. 2005 in the AAA’s Washington.
UMTRI • Traffic Injury Research Foundation of Canada • The Evaluation Center—Western Michigan University • Georgia State University. Questions and a few representative answers are shown below. Success Case Method Journal of MultiDisciplinary Evaluation: JMDE(2) 115 .g. Institute Of Public Health • Private sector safety research consultants The Workshop members were asked to help answer fundamental questions from their own diverse perspectives..Global Review: Regions • Driving School Association of the Americas • University traffic research institutes: UNC-HSRC. systematic evaluation. What should be done differently? • More comprehensive.. 1. e. e. British Columbia GDl/DE ongoing evaluation/program development process • More formative evaluation.. SPC process in the 1970s • Use of established evaluation standards—Joint Committee Standards for Educational Evaluation • Look at different approaches.g. TTI.g. What has worked in driver education evaluation? • RCTs focused on safety impacts • Quasi-experiments with statistical control 2. e.
What do we want driver education evaluation to accomplish? • To track DE effectiveness. Who are the key users for DE evaluation guidelines? • Evaluators • Policy makers.g. across states.g.. etc. real driver needs versus arbitrary objectives 4..g. legislatures Journal of MultiDisciplinary Evaluation: JMDE(2) 116 .g. e.g. indicators and measures? • Product & process. e.. licensing & crash rates 5.. program quality & consistency. • Recognize needs versus objectives. e. choices regarding investment. feedback & continuous improvement • To compare performance.. What are the key evaluation targets.. quality management • Learning outcomes—knowledge & skills. e.. needs assessment. “see if it works” • To help improve DE. e. risk perception. sample sizes • Hybrid designs. RCTs with modeling to compensate for limitations in control group equivalence 3.g. e. e. speed & space choices • Societal impacts—safety & mobility. e. e. insight • Behavioral outcomes—risk response...g. e..g. • Defend policy.Global Review: Regions • Methodological improvements.g.g.
. data acquisition for intermediate objectives and impacts The Driver Education Evaluation Guidelines project will proceed through drafting and review of materials.. What is the best format for the guidelines? • Emulate good models. e. e.g. Ottawa Health Unit • Moderate in size • User friendly to a wide range of users. e. e. program quality checklist • Support higher level evaluations. It is scheduled to be completed in the Fall of 2005.Global Review: Regions • Parents • Consumer protection • Insurers & policy holders • State level administrators • DE program managers • Researchers 6. clear definitions • Examples of good & bad practice • Cover simple needs. Journal of MultiDisciplinary Evaluation: JMDE(2) 117 ..g. as well as ongoing consultation.g.g. CDC..
Diab. (1988). & Mansmann. E. C. U. Duffy.. Human Factors. T. Glaser. et al. A. B. International Journal of Medical Informatics. (2003).. Laurie. Oslo.org. 237-248.. & Pomietto. telephone: 202-6385944 Selected Bibliography Ammenwerth. George Mason University..Global Review: Regions Principal investigator: Larry Lonero. (2002). D. (2000). Glad. N. Abdalla. & Mayhew. C. e-mail: sosberg@aaafoundation. Young drivers: A study of policies and practices. (1987)... N. C.. Can evaluation studies benefit from triangulation? A case study.ca. A. Goldberg.. R. Norway: Norwegian Center for Research. S. R. Fairfax. Iller.. Anderson. Northport Associates. A. 44(2).. Department of Transportation. D. e-mail: npa@eagle. 287-302. Journal of MultiDisciplinary Evaluation: JMDE(2) 118 . Toronto. K. VA: Center for the Advancement of Public Health. Donelson. Phase 2 Driver Education: Effect on the risk of accident. Fisher. Use of a fixed-base driving simulator to evaluate the effects of experience and PC-based risk awareness training on drivers' decisions. E. A. L. Connerney.. Driver improvement as post-licensing control: The state of knowledge.. telephone: 905377-8883 AAAFTS Project Manager: Dr. ON: Ontario Ministry of Transportation and Communications. Pollatsek. 70(2-3). D. A.. Scott Osberg.
... (1991). & Black. A. Lonero. M. L.. K. Bergisch Gladbach. P. CA: Sage Publications. (1994). H. Manitoba: Manitoba Public Insurance. Clinton. Thousand Oaks. The Program Evaluation Standards. Smiley. Joint Committee on Standards for Educational Evaluation.. Maclean.. motivational and developmental aspects of decision making in young novice drivers. & Kreuter. N. Paper presented at the International Driver Improvement Workshop. K. National Highway Traffic Safety Administration. Lonero. Lonero. E. (1994). P.S. Chipman. W. Mountain View. Mayhew. A. Persaud. K.. L. Peck. Roach. McKnight. Ottawa.. Germany: Bundesanstalt fur Strassenwesen. L. P. G. & Black. Department of Transportation. CA: Mayfield Publishing.. & Katila. education. M. D. K. Driver improvement programs: State of knowledge and trends. Hatakka. Health promotion planning: An educational and environmental approach.. C. J. D.. Wilde.. Ontario: Transport Canada. L. (unpublished). (unpublished). The roles of legislation... Training to improve the decision making of young novice drivers. A. et al. R. M. M.. D. & Smiley. M. (2000). L.. Lonero. L. B.. Volume II: Literature review: Consistency not capacity: Cognitive. Keskinen. A longitudinal analysis of Manitoba Public Insurance Driver Education Program. Winnipeg.. Clinton. K. A. Washington. Clinton. J. M..Global Review: Regions Green.). The Finnish way of driver improvement: Second phase of training. M. Clinton. Government of Canada. W. M.. S. P. DC: U. and Journal of MultiDisciplinary Evaluation: JMDE(2) 119 . (1998). How to assess evaluations of educational programs (2nd ed.
H. L. D. M. CA: Sage Publications.com Lund.). Mayhew. (2000).). M. 34(1S). W. J. Ontario: Traffic Injury Research Foundation. J. A systematic approach (7th ed.. (2003). from www. J. Rogers. E. T. & Simpson. & Freeman.. Petrosino. (2004).. (1986). A. Thousand Oaks... Evaluation methodology basics. K. & Huebner. L. (1998). (2004). New Directions for Evaluation (87). 367-370. B. Shope. K.. P. Utilization-focused evaluation: The new century text (3rd ed. T. Evaluation. R. Q. Thousand Oaks. Changing road user behavior: What works and what doesn’t. Journal of MultiDisciplinary Evaluation: JMDE(2) 120 . H. & O’Neill. 18(5). Hacsi. M. A... H. Lonero. CA: Sage Publications. E. Lipsey.. P. Accident Analysis and Prevention. & Molnar. Thousand Oaks. & Clinton. 63-69. Effectiveness and role of driver education and training in a graduated licensing system. Scriven. Journal of Safety Research.Global Review: Regions reinforcement in changing road user behaviour.drivers. Patton. quoted in Davidson. CA: Sage Publications.. Program theory in evaluation: Challenges and opportunities. M. (1997). J. Graduated driver licensing in the United States: Evaluation results from the early programs. Toronto. Perceived risks and driving behaviour. Ontario: Ontario Ministry of Transportation. A. Ottawa. A. (1997). Rossi. T.
W. R. A review of the effectiveness of driver training and licensing programs in reducing road crashes... Ray. Lonero. (1975). R. F. 36(1). J. Evaluation Models. Kellogg Foundation Evaluation Handbook. S. DC: U. G. Education for driving: An exercise in self delusion? Prepared for the Driver Research Colloquium. OH: Atomic Dog Publishing. M. (2004). & Pirozzo. & Sadof. University of Toronto. 93-101. (2002). Stock. R. Cincinnati. (2001). Weaver. K. J. Department of Transportation. Kellogg Foundation.wkkf. (1983)... L. 2004.. (2001).K.Global Review: Regions Smiley. THCU (The Health Communication Unit). France: MAIF Foundation.. A. & Chipman. Highway Safety Research Institute. D. Brink. The research methods knowledge base.pdf Waller. L.. C.S. Evaluating health promotion programs.K. Trochim. Retrieved November. Accident Analysis and Prevention. J. Paris. Final Report. M. (2004).. W. 1-106. W. H. Evaluation of Safe Performance Secondary School Driver Education Curriculum Demonstration Project (DOT HS-806 568). McClure. Washington. Journal of MultiDisciplinary Evaluation: JMDE(2) 121 . from http://www. P. Injury and risk-taking behavior . ON: Centre for Health Promotion. National Highway Traffic Safety Administration. Stufflebeam. Toronto.org/Pubs/Tools/Evaluation/Pub770. W. New Directions for Evaluation (89).A systematic review. (1998). Turner.
C. Young drivers: Reckless or unprepared? Paper presented at the International Symposium on Young Driver Accidents: In Search of Solutions. N. Alberta. Michigan. P. November.: Highway Safety Research Center.Global Review: Regions University of Michigan. Journal of MultiDisciplinary Evaluation: JMDE(2) 122 . University of North Carolina. Banff. Waller. Ann Arbor. Chapel Hill. June 4 & 5. (1983). F.
GTZ. including SIDA. DFID. A narrative conference report is available at Third AfrEA Conference Report . the Nelson Mandela Foundation and the African Capacity Building Foundation. the World Bank. 2004. The conference was attended by 479 people from 61 countries. The Conference was hosted in conjunction with the Public Service Commission of the South African Government. of which 47 were in Africa. A listserv has been established to continue communication and information sharing among those interested in monitoring and evaluation in Africa. Around 250 participants also attended the 10 pre-conference professional development workshops held on 29-30 November. South Africa.org.afrea. AfrEA is planning several follow-up activities in partnership with national associations and other interested organizations before its next conference. which will be held in Niger in 2006.Global Review: Regions African Evaluation Association AfrEA56 The African Evaluation Association (AfrEA) held its Third Conference in Cape Town. on December 1-4. For additional information on evaluation in Africa 56 This piece is based on the News section of the African Evaluation Association Website.org/. Conference proceedings will be available in the first few months of 2005. Available at http://www. Details can be obtained from the AfrEA Secretariat at info@afrea. in both English and French. Journal of MultiDisciplinary Evaluation: JMDE(2) 123 . and was supported by another 21 local and international organisations.
Journal of MultiDisciplinary Evaluation: JMDE(2) 124 .Global Review: Regions visit the AfrEA Website or see the first issue of JMDE (Part III: Global Review— Regions) at http://evaluation.html.edu/jmde/JMDE_Num001.wmich.
development and communication of best practice in impact assessment. Impact assessment. IAIA defines impact assessment as.” Conversely. Its international membership promotes development of local and global capacity for the application of environmental assessment in which sound science and full public participation provide a foundation for equitable and sustainable development. Impact Assessment and Its Relationship with Evaluation IAIA is a “shadow” professional organization of AEA. Jane Davidson defines impact as. This Journal of MultiDisciplinary Evaluation: JMDE(2) 125 . This term is similar in meaning to the terms outcome and effect. is pertinent to evaluation.” “IAIA is a forum for advancing innovation. yet differences between the definition of “impact” by evaluation experts and impact assessment experts cause confusion. Below is a discussion of the relationship between impact assessment and evaluation from the perspective of a “traditional” evaluator. Like AEA members. Youker International Association for Impact Assessment (IAIA) defines impact assessment as “the process of identifying the future consequences of current or proposed action. as defined by IAIA. evaluation most frequently refers to retrospective studies.” I. IAIA members are concerned with evaluation issues as they pertain to predicting longterm outcomes. “the process of identifying the future consequences of current or proposed action.Global Review: Regions International Association for Impact Assessment Brandon W. “change or (sometimes) lack of change caused by the evaluand. In Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation (2005). E. particularly as it relates to the human environment.
the IAIA’s definition is inconsistent with that of the Evaluation Thesaurus (1991). 241). rather. (Davidson.” Scriven’s definition mentions nothing of long-term or future outcomes. which defines an impact evaluation as “an evaluation focused on outcomes or payoff rather than process. Therefore. and/or conduct (the typically defined) program and policy Journal of MultiDisciplinary Evaluation: JMDE(2) 126 . p. However. Despite the confusion over the multiple meanings and uses for the term impact. 1991. impact assessments select relevant values and determine merit criteria in evaluating both the planned intervention and several alternative interventions to find the “best” (greatest benefit with least cost) potential intervention. It is clear that impact assessments would surely investigate beyond solely outcomes and also study the process and implementation of the planned intervention.” The IAIA definition of impact assessment and the Davidson definition of impact at least seem to agree on long-term outcomes as a focus of assessing impact. IAIA’s description and definition of impact assessment leads the author to conclude that an impact assessment is in fact an evaluation. and evaluation of long-term outcomes.Global Review: Regions term impact is often used to refer to long-term outcomes. Furthermore. an impact evaluation is focused on actual outcomes and does not examine other program or policy components. An impact assessment is a type of evaluation and may have utility for a certain evaluators. As the author previously reported. p. 2005. several aspects of impact assessments are evaluative in nature and include evaluation-type tasks. use evaluation reports. To elaborate further. (Scriven. prediction. It is an evaluation of interventions (program/policy) and alternatives based on long-term or future outcomes. impact assessments frequently incorporate evaluation methodology. an impact assessment is a process that determines the value of policies and programs in relation to future consequences. 190). delivery or implementation evaluation. IAIA uses impact assessment to be the study.
and various UN agencies. and US Council on Environmental Quality. Journal of MultiDisciplinary Evaluation: JMDE(2) 127 . practitioners and others who utilize impact assessments. the Canadian Environmental Assessment Agency. bio-physical. US Forest Service. IAIA Partnerships and Interested US Agencies IAIA has strategic partnerships formed with the Canadian International Development Agency. the Netherlands Association for Environmental Professionals. Methodology of impact assessments is beyond the scope of this paper. II. for additional information on impact assessment methodology. IAIA Members and activities Introduction IAIA was founded in 1980 aiming to provide an international forum for researchers. as it pertains to specific human activities. US Environmental Protection Agency. private consultants and policy analysts.500 members from 100 plus-countries include corporate planners and managers. Evaluations may be particularly germane in developing monitoring and management systems for these outcomes. Examples of other US federal agencies that may conduct social impact assessments or utilize its principles: US Bureau of Reclamation. US Department of Transportation. or policy. US Council on Environmental Quality.Global Review: Regions evaluations. public interest advocates. Evaluation experts may find IAIA and impact assessments especially relevant if they have interest in studying the future outcomes of social. see the IAIA Website. Its more than 2. the World Bank. health. government planners and officials. and university professors and students.
public participation. health impact assessment. and other impact assessment-related material. health. editorials. disasters and conflicts. technology assessment. social. case studies. o Quarterly. professional practice ideas. environmental legislation. project appraisal. as well as professional news related to impact assessments. it offers regional conferences. and professional exchange opportunities. cost-benefit analysis. sustainability. IAIA produces Impact Assessment and Project Appraisal. o An IAIA newsletter is published quarterly. local and regional government policy and impact assessment. a journal containing peer-reviewed research articles. Additionally. It provides information regarding association activities and events. Journal of MultiDisciplinary Evaluation: JMDE(2) 128 . The journal is only available to members of IAIA or vis-à-vis purchase. o IAIA presents an annual conference and each year IAIA chooses a different global location. article and book reviews. indigenous peoples. trainings. and social impact assessment. The Sections of IAIA are biodiversity and ecology. environmental management systems. and a professional practice section. integrated assessment of traderelated policies. strategic environmental assessment. The journal focuses on the environment. institutions and policies.Global Review: Regions IAIA Topical Sections There are 11 sections of IAIA that provide more in depth coverage of a topical debate. Additional IAIA Activities o Several topical listservs provide networking opportunities and dialogue regarding impact assessments for IAIA members.
J. References Davidson. (2005). Inc. Inc. International Association for Impact Assessment (IAIA) Website. Available at http://www.iaia. Evaluation methodology basics: The nuts and bolts of sound evaluation. Evaluation thesaurus (4th ed. (1991). CA: Sage Publications. IAIA (2005). CA: SAGE Publications. E.).org/ Scriven.Global Review: Regions o The IAIA Website offers “Key Citations” of background reference material related to the various areas of impact assessments. Newbury Park. Environmental Impact Assessment’s index of Websites is a preliminary index of useful Internet sites used as a preliminary guide for environmental impact assessments. Journal of MultiDisciplinary Evaluation: JMDE(2) 129 . M. Thousand Oaks.
submission procedures. Skills. (2) Evaluation Outputs. 2005 in Toronto.Global Review: Regions An Update on Evaluation in Canada Chris L. Consequences. and Effects of Evaluation. (4) Evaluator Professionalism. Proposals for the conference are due no later than March 11. Ontario. 2005. and (5) Evaluation Process Issues. Of notable interest is the Grey Literature database. conference presentations. a fully searchable bank of almost 500 Journal of MultiDisciplinary Evaluation: JMDE(2) 130 . For complete information on the conference including strands. and Competencies. interviews. (3) Evaluator Knowledge. The conference will be held October 24-30. Over 90 complete papers and presentations (primarily from the conferences of the ESS and the AES) are available through links within the newsletter. Canada. respectively) and is divided into five sections. (1) Benefits. and general information please visit the CES/AEA 2005 conference Website. S. and dissertations. This newsletter concentrates on contributions from the European and Australasian Evaluation Societies (EES and AES. Crossing Boundaries. book reviews. September 2004) focuses on unpublished evaluation-related articles. The most recent CES Newsletter (Vol. topical interest groups (TIGS). Coryn News and Events The most pertinent news on the Canadian front is of course the 2005 joint Canadian Evaluation Society (CES)/American Evaluation Association (AEA) conference: Crossing Borders. 4. The Grey Literature Database and Evaluation Report Bank The CES Website offers a wide array of useful resources for evaluators.
Both the Grey Literature database and Evaluation Report Bank are well organized. date.Global Review: Regions evaluation-related documents encompassing diverse areas of interest to evaluators. and the Canadian Evaluation Society). a resource for evaluation reports related to Canadian academia. for example. policy analysis. author. ethics. including those of the European Evaluation Society. the Australasian Evaluation Society. for example. theory. The Grey Literature database is searchable by recent conferences (from 2000-2003. and unpublished papers. easy to navigate. full-length evaluation reports. Also available on the CES Website is the Evaluation Report Bank. These documents include. French. Journal of MultiDisciplinary Evaluation: JMDE(2) 131 . topic. and Spanish). and accessible free of charge to non-CES members. conference presentations. and communication and reporting. government. or language (English. and the private sector.
evaluation societies within Europe. OECD. the expert network of member states. the GAO. The Web site of the evaluation unit provides information about evaluation activities and ongoing studies. Schroeter I apologize for exclusion of those national societies whose language I have not mastered and would like to encourage evaluators from the European countries to contribute a report about the state of evaluation in their country to JMDE. projects. and links to other organizations conducting evaluation (e. including links to specific commissions within the EU. and policies of the General Directorates within the EU. guides on evaluation in different contexts (General Directorates) and on evaluation specific methodology. evaluation is promoted by the Directorate General for Budget.g. Details of events and news items listed below can be found on the corresponding Web sites. policies and procedures. Evaluation and the European Union Within the European Union (EU). which commissions and conducts ex ante (prospective). Journal of MultiDisciplinary Evaluation: JMDE(2) 132 . UNICEF.Global Review: Regions An Update on Evaluation in Europe Daniela C. and ex post (retrospective) evaluations of programs. interim.. the World Bank. and the International Monetary Fund). UNDP.” information on evaluation networks. procurement of evaluation via “calls for expressions of interest.
Spain. 2004 the EES held its 6th Conference in Berlin. the EES published a summary of a proposed strategy for evaluation. September 2004). The EES board promotes (i) the provision of evaluation training through the society. which takes place September 25-30. policy designers.Global Review: Regions The European Evaluation Society In October. Attendees’ feedback and the conference program can be viewed on the EES Web site. (ii) support and consulting services for developing evaluation training by national evaluation societies in Europe. Between 24-30 experienced evaluators. and education in Europe (see http://www.pdf). 2004) reports on the 6th conference of the EES and introduces the new board for the year 2005. Germany where over 420 participants from all over the world attended. More detailed information on how to register will be posted on the EES Web site. Additionally. (iii) collaboration with institutions of higher education to develop degrees in evaluation. Papers and presentations from the conference are available through links within the Canadian Evaluation Society’s Newsletter (Vol. 4. complexity. training.org/docs/Strategy_for_Education_and_Training. and change managers are invited to participate and learn from international evaluation experts. the EES Residential Summer School—Evaluating innovative policy instruments: change. 2005 in Seville. The most recent Newsletter of the EES (December. and policy learning. is announced.european evaluation. Moreover. Journal of MultiDisciplinary Evaluation: JMDE(2) 133 . and (iv) functioning as a connector between potential collaborators to develop evaluation training for those in need.
please visit www. and Decision Support. 2005 • Third Argentine Congress of Public Administration in San Miguel de Tucumán. The call for papers is published at: http://www." EASY-ECO. Germany. Argentina with the topic Society. Of specific interest are e-learning opportunities provided by the Center for European Evaluation Expertise (Eureval-C3E). The Spanish Evaluation Society The Spanish Evaluation Society announced: • Fifth Annual Campbell Collaboration Colloquium on Supply and Demand for Evidence in Lisbon. Evaluation. State.uk/idpm/news/#easyeco. 2005 in Giessen.ac. For details and further information. the DeGEval announced an invitation for a conference on Multifunctionality of LandscapeAnalysis.sed.sustainability. Portugal. Moreover. 2005 in Manchester. the following event were announced on the listserv of the DeGEval: • EU-sponsored series of three conferences and four training opportunities on the "Evaluation of Sustainability. on May 18-19.Global Review: Regions The German Evaluation Society The German Evaluation Society (DeGEval) announced various training opportunities.manchester. Most recently. February 23-25. • The first EASY-ECO conference “Impact Assessment for a New Europe and Beyond” takes place June 15-17. and Administration The Swiss Evaluation Society The Swiss Evaluation Society (SEVAL) announced: Journal of MultiDisciplinary Evaluation: JMDE(2) 134 . 2005-2007.at/easy/.
Febuary 21-25. Participatory Approach to Monitoring and Evaluation. Switzerland • International Conference on "Visualising and Presenting Indicator Systems. The UK Evaluation Society The UK Evaluation Society (UKES) announced various events sponsored by regional networks: • London Evaluation Network: Participatory Approaches to Evaluation. Switzerland • Outcome Mapping: Practical. NWEN annual conference. Manchester. 2005 in Berne. 2005 in the Institute of Commonwealth Studies • Scottish Evaluation Network: What’s working? Improving the contribution of research and evaluation to organizational learning. 2005 at the 135 Journal of MultiDisciplinary Evaluation: JMDE(2) . The 5th International Conference on Evaluation for Practice takes place July 13-15." March 14 – 16. in Our Dynamic Earth. a 4-day workshop. Additionally and as announced on the EvalTalk listserv. 2005. 2005. Holyrood Road. Germany Others events listed on SEVAL’s Web site include training opportunities provided in Switzerland. June 3. 2005 Lebensgarten. Edinburgh • North West Evaluation Network: Evaluation today: emerging themes and improving practice. February 25.Global Review: Regions • SEVAL congress—Evaluation in Education. Flexible. July 1. Neuchâtel. February 3 or 10. Swiss Federal Statistical Office.
The conference will focus on evaluation practice and its implications in areas such as social services. education. England. The keynote speaker is Professor Michael Scriven.Global Review: Regions University of Huddersfield. Journal of MultiDisciplinary Evaluation: JMDE(2) 136 . and health services among other. social work.
and the Brazilian Evaluation Network. 1. environment. Work opportunities for evaluators in LAC There is an increasing number of job postings being advertised in three major LAC evaluation discussions lists—RELAC (Latin American and the Caribbean Evaluation Network). PREVAL (Program for Strengthening the Regional Capacity for Evaluation of Rural Poverty Alleviation Projects in Latin America and the Caribbean).Global Review: Regions An Update on Evaluation in the Latin American and Caribbean Region Thomaz Chianca This short paper updates the evaluation scene in the Latin American and Caribbean (LAC) region. and (4) the internal evaluation system supported by the Brazilian federal government. including education. child labor. and institutes) were the two main clients offering such opportunities. agriculture. Government agencies and nonprofit organizations (NGOs. and socioeconomic development. at least eleven job postings have been advertised. (2) news about national-level evaluation organizations. It is interesting to notice the broad range of areas covered by such announcements. Just in the past three months. Such augmentation in the number of job postings announced in these evaluation lists can be considered in two ways: it is not only an increase in the existing market Journal of MultiDisciplinary Evaluation: JMDE(2) 137 . (3) professional development courses available in 2005. Specifically. it covers four aspects of that scene: (1) evidence on the growing number of job opportunities for evaluators in the region. Ten of them were procurements for external program evaluators and one was a search for a full-time coordinator of monitoring and evaluation. foundations.
Global Review: Regions for professional evaluators in the region. Develop monitoring system based on Logic Model for a social-economic development program Provide leadership and manage all activities related to monitoring and evaluation within ASF Evaluation of projects in the area of race-ethnicity and education Several evaluations of projects in the area of environment and rural development in LAC countries Technical assistance in monitoring and evaluation of international education initiative to combat child labor in Latin American and other countries External final evaluation of program for eradicating child labor in Guatemala Evaluation consultant Evaluation consultant Evaluation consultant Evaluation consultant Nonprofits Evaluation consultant Coordinator for Monitoring and Evaluation Evaluation consultant Evaluation consultants Evaluation consultants Private Development Agency OIT—International Labor Organization Evaluation consultant Journal of MultiDisciplinary Evaluation: JMDE(2) 138 . Table 1. Employment Opportunities in LAC Sector Government Client Ministry of Education— Brazil Ministry of Education— Brazil Ministry of Education— Brazil Brazilian Environmental and natural Resources Institute—Brazil Fideicomisos Instituidos en Relación con la Agricultura (FIRA)— Mexican Government World Vision—Brazil Acción Sin Fronteras (ASF)—Peru Instituto Avaliação— Brazil Killefit Consult— Colombia Juárez and Associates— USA Position Evaluation consultant Description Assess intermediary results of methodology of Active School Program—improving the quality of public schools Develop evaluation methodology to assess strategic planning of education secretaries within the FUNDESCOLA Project Independent evaluation of GESTAR I Program—improving quality of Math and Language education Evaluation of projects related to the Promising Initiatives Component of the Pro-Várzea Program in the Amazon region Evaluation of Mexican government programs in livelihood/agriculture. Table 1 presents some additional details on the job positions advertised. but also a general indicator that local evaluators are being sought to fill the available positions.
Pernambuco.” including a training course they are providing (see “Evaluation Training” below for details). Rio de Janeiro.avaliabrasil. We hope to have some more to report on the Argentinean Evaluation Association in the next issue of JMDE. Bolivia.br). Chile and Venezuela have already started consistent efforts to create their evaluation networks. the “Asociación Argentina de Evaluación. Minas Gerais. one new national evaluation organization was formed and three others are taking the initial steps towards their official creation. LAC Evaluation Organizations Since November 2004. and the UNICEF continue to play important roles in the establishment of national organizations. While searching for information to write this update. Costa Rica and Peru. The Network maintains a website (www.Global Review: Regions 2.gov. Colombia. and São Paulo. PREVAL.org.ar).com. joining Brazil. I came across some references to an evaluation association in Argentina. The following are some short news items about the existing national evaluation organizations in the region. The most recent activity promoted by the Network was an evaluation seminar involving professionals working in the social development area during the World Social Journal of MultiDisciplinary Evaluation: JMDE(2) 139 . RELAC. Brazil: The Brazilian Evaluation Network has now about 300 members divided among its five state chapters in Bahia. At the closing of this issue of the journal I had not received any additional detail regarding the organization from the provided contact person—María Isabel Andrés (email@example.com) and a discussion list (ReBraMA-subscribe@yahoogrupos. The Nicaraguan Monitoring and Evaluation Network (RENICSE) became the fifth organization officially created in the region.
government agencies.cr) or Ana Laura Ibaja Jiménez (firstname.lastname@example.org. The programs of the two national events organized by SIPSE can be found at consorcio@consorcio. mission.co. nonprofit organizations. More information about ACE can be obtained by contacting Welmer Ramos (ramosacu@racsa.Global Review: Regions Forum in Porto Alegre (Brazil) in January 2005. Nicaragua: The Nicaraguan Monitoring and Evaluation Network (RENICSE) has hosted three meetings and is working towards defining its vision.br). Eduardo Centeno Cruz (ecenteno@ibw. international cooperation agencies. For more information about SIPSE contact Gloria Vela (gvela@cable. evaluation and systematization approaches as a way to foster participative democracy.org. It works via a coordinating committee that rotates every six months charged with fostering participation and articulations for accomplishing the annual plan. The Network has an active electronic discussion list. SIPSE has as its main goal applying planning. monitoring.ni) can provide further details about RENICSE. ACE has recently launched its first newsletter (http://www.com. and strategic objectives.cr). Additional information Journal of MultiDisciplinary Evaluation: JMDE(2) 140 . Costa Rica: Costa Rica hosts the Central American Evaluation Association (ACE)—the first formal evaluation organization created in LAC. and grassroots organizations.co. For more information about this organization contact Rogério Silva (rrsilva@fonte. Evaluation and Systematization Network (SIPSE) has its headquarters in Cali at the San Buenaventura University and consists of professionals working in academia.org/ace) and is regaining space as a major reference for the region.geolatina. Colombia: The Colombian Planning. Peru: The Peruvian Monitoring and Evaluation Network has developed its annual plan for 2005.co).co. Monitoring.
For additional information on RELAC contact one of the members of the coordination committee: Consuelo Ballesteros—South Cone (consueloballesteros@vtr. RELAC: The Latin American and the Caribbean Monitoring.co.org).email@example.com). public policy. evaluation standards. Gloria Vela—Colombia (gvela@cable. (ii) defining strategies for establishing alliances.br). and (iv) establishing working groups (e. Evaluation and Systematization Network (RELAC) had its first international conference in Lima. Rogerio Silva—Brazil (firstname.lastname@example.org).jara@alforja. The conference had 142 participants from 22 countries. Peru. Welmer Ramos—Central America (email@example.com. and Oscar Jara (oscar.net. Ada Ocampo (ada. A Coordination Committee for RELAC was formed with the support of representatives from 16 countries that already have or are in their way to create national evaluation organizations.org). including videos. and capacity building).g. A preliminary plan of action for 2005 was approved that include: (i) defining operational norms.pe).pe. as well as the strategic planning for RELAC will soon be available upon request at: preval3@desco.Global Review: Regions on the Network’s activities and accomplishments can be provided by Emma Rotondo (rotondoemma@yahoo. Evaluation training The availability of a number of on-line evaluation courses in Spanish is probably the most interesting fact regarding evaluation training in LAC not covered in the last issue of JMDE.cr) 3.ar).ar).com. Marco Segone (firstname.lastname@example.org. At least three institutions are offering such courses in 2005: Journal of MultiDisciplinary Evaluation: JMDE(2) 141 . in October 2004.org. A CD-Rom with all the material presented at the Conference. Luis Soberón—Peru (email@example.com).com. (iii) designing communication and support strategies for new national organizations. systematizing.. Emma Rotondo (firstname.lastname@example.org.
They are free of charge and are available year around. Journal of MultiDisciplinary Evaluation: JMDE(2) 142 . Access their website for additional information: www. The Center for Studies in Economic Development (CEDE)—University of Los Andes in Bogotá.htm (2) PROGRESO Social Projects. More information at: www. (ii) Project Monitoring and Evaluation.org (3) The Inter-American Development Bank (BID) is offering a series of four online courses covering: (i) Logic Models for Project Design. Management.progresoperu. the National School of Public Administration and the National School of Public Health in association with the Institute of Social Studies. DF. Colombia is offering a specialization course in Social Evaluation of Projects. For details access: www. a Peruvian nonprofit organization.org/int/rtc/ecourses/esp/index. and Resources.iadb. Brazil. (iii) Evaluation of Environment Impact. the University Nacional del Litoral and the Center for Development and Technological Technical Assistance Centro for Public Organizations (TOP) are offering an on-line certification course in Outcome and Impact Evaluation of Public Organizations and Programs. (The Netherlands) are offering a certification course on Evaluation of Social Programs targeting primarily public administrators.org.top.ar/curso_virt6. The courses have two quite attractive features. is offering an on-line course in Qualitative Methods for Evaluation. March to June 2005 in Brasilia. and (iv) Institutional Analysis.Global Review: Regions (1) In Argentina.htm Some courses not covered in the October 2004 JMDE issue include: The Brazilian Ministry of Social Development.
Budget and Management has taken important steps towards the development of an evaluation culture within the public administration system. Monitoring and Evaluation—Aug/Sep 2005. Chile.Global Review: Regions The Latin American Institute for Social and Economic Planning (ILPES) will be offering at least three short-term evaluation courses in 2005: (i) Use of Socioeconomic Indicators to Evaluate the Impact of Poverty Reduction Program— July 2005. The Argentinean Evaluation Association (AAE) will be offering a year-long (March-December 2005) specialization course on project identification. Antigua. The government is also trying to learn from other countries’ experiences—a group of staff members recently visited the Journal of MultiDisciplinary Evaluation: JMDE(2) 143 . Studies to develop a more participative evaluation system within the federal sphere are underway. (ii) Logic Model. Santiago. Evaluation within the Brazilian federal government The Brazilian government under the leadership of the Ministry of Planning. 4. An evaluation manual has been produced offering a framework to orient all program managers to assess their efforts as a way to improve their practices.ar). The idea is to establish a flexible monitoring system with the input from the different ministries External evaluations of four major federal programs are expected to be implemented in 2005. elaboration. Colombia. For details contact María Isabel Andrés (mandal@mecon. Important efforts have been made in the direction of establishing internal evaluation strategies for all federal programs under the umbrella of the federal Pluri-Annual Plan (PPA) for 2004-2007.gov. and evaluation as part of the Capacity Building Program of the Secretary of Economic Policy of the Argentinean Government. (iii) Planning and Evaluation of Public Investments Projects—Oct 2005. Guatemala. Cartagena de Indias.
Chianca@wmich. relevance.santos@planejamento. on the model of the American General Accounting Office (GAO) that will be able to provide candid accounts of the merit. Emma Rotondo (Peru).br).Global Review: Regions US. and Welmer Ramos (Costa Rica). Gloria Vela (Colombia). Eduardo Centeno Cruz (Nicaragua).edu. Journal of MultiDisciplinary Evaluation: JMDE(2) 144 . Andréia Rodrigues dos Santos (Brazil). open to the public and including the participation of evaluation specialists from Canada. was organized to share lessons learned from such visits. Additional information regarding the work done by the Brazilian Ministry of Planning can be obtained from Andreia Rodrigues dos Santos (andreia. Hopefully such efforts will culminate in the establishment of an independent evaluation agency. An evaluation seminar. Such evaluations will help strengthen government’s accountability as well as provide quality information to help decision makers make better use of the scarce resources available. or if you want to send additional contributions regarding evaluation in Latin America and the Caribbean please do not hesitate to contact me at Thomaz. Canada and the United Kingdom. Rogério Renato Silva (Brazil).gov. If you have additional information or corrections on any of the topics covered by this article or by the previous one. Final Note I would like to thank the many people that provided me with key information to develop this update: Ana Laura Ibaja Jiménez (Costa Rica). and significance of the federally funded programs. The advances produced in this area in Brazil are unquestionable and show a clear interest in promoting long-lasting changes.
the use of alternative ways to estimate the counterfactual. and Antoinette Krupski examine the effect of the 1997 termination of the Social Security Administration’s Disability Insurance and Supplemental Security Income benefits for persons diagnosed with drug or alcohol addiction. The paper describes and illustrates innovations and recent developments in quantitative methods for evaluation. and the American Evaluation Association (AEA).Global Review: Publications Summary of American Journal of Evaluation. propensity scoring analyses. Mark. please go to www. Dr. including a combination of the interrupted time series with growth curve modeling. The paper is a valuable example of the principled examination of quantitative data that can move us beyond overall. Orwin and his colleagues go further. These analyses suggest a far more nuanced interpretation of the effects of benefit termination than did the primary. Mark. Robert Orwin.eval. state-of-the-art tests. Importantly. Journal of MultiDisciplinary Evaluation: JMDE(2) 145 . and sensitivity analyses to empirically assess the plausibility of validity threats. Issue 4 of the American Journal of Evaluation. In the first paper. Mark The following is excerpted from the introduction to Volume 25. 2004 Melvin M. Bernadette Campbell. Melvin M. The American Journal of Evaluation is the official journal of the American Evaluation Association and is distributed to AEA members as part of their membership package. Robin Miller. Kevin Campbell. It is reprinted here with permission from Dr.org. AJE’s current editor. To learn more about AEA and how to receive AJE. carefully conducting and considering the implications of a set of post-hoc exploratory analyses. by former AJE editor Dr. Volume 25(4). global estimates of an intervention’s average effects.
Ryan asks how evaluators can contribute to more democratic forms of educational accountability. In the third paper in this issue. The paper thus will be of interest to readers who would like to learn more about propensity score analyses. The three are the seminal democratic evaluation approach of MacDonald. in part by presenting for each a vignette describing a case in which the approach was implemented. and critiques three approaches which fall under the broader umbrella of "democratic evaluation approaches". Journal of MultiDisciplinary Evaluation: JMDE(2) 146 . Leow and her colleagues illustrate a kind of sensitivity analyses. as a way of helping assess how much uncertainty one should ascribe to evaluation findings. Ryan compares and contrasts these three approaches. the bias that might arise from background factors that are not controlled for in the analyses. Katherine Ryan describes. Instead. and the emerging notion of communicative evaluation advocated by Niemi and Kemmis. illustrates. which allows them to examine how susceptible their findings are to what is called hidden bias. Leow and her colleagues use propensity score methods in an attempt to control for the biases that otherwise would result because of the systematic differences between students who take advanced courses and those who do not. and Robert Boruch address the question of whether taking advanced courses in math and science improves performance on basic achievement tests. In effect.Global Review: Publications In the second paper. Christine Leow. the deliberative democratic evaluation approach of House and Howe. Sensitivity analyses should be an important technique in the tool kit of quantitative evaluators. Elaine Zanutto. Ryan goes beyond simply examining the three democratic evaluation approaches in the abstract. that is. she considers the implications of these approaches in an environment in which educational accountability has been shaped by the No Child Left Behind legislation and related forces. Perhaps of more interest. Sue Marcus.
for example. Kevin Lund. among them: How do practitioners learn about so-called evidence-based programs? What are the processes by which they adopt such programs and eliminate their current programs? Are the evidence-based programs likely to be implemented with sufficient fidelity that one would expect good outcomes? Tena St. In one sense. and Charles Warfield address "multilevel evaluation. in a replicated case study investigating school adoption and implementation processes of an evidence-based substance abuse prevention program. In the final paper in the Articles section. and efficiency across the evaluations at the different levels. Huilan Yang. or incentives are put into place in an effort to lead practitioners to use programs that have passed some evaluative threshold. three levels: project. cluster. Yang and her colleagues argue that the literature on multisite evaluation demonstrates the need for an alignment model specifically focused on multilevel evaluations. compatibility. Jianping Shen. Pierre and D. Tricia Leakey. and Karen Glanz address an issue of considerable importance to those who Journal of MultiDisciplinary Evaluation: JMDE(2) 147 . Honggao Cao. when there are multiple sitelevel projects within a broader programs or. The findings should be noteworthy to those interested in program implementation.Global Review: Publications As one outcome of the evidence-based practice movement. Karin Koga. to facilitate congruence. in the way schools choose to adopt and adapt programs. Lynne Kaltreider address these and related questions. recommendations." which arises. However. and more generally in how mandates for evidence-based practice play out in real life. The authors of this paper lay out a process to facilitate multilevel evaluation alignment. the process can be seen as the application of sound evaluation planning in the multilevel program context. as in the example Yang and colleagues discuss. But this trend raises several questions. and initiative. that is. there seems to be a growing trend whereby mandates.
and to the steps taken to disseminate findings and facilitate use. questioned the evaluator to understand more about the various choices he or she made throughout the evaluation. to those who need to make sense of and use evaluation findings? May discusses and illustrates the use of three guidelines for formulating and presenting more meaningful statistics.Global Review: Publications evaluate programs based in schools or. This issue includes an atypical contribution in the Exemplars section. more generally. I invited Jody Fitzpatrick to reflect on the numerous interviews she had conducted. Jody Fitzpatrick. from the initial steps in planning. and describe their experiences in this article. invited Lois-ellin Datta (2002) and Nick Smith (2002) to examine previous commentators' responses to 10 ethical challenges Morris had previously posed in the section. Two years ago. Leakey and her colleagues describe a case from their own evaluation experience examining a smoking prevention program. In the past. In the Method Notes section. interpretability. section editor of Ethical Challenges. As Journal of MultiDisciplinary Evaluation: JMDE(2) 148 . May also offers several interesting and valuable examples for reporting a variety of statistics. who work with participants who are minors: obtaining parental consent. Michael Morris (2002). this section has presented a series of interviews with evaluators who discuss a specific evaluation they had conducted. They employed different consent procedures at different times. These are understandability. to the involvement of stakeholders. In those interviews the section editor. to the data collection methods and evaluation approaches employed. Such an effort to "sum up" previous work in a section of AJE is not completely new. in more meaningful ways. both simple and complex. With the naming of a new editor for the Exemplars section. Henry May addresses a classic and continuing concern for evaluators: How can we best communicate our results. especially statistical findings. and comparability.
Smith. and Michelle Berger on the process of gaining access and qualitative research. American Journal of Evaluation. Thanks to Shirley for an informative review. 25. the purpose of evaluation. L-e. Finally. 261-272. The oral history of evaluation Part I. the factors the evaluator used to organize and frame their work. The Oral History Project Team (2003). (2002). (2002). American Journal of Evaluation. 23. 187-197. The case of the uncertain bridge. Jeannine Bell. L. American Journal of Evaluation. 23. 183185. 199-206. Shirley Copeland reviews a recent book by Martha Feldman. The oral history of evaluation Part II. American Journal of Evaluation. allowing her to examine similarities and differences across a set of evaluators in terms of such important characteristics as preferred evaluation role. References Datta. 24. An analysis of ethical challenges in evaluation. after too long a delay. she treats the interviews from Exemplars as a set of case studies. Jody Fitzpatrick has provided a fascinating piece. 243253. the nature of stakeholder involvement. N. M. Reflections on the chance to work with great people: An interview with William Shadish. Morris. the Book Review section reappears. American Journal of Evaluation. An interview with Lois-ellin Datta. Ethical challenges. (2002). Journal of MultiDisciplinary Evaluation: JMDE(2) 149 . 23.Global Review: Publications was the case with the Datta and Smith reflections. and method choices. In essence. The Oral History Project Team (2003).
Peersman. and selective use of evaluation findings. The three main critiques Patton offers are: 1) the sense that the authors are overwhelmed by numbers and fail to include stories of real people affected by HIV/AIDS. While this issue deals mostly with subjects specific to HIV/AIDS prevention and treatment it does offer some insight into evaluation questions with a wider impact. The Fall 2004 issue (Rugg. and 3) the acceptance of unrealistic goals. such as the denial of problems despite compelling evidence. program improvement.Global Review: Publications New Directions for Evaluation John S. He argues for including stories of real Journal of MultiDisciplinary Evaluation: JMDE(2) 150 . 2) the “deeply entrenched mechanistic linearity” (p. 168) in evaluation. the use of evaluation for accountability vs.” Patton identifies issues touched on by the various authors that are seen in many evaluation contexts.” The Fall issue covers a wide range of topics in HIV/AIDS monitoring and evaluation including political influences. Risley The two most recent issues of New Directions for Evaluation each cover international perspectives in the field. and Carael) addressed “Global Advances in HIV/AIDS Monitoring and Evaluation” while the Winter 2004 issue (Russon and Russon) concerned “International Perspectives on Evaluation Standards. These questions are identified nicely by Michael Quinn Patton in his overview chapter “A Microcosm of the Global Challenges Facing the Field: Commentary on HIV/AIDS Monitoring and Evaluation. and specific program evaluation experiences. international perspectives focusing on the roles of the United Nations and the World Bank.
he says overly optimistic goals.” (Russon.” (p. Craig Russon (co-editor of the issue with Gabrielle Russon) provides an overview of the development of national. acting as either a “point of departure” or as an example of what some national and regional groups “did not want their standards to be. Western Europe. Africa. He notes that the Joint Committee Standards were influential on all standards that followed. 90) One such instance is addressed by Doug Fraser in his review of the experience of the Australasian Evaluation Society’s (AES) ongoing process of developing a policy on standards. 37) Patton cites Uganda and Brazil as two successful cases of countries greatly reducing their HIV/AIDS infection rates through “complex.Global Review: Publications people along with the reporting of data so that the data doesn’t take on “an abstract life of their own.” (p. The Winter issue reviews the development of evaluation standards in the United States. Specifically. should be questioned by evaluators.168) He criticizes the “input-activities-output-outcomeimpact” framework presented in one chapter as the “basic organizing framework” endorsed by all agencies “to organize the data required to monitor program progress. dynamic systems change.” (p.and regional-level evaluation standards in the years since the Joint Committee’s Program Evaluation Standards were adopted in 1994. like those set by the United Nations regarding HIV/AIDS. 169) Patton also contends that evaluators should not merely accept the program goals when evaluating a program. Fraser recounts how the Joint Committee Standards were the starting points but they “depended on a number of fundamental preconditions or assumptions that did not necessarily hold true” in the environment of Australia and Journal of MultiDisciplinary Evaluation: JMDE(2) 151 .” (p. Australasia. p. 169) In situations such as these “complex systems change mapping and networking models hold more promise than do traditional linear-logic models. and at some large international nongovernmental organizations.
(p. These risks and threats concern how evaluation is managed. G.Global Review: Publications New Zealand. not simply practicing evaluators. New Directions for Evaluation. M. 71) The Program Evaluation Standards concentrated on “risks that were internal to the evaluation itself: risks of evaluators’ overreaching themselves. & Russon.). Russon. 77) Fraser notes the prominence of transparency in this draft as contrasted with the Joint Committee Standards. and accuracy/quality/comprehensiveness. cost-effectiveness. Journal of MultiDisciplinary Evaluation: JMDE(2) 152 . an Ethics and Standards Committee did prepare a draft set of standards for the society’s 2001 conference. (2004). behaving unethically. & Carael. This draft included six categories: transparency. International perspectives on evaluation standards. supported and used.). C. overlooking key aspects of their task. (p. therefore any standards should address these audiences. (2004). exercising bias. 104. or failing to apply an appropriate range and quality of techniques. (Eds.” (p. Peersman. practicality. (Eds. planned. utility.. Many of these issues are controlled by those who fund and use evaluation. New Directions for Evaluation. Global advances in HIV/AIDS monitoring and evaluation. 71) AES members saw the risks and threats they wished to address as being external to the process of evaluation. G. Fraser recounts how the AES has long had a practitioner code of ethics but the process of developing a set of standards for evaluation stalled in 2001 owing to many factors. ethics. 103.. References Rugg. D. However.
Regional. Education. a library.Global Review: Publications Education Update Nadini Persaud Websites The World of Education http://www. economy. Government. plans and curricula. Health. dictionaries) and a Reading Room (books. communications. Journal Articles Journal of Teacher Education (Volume 56: 2005 and Volume 55: 2004) has a number of interesting articles including: Journal of MultiDisciplinary Evaluation: JMDE(2) 153 . people. The Internet Public Library Website provides links to subject areas in Arts and Humanities. calendars. Once a country is selected. transportation. can be found. government. The “forum link” directs the viewer to the Education American Network and Education Canada Network where various workshops on teacher-to-teacher and lessons. The “world facts” link is particularly informative. magazines and newspapers). a web directory and a bookstore. It also provides links to a Ready Reference database (almanacs. military and transnational issues for each country. Entertainment.net/ user friendly website provides links to jobs in education. education forums. Science and Technology and Social Sciences. Business. the visitor can get access to a country map and brief profiles on the geography. it provides a database including every country on the globe. world facts.educationworld. Computers. The “regional link” directs the viewer to databases on history and travel and tourism by Continents/Region.
Tabitha Garvin. that she realized that Journal of MultiDisciplinary Evaluation: JMDE(2) 154 . The authors conclude that a shared vision about the role of technology in teacher education has not yet emerged in the field of education. Implicit in this model for technological change is a strategy for sustainability. Bryan Waite. She explains that as a result of the unsatisfactory pass rate on the PRAXIS II test. Sutton (2004) notes that it was only after taking the test. Teaching Under High-Stakes Testing: Dilemmas And Decisions Of A Teacher Educator by Rosemary E. Sutton The article “Teaching Under High-Stakes Testing: Dilemmas And Decisions Of A Teacher Educator” reviews how an experienced teacher was forced to change her teaching strategies as a result of the introduction of the PRAXIS II: Principles of Learning and Teaching (PLT) tests. Kirsten Anderson Meymaris. the authors present a model for technological change and also describe a critical framework to facilitate discourse among education faculty from which understandings of why. and Carolyn Mears [PDF] The article “Integrating Technology Into Teacher Education: A Critical Framework For Implementing Reform” discusses the challenges of integrating technology into teacher education. Dominic Peressini. many of her students failed the PRAXIS II test when it was first introduced. Michelle Reidel.Global Review: Publications Integrating Technology into Teacher Education: A Critical Framework for Implementing Reform by Valerie Otero. the Dean’s Office in the College of Education and Human Services encouraged the faculty to take the tests. Danielle Harlow. teachers must be skilled in technology applications and knowledgeable about using technology in order to enhance and extend student learning. which are now mandated in Ohio. because they were not good standardized test takers. when and how to use technology emerge. According to Sutton (2004). In this article. Pamela Ford. According to the authors.
Scott Ridley.Global Review: Publications she would have to alter her method of assessment. Sally Hurwitz. in her education psychology courses to prepare students for the PRAXIS II tests. Now instructors meet regularly to choose a common text book. content and teaching strategies. Varghese and Tom Stritikus [PDF] Comparing PDS and Campus-Based Preservice Teacher Preparation: Is PDSBased Preparation Really Better? by D. Price and Linda Valli [PDF] "Nadie Me Dijó [Nobody Told Me]": Language Policy Negotiation and Implications for Teacher Education by Manka M. share resources and discuss topics that should be incorporated into the educational psychology courses offered by the University. and Kari Knutson Miller [PDF]. Journal of MultiDisciplinary Evaluation: JMDE(2) 155 . Mary Ruth Davis Hackett. Other interesting articles include Taking Stock in 2005: Getting Beyond the Horse Race by Marilyn CochranSmith [PDF] The Effect of Perceived Learner Advantages on Teachers' Beliefs About Critical-Thinking Activities by Edward Warburton and Bruce Torff [PDF] Shifting from Developmental to Postmodern Practices in Early Childhood Teacher Education by Sharon Ryan and Susan Grieshaber [PDF] Preservice Teachers Becoming Agents of Change: Pedagogical Implications for Action Research by Jeremy N. She concludes by noting that the implementation of PRAXIS II increased collaboration among faculty teaching education psychology at her University.
Journal of Teacher Education. B. Meymaris. P. Reidel. (2005). Garvin. 56(1): 8-23. Mears.. K. Journal of MultiDisciplinary Evaluation: JMDE(2) 156 . D. T. Waite. Harlow. Sutton.. R. 55(5): 463-475. Peressini... C.. D..Global Review: Publications References Ford. (2004). A. V. Integrating technology into teacher education: A critical framework for implementing reform. M.. Journal of Teacher Education.. Teaching under high-stakes testing: Dilemmas and decisions of a teacher educator. E. Otero.
school. and foundations with research and information to guide them as they fund new strategies and strengthen existing initiatives HFRP strives to reach its goals through providing: Journal of MultiDisciplinary Evaluation: JMDE(2) 157 . and outcomes measurement and attainment through evaluation practices • Expand and strengthen the professional development base of those who work directly with children and families • Provide policymakers. and practitioners by collecting.Global Review: Publications The Evaluation Exchange—Harvard Family Research Project Brandon W. practitioners. and synthesizing research and information. The project has aided philanthropies. and offer professional development to those who work with children and/or their families. Youker Harvard Family Research Project (HFRP) was founded by the Harvard Graduate School of Education in 1983. and community partnerships of early childhood care and education. test. program and system complexity. policymakers. promote evaluation and accountability. HFRP’s Goals: • Develop. analyzing. and communicate methods that promote continuous improvement and accountability • Promote diversity. The HFRP aims to help strengthen family.
Kellogg Foundation • John D. MacArthur Foundation • The Charles Steward Mott Foundation • The Pew Charitable Trusts • The Rockefeller Foundation Journal of MultiDisciplinary Evaluation: JMDE(2) 158 .K.Global Review: Publications • Knowledge Development • Training and Professional Development • Technical Assistance • Continuous Learning and Dialogue HFRP has two categories for research: a) Family-school-community partnerships b) Strategy consulting and evaluation HFRP-partial list of funders: • Carnegie Corporation of New York • The Annie E. Casey Foundation • The Ford Foundation • The Heinz Endowments • The W. & Catherine T.
funders and policymakers. The journal is divided in to 5 sections. program practitioners. The Evaluation Exchange.” • “Parental Involvement and Secondary School Student Educational Outcomes: A Meta-Analysis. (4) Evaluations to Watch. No. addresses issues that program evaluations frequently encounter.Global Review: Publications HFRP has an evaluation periodical. and practical applications of evaluation theory. (1) Theory & Practice. and (5) Beyond Basic Training. It is designed as an ongoing discussion medium among evaluators. Winter 2004/2005): • “Improving Parental Involvement: Evaluating Treatment Effects in the Fast Track Program. The journal.” • “Ongoing Evaluations of Programs in Parent Leadership and Family Involvement. • “What Matters in Family Support Evaluation?” • “Learning from Parents Through Reflective Evaluation Practice. (2) Promising Practices. The Evaluation Exchange emphasizes innovative methods and approaches to evaluation. (3) Spotlight. Journal subscriptions are free and contributions are encouraged.4. Examples of evaluation-related articles in the most recent journal publication (Volume X.” • “Promoting Quality Outcome Measurement: A Home-Visitation Case. published 3 or 4 times a year. emerging trends in practice.” • “Blending Evaluation Traditions: The Talent Development Model.” Past journal issues of particular relevance to evaluators: 159 Journal of MultiDisciplinary Evaluation: JMDE(2) .
1. 3. 1998 Vol. No. IX. Winter 03/04 Vol. IV. X. 4. X. No. X. 2. No. 2. Summer 04 Vol. 1995 “Harnessing Technology for Evaluation” “Early Childhood Programs and Evaluation” “Evaluating Out-of-School Time Program Quality” “Reflecting on the Past and Future of Evaluation” “Evaluation in the 21st Century” “Participatory Evaluations” Journal of MultiDisciplinary Evaluation: JMDE(2) 160 . Spring 04 Vol. Fall 2004 Vol. 1.Global Review: Publications Vol. No. No. No. 2.
Fall 2004 Chris L. Coryn The Canadian Journal of Program Evaluation's (CJPE) most recent issue contains 7 articles (5 in English and 2 in French). Following House's contribution is Using Multi-Site Core Evaluation to Provide "Scientific" Evidence by Frances Lawrenz and Douglas Huffman. Lawrenz and Huffman argue that the prevalent standards of "scientific" evidence (such as the U. that scientific rigor can be maintained while including other evaluation approaches (in this case a participatory. collaborative approach) and purposes (to not only evaluate "what" happened. a research and practice note. and loosely based upon his keynote speech from last year's CES conference (Evaluation 2004) in Saskatoon. through a case study. for example). RCTs) have not been proven superior to other approaches.Global Review: Publications Canadian Journal of Program Evaluation. S. Canada. The authors demonstrate. highly variable values and interest.S. but "how").e. Volume 19(2). and 3 book reviews. 18)—i. Journal of MultiDisciplinary Evaluation: JMDE(2) 161 . House focuses his attention on the political struggles faced by evaluators under the current Bush administration in the United States as well as the difficulties in balancing often conflicting.. Department of Education's priority on "scientifically-based evaluation methods" (p. The first article in this issue—The Role of the Evaluator in a Political World—is by Ernie House. The authors describe how to incorporate other evaluative purposes into evaluations in which the client's central concern is effectiveness (usually goal-based evaluation requiring experimental designs.
J. 99) in their article entitled Integrating Evaluative Inquiry into the Organizational Culture: A Review and Synthesis of the Knowledge Base. The author found that fewer than half of these research articles utilized systematic analytic techniques (emergent or pre-ordinate) in assessing focus group transcripts. Goh. Journal of MultiDisciplinary Evaluation: JMDE(2) 162 . The implications for future research efforts and practice are also discussed. L’évaluation des Technologies de la Santé: Comment L’introduire dans les Hôpitaux Universitaires du Québec? by Oliver Sossa and Pascale Lehoux describes the implementation of health technology in Quebec university teaching health centers and the structures that facilitate its development. Hurteau uses this article to explore the “current preoccupation in evaluation literature: the impact of organizational context on the evaluation process” (p. Le Benchmarking Et L’amélioration Continue by Marthe Hurteau describes the benchmarking process and its use to identify relevant performance indicators. In The Analysis of Focus Groups in Published Research Articles Geoffrey S. and Linda E. Lee explore the “conceptual interconnections and linkages among developments in the domains of evaluation utilization. Shannon Clark. Swee C. evaluation capacity building.Global Review: Publications Mary Sehl’s piece entitled Stakeholder Involvement in a Government-Funded Outcome Evaluation: Lessons Learned from the Front Line describes involving project stakeholders in the planning and decision making process and the strengths and limitations associated with this approach. 57). Bradley Cousins. and organizational learning” (p. Wiggins critically assesses the use of analytic methods employed in published literature from several disciplines. Even fewer utilize measures of reliability when analyzing transcripts.
Carden.. Anderson. Journal of MultiDisciplinary Evaluation: JMDE(2) 163 . The first is Organizational Assessment: A Framework for Improving Performance (Lusthaus. Nichols frames her discussion of documenting training outcomes in learning and behavior using two case study examples. 2003) is reviewed by Jennifer Carey. Morabito.and Post-Scenarios: Assessing Learning and Behavior Outcomes in Training Settings. The final section of this issue of CJPE is devoted to reviews of three books. The final review. 2002) reviewed by Stephen M. Adrien.Global Review: Publications The Research and Practice Note in this issue of CJPE is a piece by Allison Nichols. The second. Evaluating Social Programs and Problems: Visions for the New Millennium (Donaldson & Scriven. & Montalvan. entitled Pre. of the book entitled Institutionalizing IMPACT Orientation: Building a Performance Management Approach that Enhances the Impact Orientation of Research Organizations (Smith & Sutherland. Eds. 2002) is reviewed by Ronald Mackay.
five uses of evaluation—instrumental. legitimating. Schröter The most recent issue of Evaluation contains six articles. Widmer and Neuenschwander discuss in their article—Embedding Evaluation in the Swiss Federal Administration: Purpose. conceptual. Widmer and Neuenschwander demonstrate how evaluation is embedded within the different federal agencies of the Swiss government. October 2004 Daniela C.Global Review: Publications Evaluation: The International Journal of Theory. and tactical. interactive. followed by legitimizing and interactive uses. (ii) evaluation findings were most commonly utilized instrumentally. Research and Practice. and Utilization— how evaluation is embedded within the Swiss political system and conclude that currently used evaluation measures in the government can be improved through purposeful differentiation of evaluation types. improvement. and strategy. and News from the Community respectively. Key findings of their study include that (i) accountability and improvement were the most relevant purposes in these organizational contexts. They first summarize four purposes of evaluation—accountability. Institutional Design. basic knowledge. secondly. one contribution to A Visit to the World of Practice. Volume 10(4). (iii) the institutional design was of little or no relevance. and thirdly. In the second article—Utilizing Evaluation Evidence to Enhance Professional Practice—Helen Simons criticizes the current politically favored approach to Journal of MultiDisciplinary Evaluation: JMDE(2) 164 . Thereafter. and (iv) unanticipated blends of purpose and utilization existed. two predominant institutional designs in which evaluation is implemented—centralized and decentralized.
Global Review: Publications
evaluation, namely evidence-based evaluation. She states that this approach to elucidating evidence “fails to recognize the holistic nature of professional practice and disregards the complexity of professional decision making and action” (p.410). Qualitative forms of knowledge generation for evaluative purposes would enhance the quality of evaluation and increase the utilization of evaluation findings. In The Meaning Assigned to Evaluation by Project Staff: Analysis from the Project-management Perspective in the Field of Social Welfare and Healthcare in Finland, Seppänen-Järvelä examines how evaluation is understood by project staff and management and how it influences the work environment. Seppänen-Järvelä concludes that there is a need to update the current knowledge of project staff and management about evaluation and to promote and enforce evaluation culture and capacity building within organizations in the Social Welfare and Healthcare sector in Finland. Oakley, Strange, Stephenson, Forrest, and Monteiro’s article—Evaluating Processes: A Case Study of a Randomized Controlled Trial of Sex Education— exemplifies the application of RCTs to evaluate how processes and outcomes are interrelated. The authors conclude that ultimately the choice of design and quantitative or qualitative approaches is context-dependent and related to the questions asked. Following is McNamara and O’Hara’s article Trusting the Teacher: Evaluating Educational Innovation, in which the authors claim that the role of the external evaluator should be switched to that of the educating consultant for the teacher. The case for self-evaluation of the practitioner is supported by the argument that external evaluation would often fail to support improvement of the evaluand. To support “sound educational values,” (p. 472) evaluators should function as
Journal of MultiDisciplinary Evaluation: JMDE(2)
Global Review: Publications
facilitators and consultants in conducting the research and enhance the credibility of self-evaluation by meta-evaluating the internal self-evaluation processes. The last article, authored by Hanberger and Schild, discusses Strategies to Evaluate a University-Industry Knowledge-exchange Programme. The authors consider two management-oriented approaches to program evaluation (program theory evaluation and outcome analysis) and two non-management oriented approaches (policy discourse analysis and qualitative network analysis). They conclude that different evaluation methods stress the values of different stakeholder groups. An integration of various methods is necessary, reduces the bias toward one stakeholder group, and increases the validity in contexts where multiple stakeholder groups are present. In situations with only one target group and few stakeholders, a combination of various evaluation approaches would not be as essential. In A Visit to the World of Practice, Farrall and Gadd address Evaluating Crime Fears: A Research Note on a Pilot Study to Improve the Measurement of the ‘Fear of Crime’ as a Performance Indicator. Instruments intended to assess fear of crime are criticized for their poor design and neglect of crucial research concerns such as frequency and intensity. The authors suggest survey questions to be incorporated to improve instruments measuring fear of crime. In News from the Community, Nicoletta Stame, President of the EES, reports on the 6th EES conference entitled Governance, Democracy and Evaluation. These issues include: (i) evaluation as a tool for democratic government, (ii) the question of an European evaluation identity, (iii) European standards for evaluation, (iv) relationships among evaluation networks and associations, and (v) training, education, and professional development of evaluation in Europe. For example, the
Journal of MultiDisciplinary Evaluation: JMDE(2)
Global Review: Publications
EES considers the establishment of European evaluation standards but does not want to conflict or overthrow features unique to individual national characteristics. The journal concludes with translations of the article’s abstracts into French and an Annual Index of articles.
Journal of MultiDisciplinary Evaluation: JMDE(2)
provocative focus article followed by commentaries and a rejoinder article. The focus article of the inaugural issue is On the Structure of Educational Assessments by Robert J. ethnography. 2003 Chris L. Almond. debating a “focus paper” or participating in a commentary. The inaugural issue—Volume 1(1). This framework. Belgium. psychology. Mislevy. Each issue is devoted to a single. for example. This article describes a framework for assessment that makes explicit the interrelations among substantive arguments. economics. called “evidence-centered” assessment design (ECD) Journal of MultiDisciplinary Evaluation: JMDE(2) 168 . policy studies. Steinberg. and certification testing. Berkeley.Global Review: Publications Measurement: Interdisciplinary Research and Perspectives. history. U. assessment designs. Coryn Measurement: Interdisciplinary Research and Perspectives is a relatively new journal devoted to the interdisciplinary study of measurement in the human sciences and is intended to represent a broad range of disciplines and perspectives including psychometrics. Linda S. standards-based testing. 2003—was sent to us by the editors (Mark Wilson at the University of California.edu/measurement/. sociology. and law. and Pamela Moss at the University of Michigan) to encourage becoming involved in for example.berkeley. Leuven. Further information can be found at http://bearcenter. and Russell G. Paul De Boeck at K. Volume 1(1). education. Presently eight issues are available covering objectivity and trust. and operational processes. linguistics. S. social theory.
domain modeling. proficiency. for example. (2) presentation. (3) evidence identification—task-level scoring. substantive experience and theory. 56). specifies the technical details necessary for implementing the assessment. information about the domain is used to organize beliefs. In the second stage. theories. and tasks. The final stage. the four-process delivery system. statistical models and task authoring schemas. specifications. and rubrics. the information gathered in stage 1 is organized into three paradigms. research. instructional materials. and the elements and processes of operational models” (p. consists of four principal components: (1) activity selection.” In the first stage of ECD design. (2) domain modeling. The authors illustrate their ideas with examples from language testing and the article is presented parallel to the stages of the ECD design process: (1) domain analysis. Steinberg. developing a conceptual assessment framework (CAF). or assessment design objects. domain analysis. into specifications that embody the substantive arguments that underlies an assessment” (p. 4).Global Review: Publications “…entails the development. Mislevy. and Almond’s intricate approach emphasizes measurement models which incorporate the relationship between “assessment purposes. The third task. and (4) evidence accumulation—test-level scoring. statistical models. operational requirements. Eight commentaries follow the focus article which range from the limitations of Bayesian models in assessment (Earl Hunt) to critiques and comments from a Journal of MultiDisciplinary Evaluation: JMDE(2) 169 . and exemplars. subject-matter expertise. and (4) operational assessment—the “four-process delivery system. construction. evidence. and arrangement of specialized information elements. (3) conceptual assessment framework.
Steinberg. Journal of MultiDisciplinary Evaluation: JMDE(2) 170 . constructivist and situative learning perspectives. Wagner). Glas) and finally to a framework for shifting from principle to practice (Richard K. and implementation. user and statistical models.Global Review: Publications psychometric perspective (Cees A. model generalizability. W. The issue concludes with Mislevy. and Almond’s rejoinder in which the authors address the themes of critique presented in the commentaries.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.