You are on page 1of 15

GUIDELINES AND CHECKLIST FOR CONSTRUCTIVIST (a.k.a.

FOURTH GENERATION) EVALUATION
Egon G. Guba & Yvonna S. Lincoln November 2001
1

NOTE: The guidelines and checklists for constructivist evaluations and reports outlined herein are based upon Egon G. Guba and Yvonna S. Lincoln, Fourth Generation Evaluation, Newbury Park, CA: Sage Publications, 1989. Useful background information may be found in Yvonna S. Lincoln and Egon G. Guba, Naturalistic Inquiry, Beverly Hills, CA: Sage Publications, 1985. EVALUATION DEFINED Evaluation is one of the three basic forms of disciplined inquiry, the others being research and policy analysis. It is that form of inquiry whose focus is some evaluand (program, process, organization, person, etc.) and which results in “merit” and/or “worth” constructions (judgments) about it. Merit constructions converge on the intrinsic quality of an evaluand, irrespective of the setting in which it may find applications. Worth constructions converge on the extrinsic usefulness or applicability of an evaluand in a concrete local setting. Evaluation of a proposed or developing evaluand is termed “formative,” while evaluation of some developed evaluand is termed “summative.” CONSTRUCTIVIST EVALUATION DEFINED Constructivist evaluation is that form of evaluation based on the propositions (basic assumptions) undergirding the constructivist paradigm. The constructivist paradigm differs from other knowledge paradigms commonly in use, including the scientific, the artistic, the religious, the legal, and others of similar broad sweep. It is based on three fundamental assumptions, which are commonly termed the ontological, epistemological, and methodological, viz: The basic ontological assumption of constructivism is relativism, that is, that human (semiotic) sense-making that organizes experience so as to render it into apparently comprehensible, understandable, and explainable form, is an act of construal and is independent of any foundational reality. Under relativism there can be no “objective” truth. This observation should not be taken as an “anything goes” position; see the section on criteria below. The basic epistemological assumption of constructivism is transactional subjectivism, that is, that assertions about “reality” and “truth” depend solely on the meaning sets (information) and degree of sophistication available to the individuals and audiences engaged in forming those assertions. The basic methodological assumption of constructivism is hermeneutic-dialecticism, that is, a process by which constructions entertained by the several involved individuals and groups (stakeholders) are first uncovered and plumbed for meaning and then confronted, compared, and contrasted in encounter situations. The first of these processes is the hermeneutic; the second is the dialectic. See sections on “discovery” and “assimilation” below. Note that this methodological assumption is silent on the subject of methods and, in particular, on the subject of “quantitative” vs. “qualitative” methods. Both types of methods may be and often are appropriate in all forms of evaluative inquiries. It is not appropriate to “mix and match” paradigms in conducting an evaluation, for example, utilizing both scientific (positivist) and constructivist propositions within the same study. This is not a call for “purity” nor

1

Egon G. Guba is Professor Emeritus, Indiana University; Yvonna S. Lincoln is Ruth Harrington Chair of Educational Leadership and University Distinguished Professor of Higher Education, Texas A&M University. Evaluation Checklists Project www.wmich.edu/evalctr/checklists

Authors of evaluation literature. work (explain what happens). Identify the full array of stakeholders who are at risk by virtue of the stakes they hold in the entity being evaluated. proof. In other cases the appearance of key concepts such as generalizability. disempowerment. The discovery phase may not be needed (or may be needed only minimally) if there is a preexisting construction or constructions relating to the evaluand on which to build (e.” the “here” being the evaluand and its context. depending on what specific relevant and preexisting constructions are brought to the inquiry by the evaluator and by local informants and respondents. replacing them) so that the “new” (more informed and sophisticated) construction will fit (subsume older and newer meanings. and in particular from the professional literature. opportunity. which in constructivist terms they do not deserve. CAVEAT: If the preexisting constructions are drawn from sources external to the subject evaluation. power. Discoveries are themselves semiotic organizations. i..g. and the like. Given that mandate. it is possible to list the nine major responsibilities that the constructivist evaluator must discharge.. serious disjunctions could easily be overlooked. drawing data from a study cast in positivist terms imbues those data with a truth value. or other coin. and it utilizes the methodology of the constructivist paradigm.e. but may overlap or be carried out in parallel. status. from a prior evaluation or from a project proposal). Discovery and assimilation are not necessarily sequential processes. typical of positivism and other nonconstructivist approaches. objectivity. including even those who may wish to maintain low visibility or to absent themselves entirely. if those bases are not constructivism. THE PROCESS OF CONSTRUCTIVIST EVALUATION: RESPONSIBILITIES OF THE CONSTRUCTIVIST EVALUATOR Constructivist evaluation is a process for doing evaluation that meets two conditions: It is organized by the claims. Within a constructivist framework those same data are seen as variable and transformable. Guidelines and Checklist for Constructivist (a. demonstrate relevance (enable the core problems to be resolved. There are many ways in which the discovery question can be answered. ameliorated. mental constructions. Negative stakes may include possible exploitation. that are based on constructivist principles will almost certainly make their intent plain. It is simply a caveat that mixing paradigms may well result in nonsense approaches and conclusions. those stakes are determined by and defined by the stakeholders (in their own terms) and not only by the evaluator or the client seeking the evaluation (although they too are stakeholders and may identify their own stakes and definitions). It is the evaluator’s responsibility to seek out all stakeholders. So. if the new discovery is sufficiently different from or in conflict with the existing construction or constructions. or better defined). He or she must: 1.is it intended to be exclusionary. that is. a hard and fast character. THE TWO PHASES OF CONSTRUCTIVIST EVALUATION: DISCOVERY AND ASSIMILATION The discovery phase of constructivist evaluation represents the evaluator’s effort to describe “what’s going on here. The assimilation phase of constructivist evaluation represents the evaluator’ s effort to incorporate new discoveries into the existing construction or constructions (or. the several stakes entering into the evaluation mix are assessed and refined in the effort to come as close as possible to negotiated agreement. and exhibit modifiability (be itself open to change). and disenfranchisement. some meanings (information) and some level of sophistication in their interpretation are already available. Such stakes may include but not be limited to money. and issues of stakeholding audiences.k. for example. care must be taken to assess their paradigmatic bases. Stakeholders are entitled to receive and evaluate in their own terms all information that the evaluation may disclose. may be key signals as to the intent of the author. depending on the view of the constructor.a Fourth Generation) Evaluation 2 . concerns. face. In the subsequent hermeneutic/dialectic process. To use those positivist data within a constructivist evaluation undermines the essence of the evaluation. including evaluation reports.

as possible. concerns. and different claims. can be understood. Because more information may be required than it is possible to obtain. and issues they wish to raise in relation to it. concerns. 5. or incomplete. and issues. Constructivist evaluations are never completed. since this form is neither widely known nor commonly understood. The most useful form for such report(s) is the case study. arranged by the evaluator. 7. The initial list may be rearranged. The process is first carried out within specific stakeholder groups. The evaluator’s task is to identify the information needed. A contract should be drawn that protects both the client from evaluator misrepresentation or malpractice and the evaluator from client misunderstanding or misexpectation.a Fourth Generation) Evaluation 3 . Prepare an agenda for negotiation on items about which there is no. it can be eliminated from further discussion. training must be provided. If consensus can be achieved with respect to an item. 3. Further. Provide a context and a methodology (the hermeneutic/dialectic) through which different constructions of the evaluand. in dialogic. but especially so for a constructivist evaluation. Recycle the evaluation to take up still unresolved constructions and their attendant claims. given time and/or resource constraints. adversarial. concerns. consensus. or have additions made as the evaluation proceeds. and issues. are reviewed in light of the new information and/or level of sophistication. or confrontational settings. Unresolved differences in constructions. but retained for further action (and inclusion in the evaluation report) if there is agreement on that action. 6. in the hope that their number can be reduced. Develop a report. Generate consensus with respect to as many constructions. deleted. and issues that they have raised (as well as those raised by other groups that appear relevant to that group). that communicate to each stakeholder group any consensus on constructions and any resolutions regarding the claims. New aspects may be explored that have emerged on the basis of the first-round evaluation. 8. Such a contract should cover the following points at a minimum (other stipulations may be added as seems appropriate in the actual situation).k. Outcomes of this forum must include action steps if the negotiation is to be regarded as successful. which disjunction(s) can be ameliorated only through the introduction of new information or an increase in the level of analytic sophistication. probably several targeted reports. subjected to critique. Collect and provide the information called for in the agenda for negotiation. concerns. The provision of needed information cannot be guaranteed. lest this need be taken as an opportunity to disempower selected stakeholders. Elicit from the stakeholder groups their constructions about the form and process of the evaluand and the range of claims. 4. which may provide the vicarious experience needed to influence stakeholder constructions. if necessary. but the evaluator must make every good faith effort to do so. and issues. and issues. later round of evaluation activity. Stakeholder inputs are essential in this determination. concerns. and their related claims. and taken into account. they pause until a further need or opportunity for review and reassessment emerges. Failure to reach consensus implies the continuation of competing constructions. if stakeholders lack the sophistication to deal with obtained information. Establish and mediate a forum of stakeholder representatives in which negotiation can take place. It is likely that some items will remain unresolved.) 9. and issues) are further negotiated in hermeneutic circles that cut across stakeholder groups. claims. concerns. as well as unresolved claims.2. Guidelines and Checklist for Constructivist (a. concerns. thereby setting the stage for another. then the products of these intragroup negotiations (defined constructions. the evaluator must devise some means (preferably also through a hermeneutic/dialectic process) for prioritizing the unresolved items. (See below for additional observations on the reporting process. Consensus should first be sought on an intragroup basis and then on an intergroup basis. CONTRACTING FOR A CONSTRUCTIVIST EVALUATION It is prudent to initiate a contract for an evaluation of any sort.

and challenging constructions of various stakeholders in ways that lead to reassessment and reconstruction. 8. and a listing of likely products. formative/worth. The case report is the preferred mode. 10. for example. Guidelines and Checklist for Constructivist (a. summative/worth. and psychotic or self-deluded personalities). Stakeholders in an evaluation are entitled to know who the client or sponsor is since that position clearly represents a priori power and vested interests.” These include a commitment from all parties to work from a position of integrity (always recalling that false commitments can be and sometimes are made). documents. and issues (including those of the client or sponsor) are to serve as the focus for the evaluation. The purposes of the case report include providing thick description. because they fear to make their positions known because of retaliation from more powerful groups. information. existing documents. Identification of the evaluand (the entity to be evaluated). Some stakeholders may refuse to participate. The contract should also note and make provision for an emergent evaluation design. A description of the reporting modes to be utilized. If stakeholder claims. residues from earlier actions such as legal cases. or some combination of these forms. or sophistication to deal with them. minimal competence on the part of all parties to communicate (a condition that may exclude children. Identification of the client or sponsor of the evaluation. other knowledgeable informants. 9.1. In those cases the evaluator must make every effort to construct their likely positions from whatever sources may be available.. 4. A statement of agreement from the client or sponsor particularly. A brief description of the methodology to be used. and the like. It should be noted that an evaluator does not enjoy special privilege as does. and the evaluator must take the responsibility to explicate reports for those stakeholders who may lack the background. The search for stakeholders should continue throughout the period of the evaluation. it is clearly necessary to identify. summative/merit. A statement of intent from the evaluator with respect to stakeholding audiences. giving vicarious experience. to adhere to the “Conditions for a Productive Hermeneutic/Dialectic. the mentally handicapped. since it is highly unlikely that all stakeholders will be known at the onset of the evaluation. 5. particularly since clients are likely to assume that an a priori and thereafter fixed design can be devised. The hermeneutic/dialectic methodology employed in constructivist evaluations clearly militates against that possibility. Reports must be freely available to all stakeholder groups. a willingness to consider change. a budget (at all stages a good faith “best estimate”). 7. seek out. A statement of the evaluator’s intent to guarantee confidentiality and anonymity of information sources insofar as that can be legally accomplished. concerns. The construction held by the client or sponsor is at best an initial form that is likely to evolve into a series or progression of constructions that emerge. but also from stakeholder groups.g. or physician. a tentative schedule (not a design). and a willingness to commit the time and energy needed. and involve all relevant stakeholders. for example. a willingness to share power. This guarantee must take account of legal protections where they exist and must provide for procedures to be followed in the event that access becomes blocked. 3. an attorney. serving as a metaphoric springboard (a form of naturalistic generalization).a Fourth Generation) Evaluation 4 . e. 2. A statement of purpose for the evaluation: formative/merit. clergyman. including the names and background of the agents who will carry out the evaluation (allowing for additions and deletions as may be required). A listing of technical specifications. 6. a willingness to reconsider one’s own value positions. A guarantee of access to records. and respondents.k.

these are set aside for later reconsideration in a subsequent recycling . THE USE OF THE HERMENEUTIC/DIALETIC METHODOLOGY The constructivist evaluation is carried out through a series of steps which. concerns. The result is a composite construction that includes all forms of the evaluand constructions as well as their relevant claims. 9. and other sources found to be relevant. making entree arrangements. 8. Preparing the agenda for negotiation by defining and elucidating competing constructions. The listing begins at the point at which a contract satisfactory to all parties has been agreed upon. setting these aside as possible case report components. concerns. and assessing local political/cultural factors. Reporting on the results for Step 9. identifying “beneficiaries” as well as “victims” of the evaluand’s action. Collecting additional information and adding sophistication in its use by training negotiators. formative/worth. summative/merit. and issues that emerge. claims. 2. assessing trade-offs and sanctions. Organizing the evaluation: Selecting the initial team of evaluators. Sorting out constructions. soliciting descriptions (constructions) of the evaluand and identifying and probing claims. culminating so far as possible in negotiated agreements on all identified. Enlarging joint intrastakeholder group constructions utilizing the evaluator’s prior construction (but allotting it no special privilege). 10.CONDUCTING THE CONSTRUCTIVIST EVALUATION. 3. It is virtually certain that some items will not have been negotiated to the satisfaction of all stakeholder groups. interplay of in-group interview data with observational data. Agreements on elements of these reports may lead to proposed action steps. and issues. literature analects. making logistical arrangements. Developing intrastakeholder group constructions: forming multiple hermeneutic circles of 1012 members each representing one stakeholder audience. 5. 6. 1. Prioritizing unresolved items via a negotiated prioritizing process determined by and involving the stakeholder group members. Step 8 will have resulted in a negotiated agenda for each of the several stakeholder groups. formative/merit. 11. There may be several reports tailored to the claims. may well be iterative and reiterative in practice as constructions evolve and as particular claims. concerns. supporting. concerns. Identifying stakeholders: Identifying agents commissioning and carrying out the evaluand. and issues are dealt with. The serial form below is used as a matter of convenience. while listed here in serial form. or refuting items (providing additional training as needed). and/or summative worth.k. Guidelines and Checklist for Constructivist (a. The report should be aimed particularly at the stipulated purpose(s) of the evaluation. and formalizing agreements with and among them. 7. mounting continuing search strategies for other stakeholders. working at illuminating. and testing the agenda derived. and issues of specific stakeholder groups. and issues resolved by consensus.a Fourth Generation) Evaluation 5 . concerns. performing special studies as needed. seeking new information. Recycle the entire process to take particular account of elements set aside in step 9 that were irresolvable at that time. This step 9 effectively recapitulates steps 3-8 for a newly formed hermeneutic circle consisting of persons selected by the individual circles as their representatives. 4. that is. existing documentary information. Developing intergroup constructions.

In a sense. 233-43): a. roughly parallel to internal validity.e. i. and objectivity. more sophisticated ways of analysis and interpretation until some level of consensus is reached. who examines the record of the Guidelines and Checklist for Constructivist (a. persistent observation. a case study is never finished. The case report is characterized by a thick description that not only clarifies the allimportant context but that makes it possible for the reader to experience it vicariously. pp. and interpretations with members of stakeholding audiences. targeted to specific stakeholder audiences. or recommendations except insofar as these are concurred on by relevant respondent stakeholders. reliability. and rationales leading to those beliefs. roughly parallel to external validity. for example. There may be multiple reports. They are probably most useful. roughly parallel to reliability. However. The case report helps the reader realize (in the sense of making real). internal and external validity.. but also the underlying motives. These parallel criteria are (full definitions can be found in Fourth Generation Evaluation. These evolved from an effort to produce criteria more or less parallel to those conventionally used. feelings. They are exposed to new information and new. not only the states of affairs that are believed by stakeholders to exist. in guiding methodological decisions during the evaluation and later in auditing the overall evaluation process (see c and d below). are inappropriate for constructivist evaluations precisely because they are based upon a fundamentally different theoretical paradigm (as explained in the opening paragraphs of this statement). since it is subject to successive iterations) is the case report. CRITERIA FOR ASSESSING THE QUALITY OF CONSTRUCTIVIST EVALUATIONS AND REPORTS Standards normally applied in making quality judgments of evaluations. contain an appendix that describes in detail the methodology followed and makes it possible to judge the extent to which quality criteria (those listed in the following section) are met. the Joint Committee Standards or the Guiding Principles for Evaluators of the American Evaluation Association. established not by the evaluator but by receivers of evaluation reports who make personal judgments of the degree to which findings are sufficiently similar to their own situations (judged from the thick description) to warrant testing for the viability of local application (testing for localization rather than the more usual generalization). c. if such a report is beyond the competence of a stakeholding audience to deal with. and they may take many forms. their very “parallelism” to positivist tenets renders them less than fully adequate for determining the quality of a constructivist approach. it is merely due. continuous testing of hypotheses. Credibility. Instead. Dependability. finally. Transferability. peer debriefing (a kind of external critic). established through the use of the dependability audit with the assistance of an external auditor. Throughout this process the stakeholders—individually. data. and across groups— are chosen to uncover widely variable viewpoints. first. the case report is the joint construction that emerges as the result of the hermeneutic/dialectic process. both are useful during the evaluation process as procedural checklists and afterward in assessing the completed evaluation report (product) for quality: 1. negative case analysis (a process of reworking postulated hypotheses). established by prolonged engagement at the site. The report does not culminate in judgments.CONSTRUCTIVIST EVALUATION REPORTS The end product of a constructivist evaluation (but never a final product. Two different approaches have been generated to deal with this dilemma. The “parallel” criteria (sometimes called “trustworthiness” or “foundational” criteria). preliminary categories. conclusions. b. progressive subjectivity (continuous checking of developing constructions against records of constructions that were expected prior to data collection).a Fourth Generation) Evaluation 6 . possibly not including what might normally be termed a “technical” report. in similar groups. and (most important) member checks. The case report must.k.

revision. determined by an assessment of the extent to which all competing constructions have been accessed. and data can be traced to their sources. sharpening values) is stimulated and facilitated by the evaluation. some of which may be highly technical. correction. Guidelines and Checklist for Constructivist (a. 2. concerns. thus. the ability of the hermeneutic/dialectic process to act as a powerful source of quality control should not be overlooked. Catalytic authenticity. that is. elaboration. The required balance between these roles is delicate. and the evaluator will need to exercise great care to avoid bias and favoritism. the representation) they need. 245-250): a. exposed. Confirmability. and issues they experience. Fairness. it is necessary for the evaluator to play a dual (and sometimes conflicting) role: advocate and educator. be incorporated into the joint. and taken into account in the evaluation report. The authenticity criteria. d. It is difficult to maintain false fronts or support deliberate deception when information is subject to continuous and multiple challenges from a variety of stakeholders. Opportunities for errors to go undetected and/or challenged are very small indeed under those circumstances. the inspection being done by an external auditor (who may be the same or different from the dependability auditor). In this process. In virtually every situation the stakeholding audiences will differ greatly in the amount of information they bring to the table. to determine the methodological decisions made and to understand the reasons for them. b. e. Ontological authenticity. furthermore. pp. And finally. moving to eliminate or ameliorate problems.a Fourth Generation) Evaluation 7 . assertions. Educative authenticity. determined by an assessment of the extent to which action (clarifying the focus at issue. determined by an assessment of the extent to which individual constructions (including those of the evaluator) have become more informed and sophisticated. determined by an assessment of the extent to which individuals (including the evaluator) have become more understanding (even if not more tolerant) of the constructions of others.inquiry in the way a fiscal auditor examines fiscal records. collaborative reconstruction that emerges as the process continues. The publicly inspectable and inspected nature of the hermeneutic/dialectic process itself prevents much of the kinds of secrecy and information poverty that have characterized client-focused evaluations. and the degree of sophistication they possess in processing new information that emerges. expansion. facts.k. it is incumbent on the evaluator to provide the training (and if necessary. Furthermore. determined by an assessment of the extent to which individuals are empowered to take the action that the evaluation implies or proposes. The “raw products” and the “processes used to compress them” are inspected and confirmed as appropriate. Those inputs will. Whereas the parallel criteria are embedded in the assumptions of positivism. Tactical authenticity. the degree to which they can articulate their existing constructions of the evaluand and the claims. d. c. data inputs are analyzed immediately on receipt. any intent on the part of the evaluator to favor particular stakeholders is at least equally detectable. which determines the extent to which constructions. for a quality evaluation to result. roughly parallel to objectivity. First. or emendation to the very respondents who provided them just a moment before. They are “fed back” for comment. Second. in the negotiated emergent construction. the hermeneutic/dialectic process itself is not one in which they are well versed. the authenticity criteria are based directly on the assumptions of constructivism and are responsive to the hermeneutic/dialectic aspects of that paradigm. These criteria are (full definitions can be found in Fourth Generation Evaluation. Two other observations are appropriate in respect to the quality question. It is the immediate and continuing interplay of information that militates against the possibility of noncredible outcomes.

that is. that human (semiotic) sense-making that organizes experience so as to render it into apparently comprehensible. is an act of construal and is independent of any foundational reality. process. that assertions about “reality” and “truth” depend solely on the meaning sets (information) and degree of sophistication available to the individuals and audiences engaged in forming those assertions.k. It is this fourth generation form of evaluation that is the subject of this checklist and set of guidelines.” It requires the evaluator to play multiple roles which at times may appear to be in conflict. that is. Yet from a value-oriented view. It denies the possibility of reliable generalizations and of determining solutions “that work” everywhere. In Fourth Generation Evaluation we described the historical evolution of evaluation practice: a first generation focused on measurement. It is one of the more realistic and socially—and politically—sensitive approaches to performing useful—and utilized—evaluations.) and which results in “merit” and/or “worth” constructions (judgments) about it. compared.a Fourth Generation) Evaluation 8 . This observation should not be taken as an “anything goes” position. the others being research and policy analysis. and an overcommitment to the scientific (positivist) paradigm of inquiry.CODA Constructivist evaluation differs in fundamental ways from other forms of evaluation. which are commonly termed the ontological. it is. a process by which constructions entertained by the several involved individuals and groups (stakeholders) are first uncovered and plumbed for meaning and then confronted. Worth constructions converge on the extrinsic usefulness or applicability of an evaluand in a concrete local setting. a second generation focused on description. concerns. see the section on criteria below. and a fourth generation focused on negotiation (the hermeneutic/dialectic). and explainable form. and disenfranchising to selected stakeholders. etc. and exhibit continuing modifiability. organization. person. and others of similar broad sweep. an evaluation approach that favors the point of view of the client or funder.” CONSTRUCTIVIST EVALUATION DEFINED Constructivist evaluation is that form of evaluation based on the propositions (basic assumptions) undergirding the constructivist paradigm. The basic methodological assumption of constructivism is hermeneutic-dialecticism. a failure to accommodate value-pluralism. It is highly labor intensive. The first of these processes is the hermeneutic. and issues widely felt and to the formulation of constructions widely seen to fit. Merit constructions converge on the intrinsic quality of an evaluand. its personnel and resource commitments can at best be “guesstimated. the best way to evolve viable and acceptable solutions to claims. understandable. the second is Guidelines and Checklist for Constructivist (a. the legal. It is that form of inquiry whose focus is some evaluand (program. irrespective of the setting in which it may find applications. We believe this form obviates the major problems of the first three generations: a tendency toward managerialism. The constructivist paradigm differs from other knowledge paradigms commonly in use. and contrasted in encounter situations. a third generation focused on judgment. and that is disempowering. that inappropriately saves the manager harmless. work. that is. demonstrate relevance. Under relativism there can be no “objective” truth. unfair. including the scientific. EVALUATION DEFINED Evaluation is one of the three basic forms of disciplined inquiry. If is often adversarial and confrontational. the religious. Evaluation of a proposed or developing evaluand is termed “formative.” while evaluation of some developed evaluand is termed “summative. that is. Constructivist evaluation is a difficult model to adopt. It is ever-recursive and requires frequent recapitulations. It is a diffuse process impossible to specify in detail (in design form). and methodological. hence. The basic epistemological assumption of constructivism is transactional subjectivism. we think. of which there are many. the artistic. It is based on three fundamental assumptions. now dubbed constructivist evaluation. viz: The basic ontological assumption of constructivism is relativism. epistemological.

opportunity. replacing them) so that the “new” (more informed and sophisticated) construction will fit (subsume older and newer meanings. i. drawing data from a study cast in positivist terms imbues those data with a truth value. and in particular from the professional literature. face. or other coin. Stakeholders are entitled to receive and Guidelines and Checklist for Constructivist (a. THE TWO PHASES OF CONSTRUCTIVIST EVALUATION: DISCOVERY AND ASSIMILATION The discovery phase of constructivist evaluation represents the evaluator’s effort to describe “what’s going on here. To use those positivist data within a constructivist evaluation undermines the essence of the evaluation. for example.” the “here” being the evaluand and its context. and issues of stakeholding audiences. for example. He or she must: 1. “qualitative” methods.a Fourth Generation) Evaluation 9 . which in constructivist terms they do not deserve. Both types of methods may be and often are appropriate in all forms of evaluative inquiries. and disenfranchisement. It is simply a caveat that mixing paradigms may well result in nonsense approaches and conclusions. ameliorated. mental constructions. from a prior evaluation or from a project proposal). a hard and fast character. that is. Authors of evaluation literature. demonstrate relevance (enable the core problems to be resolved. some meanings (information) and some level of sophistication in their interpretation are already available. including evaluation reports. Within a constructivist framework those same data are seen as variable and transformable. Discovery and assimilation are not necessarily sequential processes.. THE PROCESS OF CONSTRUCTIVIST EVALUATION: RESPONSIBILITIES OF THE CONSTRUCTIVIST EVALUATOR Constructivist evaluation is a process for doing evaluation that meets two conditions: It is organized by the claims. if those bases are not constructivism. concerns. and it utilizes the methodology of the constructivist paradigm. those stakes are determined by and defined by the stakeholders (in their own terms) and not only by the evaluator or the client seeking the evaluation (although they too are stakeholders and may identify their own stakes and definitions). There are many ways in which the discovery question can be answered. It is not appropriate to “mix and match” paradigms in conducting an evaluation. but may overlap or be carried out in parallel. objectivity. work (explain what happens). care must be taken to assess their paradigmatic bases. See sections on “discovery” and “assimilation” below. and exhibit modifiability (be itself open to change).k. Given that mandate. in particular. or better defined).g. depending on what specific relevant and preexisting constructions are brought to the inquiry by the evaluator and by local informants and respondents. power. Discoveries are themselves semiotic organizations. Note that this methodological assumption is silent on the subject of methods and. Negative stakes may include possible exploitation. The discovery phase may not be needed (or may be needed only minimally) if there is a preexisting construction or constructions relating to the evaluand on which to build (e. disempowerment. In other cases the appearance of key concepts such as generalizability. CAVEAT: If the preexisting constructions are drawn from sources external to the subject evaluation. and the like. proof. utilizing both scientific (positivist) and constructivist propositions within the same study. This is not a call for “purity” nor is it intended to be exclusionary. status. Identify the full array of stakeholders who are at risk by virtue of the stakes they hold in the entity being evaluated. typical of positivism and other nonconstructivist approaches. that are based on constructivist principles will almost certainly make their intent plain. So. serious disjunctions could easily be overlooked. it is possible to list the nine major responsibilities that the constructivist evaluator must discharge. if the new discovery is sufficiently different from or in conflict with the existing construction or constructions. depending on the view of the constructor.e. The assimilation phase of constructivist evaluation represents the evaluator’ s effort to incorporate new discoveries into the existing construction or constructions (or..the dialectic. on the subject of “quantitative” vs. Such stakes may include but not be limited to money. may be key signals as to the intent of the author.

they pause until a further need or opportunity for review and reassessment emerges.) 9. concerns. later round of evaluation activity. arranged by the evaluator. then the products of these intragroup negotiations (defined constructions. in the hope that their number can be reduced. Further. or incomplete. and their related claims. New aspects may be explored that have emerged on the basis of the first-round evaluation. as possible. and different claims. 7. the several stakes entering into the evaluation mix are assessed and refined in the effort to come as close as possible to negotiated agreement. Because more information may be required than it is possible to obtain. or confrontational settings. concerns. Failure to reach consensus implies the continuation of competing constructions. and issues. Generate consensus with respect to as many constructions. lest this need be taken as an opportunity to disempower selected stakeholders. It is the evaluator’s responsibility to seek out all stakeholders.evaluate in their own terms all information that the evaluation may disclose. and issues that they have raised (as well as those raised by other groups that appear relevant to that group). and issues. adversarial. concerns. and issues. Develop a report. The evaluator’s task is to identify the information needed. 6. subjected to critique. Constructivist evaluations are never completed. (See below for additional observations on the reporting process. 8. as well as unresolved claims. concerns. but retained for further action (and inclusion in the evaluation report) if there is agreement on that action. it can be eliminated from further discussion. which may provide the vicarious experience needed to influence stakeholder constructions. can be understood. are reviewed in light of the new information and/or level of sophistication. Outcomes of this forum must include action steps if the negotiation is to be regarded as successful. given time and/or resource constraints. concerns. Unresolved differences in constructions. including even those who may wish to maintain low visibility or to absent themselves entirely. concerns. deleted. It is likely that some items will remain unresolved. Prepare an agenda for negotiation on items about which there is no. Stakeholder inputs are essential in this determination. that communicate to each stakeholder group any consensus on constructions and any resolutions regarding the claims. The initial list may be rearranged. 2. but the evaluator must make every good faith effort to do so. concerns. Recycle the evaluation to take up still unresolved constructions and their attendant claims. Elicit from the stakeholder groups their constructions about the form and process of the evaluand and the range of claims. The process is first carried out within specific stakeholder groups. in dialogic. The most useful form for such report(s) is the case study. If consensus can be achieved with respect to an item.a Fourth Generation) Evaluation 10 . thereby setting the stage for another. and issues) are further negotiated in hermeneutic circles that cut across stakeholder groups. Collect and provide the information called for in the agenda for negotiation. Consensus should first be sought on an intragroup basis and then on an intergroup basis. Establish and mediate a forum of stakeholder representatives in which negotiation can take place. Provide a context and a methodology (the hermeneutic/dialectic) through which different constructions of the evaluand. if necessary. In the subsequent hermeneutic/dialectic process. the evaluator must devise some means (preferably also through a hermeneutic/dialectic process) for prioritizing the unresolved items. and issues. CONTRACTING FOR A CONSTRUCTIVIST EVALUATION Guidelines and Checklist for Constructivist (a. probably several targeted reports. claims. consensus. training must be provided.k. 4. and issues they wish to raise in relation to it. The provision of needed information cannot be guaranteed. or have additions made as the evaluation proceeds. 3. and taken into account. if stakeholders lack the sophistication to deal with obtained information. which disjunction(s) can be ameliorated only through the introduction of new information or an increase in the level of analytic sophistication. 5.

It is prudent to initiate a contract for an evaluation of any sort. it is clearly necessary to identify. The construction held by the client or sponsor is at best an initial form that is likely to evolve into a series or progression of constructions that emerge. or physician. existing documents. It should be noted that an evaluator does not enjoy special privilege as does. for example. A statement of intent from the evaluator with respect to stakeholding audiences. Stakeholders in an evaluation are entitled to know who the client or sponsor is since that position clearly represents a priori power and vested interests. The hermeneutic/dialectic methodology employed in constructivist evaluations clearly militates against that possibility. 9.k. seek out. A statement of agreement from the client or sponsor particularly. and the like. A contract should be drawn that protects both the client from evaluator misrepresentation or malpractice and the evaluator from client misunderstanding or misexpectation. but also from stakeholder groups. 4. 5. a willingness to reconsider one’s own value positions. The case report is the preferred mode. Guidelines and Checklist for Constructivist (a. other knowledgeable informants. 7. Such a contract should cover the following points at a minimum (other stipulations may be added as seems appropriate in the actual situation). and psychotic or self-deluded personalities). e. Reports must be freely available to all stakeholder groups. 2. summative/worth. and the evaluator must take the responsibility to explicate reports for those stakeholders who may lack the background..a Fourth Generation) Evaluation 11 . A description of the reporting modes to be utilized. The contract should also note and make provision for an emergent evaluation design. an attorney. A brief description of the methodology to be used. to adhere to the “Conditions for a Productive Hermeneutic/Dialectic. giving vicarious experience. If stakeholder claims. the mentally handicapped. This guarantee must take account of legal protections where they exist and must provide for procedures to be followed in the event that access becomes blocked. Identification of the evaluand (the entity to be evaluated).g. information. but especially so for a constructivist evaluation. minimal competence on the part of all parties to communicate (a condition that may exclude children. since it is highly unlikely that all stakeholders will be known at the onset of the evaluation. serving as a metaphoric springboard (a form of naturalistic generalization). a willingness to consider change. a willingness to share power. and a willingness to commit the time and energy needed. 6.” These include a commitment from all parties to work from a position of integrity (always recalling that false commitments can be and sometimes are made). summative/merit. for example. and challenging constructions of various stakeholders in ways that lead to reassessment and reconstruction. and involve all relevant stakeholders. The search for stakeholders should continue throughout the period of the evaluation. formative/worth. 8. clergyman. or some combination of these forms. 1. In those cases the evaluator must make every effort to construct their likely positions from whatever sources may be available. A statement of purpose for the evaluation: formative/merit. Identification of the client or sponsor of the evaluation. since this form is neither widely known nor commonly understood. The purposes of the case report include providing thick description. particularly since clients are likely to assume that an a priori and thereafter fixed design can be devised. or sophistication to deal with them. documents. Some stakeholders may refuse to participate. and issues (including those of the client or sponsor) are to serve as the focus for the evaluation. A guarantee of access to records. concerns. and respondents. because they fear to make their positions known because of retaliation from more powerful groups. A statement of the evaluator’s intent to guarantee confidentiality and anonymity of information sources insofar as that can be legally accomplished. 3. residues from earlier actions such as legal cases.

Collecting additional information and adding sophistication in its use by training negotiators.k. soliciting descriptions (constructions) of the evaluand and identifying and probing claims. formative/worth. This step 9 effectively recapitulates steps 3-8 for a newly formed hermeneutic circle consisting of persons selected by the individual circles as their representatives. concerns. It is virtually certain that some items will not have been negotiated to the satisfaction of all stakeholder groups. concerns. 9. A listing of technical specifications. and issues. Organizing the evaluation: Selecting the initial team of evaluators. and other sources found to be relevant. 2. 1. Developing intergroup constructions. 6. seeking new information.10. Guidelines and Checklist for Constructivist (a. and/or summative worth. a tentative schedule (not a design). interplay of in-group interview data with observational data. concerns. Developing intrastakeholder group constructions: forming multiple hermeneutic circles of 1012 members each representing one stakeholder audience. THE USE OF THE HERMENEUTIC/DIALETIC METHODOLOGY The constructivist evaluation is carried out through a series of steps which. claims. Prioritizing unresolved items via a negotiated prioritizing process determined by and involving the stakeholder group members. The listing begins at the point at which a contract satisfactory to all parties has been agreed upon. supporting. 5. identifying “beneficiaries” as well as “victims” of the evaluand’s action. Identifying stakeholders: Identifying agents commissioning and carrying out the evaluand. concerns. making entree arrangements. Reporting on the results for Step 9. working at illuminating. mounting continuing search strategies for other stakeholders. making logistical arrangements. or refuting items (providing additional training as needed). CONDUCTING THE CONSTRUCTIVIST EVALUATION. concerns. The report should be aimed particularly at the stipulated purpose(s) of the evaluation. culminating so far as possible in negotiated agreements on all identified. performing special studies as needed. Preparing the agenda for negotiation by defining and elucidating competing constructions. There may be several reports tailored to the claims. Agreements on elements of these reports may lead to proposed action steps. The result is a composite construction that includes all forms of the evaluand constructions as well as their relevant claims. 4.a Fourth Generation) Evaluation 12 . and formalizing agreements with and among them. may well be iterative and reiterative in practice as constructions evolve and as particular claims. while listed here in serial form. Step 8 will have resulted in a negotiated agenda for each of the several stakeholder groups. 7. and issues are dealt with. setting these aside as possible case report components. The serial form below is used as a matter of convenience. literature analects. and a listing of likely products. and issues of specific stakeholder groups. and issues resolved by consensus. including the names and background of the agents who will carry out the evaluation (allowing for additions and deletions as may be required). that is. a budget (at all stages a good faith “best estimate”). 8. these are set aside for later reconsideration in a subsequent recycling . formative/merit. 10. assessing trade-offs and sanctions. summative/merit. existing documentary information. and issues that emerge. 3. and testing the agenda derived. and assessing local political/cultural factors. Enlarging joint intrastakeholder group constructions utilizing the evaluator’s prior construction (but allotting it no special privilege). Sorting out constructions.

. and objectivity.11. peer debriefing (a kind of external critic). b. a case study is never finished. targeted to specific stakeholder audiences. in guiding methodological decisions during the evaluation and later in auditing the overall evaluation process (see c and d below). These evolved from an effort to produce criteria more or less parallel to those conventionally used. more sophisticated ways of analysis and interpretation until some level of consensus is reached.e. established by prolonged engagement at the site. and (most important) member checks. roughly parallel to internal validity. Recycle the entire process to take particular account of elements set aside in step 9 that were irresolvable at that time. finally. CONSTRUCTIVIST EVALUATION REPORTS The end product of a constructivist evaluation (but never a final product. but also the underlying motives. Throughout this process the stakeholders—individually. and they may take many forms. The “parallel” criteria (sometimes called “trustworthiness” or “foundational” criteria). first. their very “parallelism” to positivist tenets renders them less than fully adequate for determining the quality of a constructivist approach. i. contain an appendix that describes in detail the methodology followed and makes it possible to judge the extent to which quality criteria (those listed in the following section) are met. Two different approaches have been generated to deal with this dilemma. data. pp. internal and external validity. Credibility. it is merely due. reliability. CRITERIA FOR ASSESSING THE QUALITY OF CONSTRUCTIVIST EVALUATIONS AND REPORTS Standards normally applied in making quality judgments of evaluations. However. for example. and rationales leading to those beliefs. There may be multiple reports. Instead. negative case analysis (a process of reworking postulated hypotheses). since it is subject to successive iterations) is the case report. In a sense. The report does not culminate in judgments. roughly parallel to external validity. They are probably most useful. or recommendations except insofar as these are concurred on by relevant respondent stakeholders. Transferability. are inappropriate for constructivist evaluations precisely because they are based upon a fundamentally different theoretical paradigm (as explained in the opening paragraphs of this statement). both are useful during the evaluation process as procedural checklists and afterward in assessing the completed evaluation report (product) for quality: 1. The case report is characterized by a thick description that not only clarifies the allimportant context but that makes it possible for the reader to experience it vicariously. persistent observation. The case report helps the reader realize (in the sense of making real). conclusions. in similar groups. not only the states of affairs that are believed by stakeholders to exist.k. the case report is the joint construction that emerges as the result of the hermeneutic/dialectic process. and interpretations with members of stakeholding audiences. possibly not including what might normally be termed a “technical” report. the Joint Committee Standards or the Guiding Principles for Evaluators of the American Evaluation Association. progressive subjectivity (continuous checking of developing constructions against records of constructions that were expected prior to data collection). They are exposed to new information and new. continuous testing of hypotheses. The case report must. 233-43): a.a Fourth Generation) Evaluation 13 . These parallel criteria are (full definitions can be found in Fourth Generation Evaluation. feelings. and across groups— are chosen to uncover widely variable viewpoints. preliminary categories. if such a report is beyond the competence of a stakeholding audience to deal with. established not by the evaluator but by receivers of evaluation reports who make personal judgments of the degree to which findings are sufficiently similar to their own situations (judged from the thick description) to Guidelines and Checklist for Constructivist (a.

which determines the extent to which constructions. Whereas the parallel criteria are embedded in the assumptions of positivism. In this process. Tactical authenticity. and the Guidelines and Checklist for Constructivist (a. exposed.warrant testing for the viability of local application (testing for localization rather than the more usual generalization). facts.k. Educative authenticity. to determine the methodological decisions made and to understand the reasons for them. These criteria are (full definitions can be found in Fourth Generation Evaluation. Confirmability. The publicly inspectable and inspected nature of the hermeneutic/dialectic process itself prevents much of the kinds of secrecy and information poverty that have characterized client-focused evaluations. Catalytic authenticity. and data can be traced to their sources. assertions. the degree to which they can articulate their existing constructions of the evaluand and the claims. moving to eliminate or ameliorate problems. determined by an assessment of the extent to which individuals are empowered to take the action that the evaluation implies or proposes. be incorporated into the joint. Dependability. The “raw products” and the “processes used to compress them” are inspected and confirmed as appropriate. sharpening values) is stimulated and facilitated by the evaluation. it is necessary for the evaluator to play a dual (and sometimes conflicting) role: advocate and educator. any intent on the part of the evaluator to favor particular stakeholders is at least equally detectable. collaborative reconstruction that emerges as the process continues. pp. And finally. determined by an assessment of the extent to which action (clarifying the focus at issue. e. Those inputs will. the authenticity criteria are based directly on the assumptions of constructivism and are responsive to the hermeneutic/dialectic aspects of that paradigm. the ability of the hermeneutic/dialectic process to act as a powerful source of quality control should not be overlooked. for a quality evaluation to result. in the negotiated emergent construction. concerns. It is difficult to maintain false fronts or support deliberate deception when information is subject to continuous and multiple challenges from a variety of stakeholders. In virtually every situation the stakeholding audiences will differ greatly in the amount of information they bring to the table. roughly parallel to objectivity. c. d. expansion. Second. It is the immediate and continuing interplay of information that militates against the possibility of noncredible outcomes. roughly parallel to reliability. They are “fed back” for comment. furthermore. determined by an assessment of the extent to which individuals (including the evaluator) have become more understanding (even if not more tolerant) of the constructions of others. determined by an assessment of the extent to which individual constructions (including those of the evaluator) have become more informed and sophisticated. established through the use of the dependability audit with the assistance of an external auditor. or emendation to the very respondents who provided them just a moment before. correction. The authenticity criteria. Opportunities for errors to go undetected and/or challenged are very small indeed under those circumstances. b.a Fourth Generation) Evaluation 14 . d. 2. Ontological authenticity. and taken into account in the evaluation report. Two other observations are appropriate in respect to the quality question. Fairness. and issues they experience. that is. the inspection being done by an external auditor (who may be the same or different from the dependability auditor). elaboration. First. determined by an assessment of the extent to which all competing constructions have been accessed. who examines the record of the inquiry in the way a fiscal auditor examines fiscal records. data inputs are analyzed immediately on receipt. 245-250): a. c. revision.

work. of which there are many. We believe this form obviates the major problems of the first three generations: a tendency toward managerialism. Furthermore. Guidelines and Checklist for Constructivist (a. we think. It is a diffuse process impossible to specify in detail (in design form). It is ever-recursive and requires frequent recapitulations.degree of sophistication they possess in processing new information that emerges. thus. and an overcommitment to the scientific (positivist) paradigm of inquiry. concerns. a second generation focused on description. It is this fourth generation form of evaluation that is the subject of this checklist and set of guidelines. it is incumbent on the evaluator to provide the training (and if necessary. and the evaluator will need to exercise great care to avoid bias and favoritism. an evaluation approach that favors the point of view of the client or funder. The provider of the checklist has not modified or adapted the checklist to fit the specific needs of the user and the user is executing his or her own discretion and judgment in using the checklist. The provider of the checklist makes no representations or warranties that this checklist is fit for the particular purpose contemplated by user and specifically disclaims any such warranties or representations.k. and that is disempowering. Yet from a value-oriented view. unfair. hence. It is highly labor intensive.a Fourth Generation) Evaluation 15 . that is. and exhibit continuing modifiability.” It requires the evaluator to play multiple roles which at times may appear to be in conflict. a failure to accommodate value-pluralism. It is one of the more realistic and socially—and politically—sensitive approaches to performing useful—and utilized—evaluations. some of which may be highly technical. The required balance between these roles is delicate. and disenfranchising to selected stakeholders. its personnel and resource commitments can at best be “guesstimated. the representation) they need. Constructivist evaluation is a difficult model to adopt. the hermeneutic/dialectic process itself is not one in which they are well versed. that inappropriately saves the manager harmless. CODA Constructivist evaluation differs in fundamental ways from other forms of evaluation. It denies the possibility of reliable generalizations and of determining solutions “that work” everywhere. the best way to evolve viable and acceptable solutions to claims. If is often adversarial and confrontational. now dubbed constructivist evaluation. This checklist is being provided as a free service to the user. In Fourth Generation Evaluation we described the historical evolution of evaluation practice: a first generation focused on measurement. and issues widely felt and to the formulation of constructions widely seen to fit. it is. a third generation focused on judgment. and a fourth generation focused on negotiation (the hermeneutic/dialectic). demonstrate relevance.