Permit evaluator to enter into and understand situation/context

Provide good opportunities for identifying unanticipated outcomes

Exist in natural, unstructured, and flexible setting

Disadvantages

Expensive and time consuming

Need well-qualified, highly trained observers; may need to be content experts

May affect behavior of participants

Selective perception of observer may distort data

Investigator has little control over situation

Behavior or set of behaviors observed may be atypical

Observations during the summative phase of evaluation can be used to determine whether or not the project is
successful. The technique would be especially useful in directly examining teaching methods employed by
the faculty in their own classes after program participation. Exhibits 3 and 4 display the advantages and
disadvantages of observations as a data collection tool and some common types of data that are readily
collected by observation.

Readers familiar with survey techniques may justifiably point out that surveys can address these same
questions and do so in a less costly fashion. Critics of surveys find them suspect because of their reliance on
self-report, which may not provide an accurate picture of what is happening because of the tendency,
intentional or not, to try to give the "right answer." Surveys also cannot tap into the contextual element.
Proponents of surveys counter that properly constructed surveys with built in checks and balances can
overcome these problems and provide highly credible data. This frequently debated issue is best decided on a
case-by-case basis.

Recording Observational Data

Observations are carried out using a carefully developed set of steps and instruments. The observer is more
than just an onlooker, but rather comes to the scene with a set of target concepts, definitions, and criteria for
describing events. While in some studies observers may simply record and describe, in the majority of
evaluations, their descriptions are, or eventually will be, judged against a continuum of expectations.

Observations usually are guided by a structured protocol. The protocol can take a variety of forms, ranging
from the request for a narrative describing events seen to a checklist or a rating scale of specific
behaviors/activities that address the evaluation question of interest. The use of a protocol helps assure that all
observers are gathering the pertinent information and, with appropriate training, applying the same criteria in
the evaluation. For example, if, as described earlier, an observational approach is selected to gather data on
the faculty training sessions, the instrument developed would explicitly guide the observer to examine the
kinds of activities in which participants were interacting, the role(s) of the trainers and the participants, the
types of materials provided and used, the opportunity for hands-on interaction, etc. (See Appendix A to this
chapter for an example of observational protocol that could be applied to the hypothetical project.)

Exhibit 4.
 Types of information for which observations are a good
source

The setting - The physical environment within which the project takes place.

The human, social environment - The ways in which all actors (staff, participants,
others) interact and behave toward each other.

Project implementation activities - What goes on in the life of the project? What
do various actors (staff, participants, others) actually do? How are resources
allocated?

The native language of the program - Different organizations and agencies have
their own language or jargon to describe the problems they deal with in their work;
capturing the precise language of all participants is an important way to record how
staff and participants understand their experiences.

Nonverbal communication - Nonverbal cues about what is happening in the

project: on the way all participants dress, express opinions, physically space
themselves during discussions, and arrange themselves in their physical setting.

Notable nonoccurrences - Determining what is not occurring although the

expectation is that it should be occurring as planned by the project team, or noting
the absence of some particular activity/factor that is noteworthy and would serve as
added information.

The protocol goes beyond a recording of events, i.e., use of identified materials, and provides an overall
context for the data. The protocol should prompt the observer to

Describe the setting of program delivery, i.e., where the observation took place and what the physical
setting was like;
Identify the people who participated in those activities, i.e., characteristics of those who were present;
Describe the content of the intervention, i.e., actual activities and messages that were delivered;
Document the interactions between implementation staff and project participants;
Describe and assess the quality of the delivery of the intervention; and
Be alert to unanticipated events that might require refocusing one or more evaluation questions.

Field notes are frequently used to provide more indepth background or to help the observer remember salient
events if a form is not completed at the time of observation. Field notes contain the description of what has
been observed. The descriptions must be factual, accurate, and thorough without being judgmental and
cluttered by trivia. The date and time of the observation should be recorded, and everything that the observer
believes to be worth noting should be included. No information should be trusted to future recall.

The use of technological tools, such as battery-operated tape recorder or dictaphone, laptop computer,
camera, and video camera, can make the collection of field notes more efficient and the notes themselves
more comprehensive. Informed consent must be obtained from participants before any observational data are
gathered.

The Role of the Observer

There are various methods for gathering observational data, depending on the nature of a given project. The
most fundamental distinction between various observational strategies concerns the extent to which the