You are on page 1of 5


The National Academies of Sciences, Engineering and Mathematics in Washington

drafted a guideline for evaluating instructional materials in the sciences. They created

a committee that reviewed their evaluation guidelines.

Here are the principles behind the guideline:

1. The evaluation tool should fulfil needs not met by other instruments.

The Committee identified unmet needs from its analysis of the review tools

available for instructional materials. They found that National Science Foundation’s

(NSF) Framework for Review has narrow scope as it only addresses the question

if the instructional materials will lead to student learning and understanding.

2. The evaluation tool should assume that a set of standards and a

curriculum program or framework will inform the work of evaluators in

appraising the effectiveness of instructional materials.

Evaluation of science instructional materials is a formidable task. A set of standards

and a curriculum program or framework documents the school district's expectations

for science education and serves as an important reference for the evaluation.

Moreover, the existence of such policies implies an established community support for

the science education program, which in turn can promote acceptance of

recommended instructional materials. However, the committee chose local standards

understood by the locality and matching ground situation. They also created flexible

tool that is compatible with any standards, in any situation including the review of a

whole instructional program, a series of units, or individual units of instruction.

3. An evaluation process should require reviewers to provide evidence to

support their judgments about the potential of the instructional materials

to result in student learning.

Most evaluation tools used in a limited time period are checklist of items for

consideration, a numerical scale, and weighted averages of the numerical evaluations.

These tools have superficial nature because they do not examine and foster the

teaching - learning process and the relationship between the students and their

teachers. The Committee recommended that evaluators must conduct rigorous

evaluation process to identify evidences of the materials' potential effectiveness.

4. Evaluators will more likely provide critical and well-thought-out

judgments if they are asked to make a narrative response to evaluation

questions or criteria, rather than make selections on a checklist.

They also concluded that narrative responses and not checklist are better a format for

giving feedback. Evaluators response when narrated is more relevant and to the point

because they have more time think about their answers compared to merely putting a

mark on a checklist. Narrative responses also help in the professional development of

an evaluator because of the depth of understanding that they gain while evaluating. In

addition, evaluators have more latitude to assess materials in the context of local goals

and needs and allow the evaluators (teachers and scientists alike) to contribute their

own knowledge and experience to the task when they adapt narrative responses as

form of evaluation. The Committee concluded that narrative should become the

standard when evaluators pass professional judgment or criticism on instructional

5. An effective evaluation process must include one or more scientists

on the review teams.

Published materials, books and journals become outdated and obsolete in time. Some

instructional materials include irrelevant facts and unsubstantiated data in a particular

field or discipline. Some books have detailed information which is of little or no

relevance, unnecessary vocabulary and casual treatment of the essential concepts.

Experts and scientists in the review team are helpful in judging the accuracy of facts.

6. An evaluation instrument needs to serve diverse communities, each one

of which has its own needs.

Since the evaluation instrument will be used by different groups for a variety of

purposes, the Committee proposed that no single model will be implemented. In most

cases, evaluation will be school – based, district or state – wide. These groups will

have diverse resources and the students taught will have varying languages, abilities,

economic status, strengths and weaknesses, and home support resources. Thus, the

committee determined to design an evaluation tool that is adaptable.

7. Tension exists between the need for well-informed, in-depth analyses of

instructional materials and the real limitations of time and other


In January 1996, to double – check their view of the barriers to implementation of

national standards, the National Science Teachers Association, surveyed 10% of its

members. Two major hurdles were identified: lack of time and lack of other resources.
It is therefore, paramount to develop a tool that verifies the real limitations experienced

by evaluators.

8. Many evaluators (including teachers, administrators, parents, and

scientists) using the tool will be unfamiliar with current research on


Decision makers that formulate curriculums are not always informed by research on

learning, but rather on what feels comfortable to teachers, what seems to work, or

what is expected (Orpwood, 1998). After graduation, when they are already teaching,

teachers become absorbed with their workloads and seldom access research

publications about education. Seminars, trainings and workshops rarely devote time

to in-depth study of current research on learning and teaching processes. Thus,

evaluation coordinators should provide the most updated resources and references,

including findings on recent studies in education.

9. It is more important to evaluate materials in-depth against a few relevant

standards than superficially against all standards.

The pressures of limited time and funds can drive an evaluation team to inspect

instructional materials superficially against all relevant standards. The Committee

concluded that if time and funds are limited, it is preferable for the team to select a

small number of high-priority standards for an in-depth examination.

10. The review and selection processes should be closely connected even

when reviewers are not members of the selection committee.

In some school districts, one team evaluates instructional materials and reports to

another group that is responsible for final approval and selection. In others, one team

is responsible for both evaluating and selecting instructional materials. Considerations

such as cost, the local district's ability to refurbish materials, and political acceptability

(e.g., attitudes about teaching evolution) may play a role in the final selections. The

Committee concluded that in all instances it is important that final selections be based

primarily on a standards-based review. It is therefore important that one or more of the

members of the evaluation team be on the selection committee.