You are on page 1of 3

Practice: Get quality data into your evaluator’s hands

Key Action: Use technique to ensure valid and reliable data

TOOL: Making Sure Data Are Valid and Reliable

Purpose: Before you can confidently interpret and analyze your evaluation data, you
must ensure that the data you collect are valid and reliable. Otherwise, they
won’t adequately support your outcomes. Use the questions and suggestions
in this table to ensure the data you collect are valid and reliable.

Instructions: 1. Review the “Questions to Consider” in the table to ensure that the data
you collect for your evaluation are valid and reliable, and that your sample
size is adequate.

2. Consider the suggestions or proactive measures you might take to ensure
valid and reliable data.

3. Based on what you learn, note specific actions you might use in your
district to ensure that your evaluation generates valid and reliable data.

1

or pilot test the instruments in real settings and stable responses regardless of among members of your target audience.g. differences between interviewers or data collectors. open-ended interviews) poses different validity challenges. Practice: Get quality data into your evaluator’s hands Key Action: Use technique to ensure valid and reliable data Making Sure Data Are Valid and Reliable Questions to consider Suggestions Actions we’ll consider Do the data represent the First make sure you select instruments that measure the outcomes that the instrument kinds of outcomes your magnet program is expected to is supposed to measure? Are produce. have an expert panel review and react to new A reliable measure produces instruments. Build in time and the data collector. Ideally. observations. which Note: State assessments have been recalibrated or revised allows for comparison of results during the years of your study.. Provide time and resources for researcher training in use of the instruments to minimize differences in participant responses across data collectors. compare data from year to year. Have we ensured that our Discuss with your evaluator how you will test for reliability of measures are reliable? your instruments. A valid measure assesses what it is designed to measure. the instruments you create for these purposes will require low levels of inference. so you may not be able to across studies. An unreliable resources to test for reliability and calculate reliability measure will yield varied coefficients so that you can assure stakeholders that you responses depending on have strong instruments. You can increase measurement validity by using the data valid? field-tested instruments that have demonstrated reliability and validity. You may want to borrow from existing instruments. 2 . Note that qualitative data collection (e.

sufficient number of participants Otherwise you may not have enough students in particular to be representative of your subgroups to make those data valid. Practice: Get quality data into your evaluator’s hands Key Action: Use technique to ensure valid and reliable data Questions to consider Suggestions Actions we’ll consider Do we have the right data? You will need an adequate sample size to ensure your data Do we have enough data? are valid. Make sure you begin with large enough numbers of students and schools in your evaluation study. check your data to make sure that subgroup data questions about program are appropriately coded and that sufficient numbers of outcomes and include a students in subgroups have taken the tests as planned. projected attrition into account. or multiple researchers conduct a structured observation of the same phenomenon. target population and its various subgroups. as data collection data must answer your begins. taking To be valid for decision-making. 3 . Then.” for example. when various methodologies are used to measure the same phenomenon. Validity of data is improved when data collection is “triangulated.